Anthropic has proposed legislation that could suspend the release of AI models if independent evaluators find “imminent risks” involving biological weapons, evasion of controls, or cyberattacks, where the risk could result in more than 50 deaths or cause over $1 billion in property damage.
If a risk is identified, evaluators would have 72 hours to notify the U.S. Department of Commerce. The department would then have to seek an order from a judge in Washington, D.C. to halt the release. Federal agencies would not have the authority to order an immediate suspension on their own.
Anthropic has proposed AI safety legislation for Congress to consider, building on the proposals in the Advanced AI Framework that the company has promoted since June.
The core idea is to have independent evaluators registered with the U.S. Department of Commerce assess model risks. The risks must involve the creation or deployment of biological weapons, evasion of developer controls, or cyberattacks before they could justify a request to suspend a release.
When a Model Could Become an Unstoppable Threat
Recently, there have been incidents in which AI programs appeared not to follow human instructions and proceeded to attack websites on the internet, raising questions about whether models have sufficient security controls.
The key issue is therefore not simply how capable a model is, but when its release should be stopped and who should make that decision. If authorities wait until real harm occurs, it may already be too late to intervene.
From Anthropic’s Safety Framework to Legislation
This legislation builds on Anthropic’s Advanced AI Framework, which the company has promoted to lawmakers at both the federal and state levels since June. It proposes rules for situations in which a model presents a “foreseeable, imminent risk.”
The idea is to use independent evaluators to assess risks and report them to the Department of Commerce within 72 hours if a serious risk is found, before referring the matter to a judge in Washington, D.C. to consider suspending the model’s release. This makes Anthropic’s proposal go further than the voluntary evaluation systems previously proposed by several companies in the industry.
From Internal Evaluators to Independent Evaluators
Anthropic is moving from a voluntary system involving third-party evaluators embedded within organizations to independent evaluators registered with the Department of Commerce. The difference is that the evaluation process would be positioned further outside the company’s structure.
| Factor | Voluntary industry system | Anthropic legislation |
|---|---|---|
| Evaluator model | Third party within the organization | Independent evaluators |
| Oversight | Voluntary system | Registered with the Department of Commerce |
| Payer | AI companies | AI companies, but payment cannot be tied to evaluation results |
However, AI companies could still pay the evaluators, raising questions about impartiality even though the legislation would require payments not to depend on evaluation results.
The Path from Risk Assessment to a Court Order
The process would begin with independent evaluators assessing whether a model presents a “foreseeable, imminent risk,” such as the potential to cause more than 50 deaths or over $1 billion in property damage. The risk must involve biological weapons, evasion of developer controls, or cyberattacks.
If a risk is identified, the evaluators must notify the Department of Commerce within 72 hours. The department would then have to ask a judge in Washington, D.C. to order a suspension of the model’s release. The legislation would also prohibit federal agencies from ordering a release to be halted on their own.
Anthropic’s Position Compared with Other Industry Proposals
Anthropic is proposing legislation that would allow models to be halted when there is a “serious risk,” while California Senate Bill 53 uses a similar risk threshold. Jensen Huang, meanwhile, believes that the AI industry does not need new legislation.
| Factor | Anthropic legislation | California Senate Bill 53 |
|---|---|---|
| Status | Proposed legislation | Enacted into law |
| Risk threshold | Could result in more than 50 deaths or more than one billion dollars in property damage | Defines a similar situation as a serious risk |
| Approach | Increases the role of government oversight | Establishes the risk framework in law |
The difference therefore lies in their attitudes toward new legislation: Anthropic is pushing for a greater government role, while Jensen Huang believes additional laws should not be introduced.
Strengths and Outstanding Questions
The legislation’s advantage is that it clearly defines the risk thresholds—more than 50 potential deaths or over $1 billion in property damage—and establishes a process for courts in Washington, D.C. to consider suspending a model’s release.
The key question concerns the impartiality of independent evaluators, since AI labs could still pay them even though the law would state that payments must not depend on evaluation results.
Pros
- +Clearly defines the risk thresholds and court process
- +Allows independent evaluators to assess model risks
Cons
- −It remains unclear how influence from the payer would be prevented
- −Evaluators could lack impartiality if they depend on AI labs
The Price That Does Not Appear in the Numbers 50 or $1 Billion
The first cost would be the burden placed on independent evaluators, who would have to analyze model risks and notify the Department of Commerce within 72 hours when a serious risk is found. The system would still allow AI labs to pay the evaluators, even though payments could not be tied to evaluation results.
Another layer is legal complexity. The Department of Commerce would not have the authority to order a release to be halted on its own, but would have to petition a judge in Washington, D.C. This raises questions about both the speed of the response and how to ensure that evaluations are genuinely independent in practice.
Conclusion: Who Should Have the Right to Press the Model’s Stop Button?
The key issue is not simply who presses the stop button, but who oversees the people with that authority. Independent evaluators would have to be registered with the Department of Commerce, while the courts would make the final decision, creating a balance of power among AI companies, evaluators, the executive branch, and the justice system.
This model attempts to ensure that release suspensions are supported by clear evidence and procedures, but it still has to prove how independent the process can truly be—especially when AI companies are paying the evaluators themselves. This may determine whether safety rules become a genuine oversight mechanism or merely a new form of self-regulation.