Anthropic Pitches AI Safety Bill With a 50 Death Threshold for Blocking a Model

Anthropic has been pitching United States lawmakers a draft AI safety bill that would let the government block the release or use of an AI model, but only in a narrow set of catastrophic cases, Politico reported in a story TechCrunch covered on Tuesday, October 20. It is the clearest statement yet of how the company behind Claude wants Washington to police the most powerful AI systems.

Under the draft, the Commerce Department could ask a judge to halt a model only if it poses a "foreseeable, imminent risk" of killing or seriously injuring more than 50 people, or of causing more than $1 billion in property damage, Politico wrote. The risk would have to come from one of three sources: the creation or release of a biological weapon, a model evading its developer's controls, or a cyberattack.

How the system would work

The draft relies on independent evaluators, registered with the Commerce Department, who would periodically assess AI labs and their frontier models, Politico reported. The evaluators could be paid by the AI labs themselves, as long as the payment is not tied to the results of the evaluation. TechCrunch noted that the draft does not make clear how that arrangement would stop money from influencing an evaluator's findings.

If an evaluator found an imminent catastrophic risk, they would have 72 hours to alert the Commerce Department. The department would then have to ask a judge in Washington, DC to halt the model's release. No other federal department or agency could block a model's distribution on its own under the draft, Politico wrote.

The two thresholds, 50 people and $1 billion, mirror the definition of catastrophic risk in California's recently passed Senate Bill 53, TechCrunch reported.

Critics say the bar is set very high

The thresholds drew quick criticism. Cybernews reported that critics say the proposal lets AI companies help write their own rules while ignoring the harms AI systems cause today, and noted that a bioweapon risk killing fewer than 50 people would not trigger a government response under the draft.

An Anthropic spokesperson confirmed the draft bill to Politico and said it represents the proposals in the company's Advanced AI Framework, which Anthropic says it has been advocating with federal and state lawmakers since June. In late September, chief executive Dario Amodei set out a three step plan to slow the pace at which companies improve model capabilities, including third party evaluators with employee like access, and common safety standards backed by democratic governments, Cybernews reported.

Part of a wider fight over AI rules

The pitch lands in a live argument about who should police frontier AI. TechCrunch reported that AI labs have warned loudly about extreme risks from their own technology while resisting mandatory government oversight, and have mostly offered voluntary systems instead. Nvidia chief executive Jensen Huang has recently argued that the AI industry does not need new laws. The debate has sharpened after a series of incidents in which AI programs appeared to disobey their operators and hack websites on the open internet, TechCrunch noted.

Nothing in the draft is law. It is a proposal from one company, circulated in Washington, in a Congress that has not passed a broad AI safety statute. What it sets out, in specific numbers, is where one leading lab thinks the line for government intervention should sit: above 50 lives and $1 billion in damage, judged by evaluators the industry may pay, and enforced only through a court.