AI

Anthropic says AI is speeding up its own research enough to trip its safety rules

Adrian Kessler
Add us on Google

Anthropic has formally concluded that artificial intelligence is now speeding up its own research enough to cross a line the company drew for itself. In its amended IPO prospectus, the Claude maker says it has met the AI research and development capability threshold in its Responsible Scaling Policy, the internal rulebook that decides how much security and oversight its models need. The policy describes that threshold as the onset of recursive self-improvement, the point where AI systems start to accelerate the work of building better AI.

What makes the determination unusual is what it does not rest on. Anthropic writes that it reached it “based on what we expect from future models and our overall rate of progress, rather than on what we have observed from current models.” No Claude model on sale has been shown to cross the line. The company is acting on its own forecast, and it is telling investors so before its shares start trading.

The evidence it points to is internal. The prospectus says Claude wrote more than 90% of the contributions to Anthropic’s codebase during one recent month, and that the volume of code each engineer contributes per day has grown more than 45-fold. Anthropic adds that its models “write a large and growing share of our code and increasingly perform our research, including our safety research.”

The policy sets two ways to cross this threshold. One is a model that can fully replace Anthropic’s entire staff of research scientists and engineers at a competitive cost. The other is a “dramatic acceleration” of AI progress, which the policy defines as roughly two years of progress packed into one, plausibly driven by automated research. Because that second test counts if the company can “observe or expect” such a jump, a forecast is enough to trigger it.

Crossing the line commits Anthropic to a heavier set of safeguards. The policy calls for “moonshot” security projects against the best-resourced attackers in the world, and for an “eyes on everything” state in which every critical AI-development activity is logged and analyzed by AI for threats, including from insiders, human or machine. Its internal red team must also find more jailbreaks than all the outside researchers in its bug bounty programs combined. The prospectus says the company already delayed pre-training a new frontier model by more than a month to tighten protections against theft of its model weights, and that it is weighing rules requiring human sign-off on some research decisions as Claude does more of that work.

For people who use Claude at work or in Claude Code, the practical effect could be slower releases. The founders’ letter published alongside the IPO material says there may be times when Anthropic chooses “to advance more slowly than we could in order to make our systems safer, even if that puts us at a competitive disadvantage.” Nothing changes in the apps today.

The determination is a self-assessment under a voluntary policy that Anthropic wrote and can revise, and no outside auditor has confirmed it. The policy admits the threshold “has proven difficult to operationalize,” and the filing concedes that much of the evidence comes from lagging indicators, so a real acceleration might only be spotted after it happened. Risk disclosures in a prospectus are also written to protect a company from future lawsuits, and an earlier version of the same document, as Gizmodo reported, said only that Anthropic “may imminently meet” the criteria.

Anthropic says it is still assessing the underlying data and will publish further reports on what meeting the threshold means. Its shares are expected to be priced on November 18 and to begin trading on Nasdaq under the ticker ANT on November 19, with a proposed range of $70 to $78 per share. For now, the only proof that the line was crossed is Anthropic’s own forecast.

Tags: , , , , ,

Add us on Google

Discussion

There are 0 comments.