Anthropic METR Report Shows AI Speeding Up AI Research

The Anthropic METR report, published on Thursday by AI safety group METR after an 11-week review inside the company, says Anthropic's AI models are now improving roughly twice as fast as a year ago, mostly because AI is helping build the next generation of AI. The Verge reports that heavy redactions in the public version have alarmed many researchers.

The findings are being read as some of the clearest evidence yet that AI is speeding up its own development. They arrive as Anthropic prepares for a stock market listing while its chief executive, Dario Amodei, has publicly urged AI labs to slow down.

What the Anthropic METR report found

According to The Verge, METR's benchmark data suggests Anthropic has made about two years of historical progress in the last year. The researchers said they were confident this was not simply the result of using more computing power for training, which has long been the main driver of AI gains. Instead, they concluded that most of the acceleration as of October was due to AI itself, and that AI's share was growing.

Main points reported from the assessment:

• Staff using Anthropic's most capable internal model self-reported being around 12 times as productive as without AI help, about double the figure from a few months earlier.
• Most of the speed-up still comes from AI assisting human researchers, but METR warned there are signs fully autonomous work will soon play a bigger role.
• Researchers found no clear signs of diminishing returns or bottlenecks that would stop even faster acceleration.
• Anthropic finished training three unreleased models in the past six months, codenamed Model 2, Model 3 and Model 4 in the report.

The evidence included informal Slack polls and interviews with employees, as well as confidential research and model performance data that Anthropic chose to share.

Redactions spark a backlash

Anthropic had the right to redact parts of the final report under its agreement with METR, and it used that power widely. METR said on X that it handed Anthropic a report of roughly 100 pages, and that the public version had a majority of the most informative evidence removed. METR also said a small number of its data requests were denied.

Anthropic co-founder Sam McCandlish accepted on X that the redactions make METR's work harder to check, but argued that revealing what drives the speed-up would be commercially sensitive and could hand rivals a roadmap to go faster. He said the company will publish its own report on the pace of AI progress in the coming weeks, along with new safety steps. Anthropic declined to answer The Verge's questions directly.

Why researchers are worried

The concern is recursive self-improvement, where AI systems develop more powerful successors with little human involvement. Marius Hobbhahn, chief executive of Apollo Research, told The Verge the pace looked even faster than the influential AI 2027 scenario. Alan Chan of GovAI said sceptics should reconsider how soon tasks like software engineering could be fully automated.

Critics also accuse Anthropic of hypocrisy. Daniel Kokotajlo, who runs the AI Futures Project, said the company was "recklessly gambling with our lives". Brianna Rosen of the Institute for AI Policy and Strategy said it was concerning that companies rather than governments were effectively deciding how far to automate AI research, and that the report was not transparent enough about its evidence.

What the Anthropic METR report means for users

For businesses and developers in Thailand and across Asia, the report suggests the AI tools they rely on may keep improving quickly as frontier labs compete. Rival OpenAI said in September that it was making strong progress toward an AI research assistant by March 2028, and its funding has surged, as covered in our report on the OpenAI valuation.

At the same time, investors have shown signs of nerves about AI spending, as seen in the recent tech stocks plunge. Thailand is also trying to secure its place in the industry, as explained in our look at Thailand's AI supply chain role.

The debate is likely to intensify when Anthropic releases its own report, and as calls grow for independent evaluators to have more freedom to explain their conclusions.

This article is based on reporting by The Verge.

Photo: Anthropic CEO Dario Amodei, whose company is the subject of the Anthropic METR report, at TechCrunch Disrupt 2023 (TechCrunch via Wikimedia Commons, CC BY 2.0).