METR, the nonprofit evaluator, spent eleven weeks inside Anthropic and concluded that the lab is improving its models on well-specified tasks at about twice the rate it did between March 2024 and October 2025, with AI itself doing most of the accelerating. The headline number is 1.9 times faster since October 2025, with a 95 percent confidence interval of 1.5 to 2.8 times, which METR reads as roughly two years of historical progress in a single year. The report is unusually honest about what it cannot prove, because Anthropic redacted most of the evidence METR found most convincing. This article audits every headline number in the report, including what was redacted behind it.
The audit table: every headline number, checked
The table below is the whole pillar in one place. Each row gives the claim as METR states it, the number, the confidence or range METR attaches to it, where inside the report it comes from, and what remains uncertain. Nothing in the table is ByteHerald's own estimate. Where a number is ours, it is labeled as ByteHerald arithmetic.
| Claim | Number | Confidence or range | What is uncertain |
|---|---|---|---|
| Benchmark scores growing faster since October 2025 (AECI) | 1.9x | 95% confidence interval 1.5x to 2.8x; robustness analyses 1.1x to 2.9x | AECI measures verifiable tasks; the trend could reflect benchmark overfitting or an artifact of the index; data points are sparse and lagging |
| Annual AECI growth, before vs after October 2025 | 15 points a year to 29 points a year | Implies roughly two years of progress in one year, about the distance from Claude 2.1 to Sonnet 4.5 | Internal-only models included; snapshots dated by when most employees could use them |
| Broad judgment: improvement rate vs March 2024 to October 2025 | About 2x | METR would not be surprised at 1.3x or 3x | High uncertainty; translation to broader, less well-specified tasks is unclear |
| Haiku 5.5 system card estimate | 1.7x | Lower than METR, mainly because it excludes newer unreleased models | System card measure, not METR's |
| Opus 5.5 system card trend-break fit | 1.53x | 95% range 1.20x to 1.82x | Different measure from METR's total acceleration; the card itself says its internal measures did not show a sustained twofold acceleration attributable to AI |
| Share of current acceleration due to AI | About 80% | Substantial uncertainty; late-2025 acceleration likely influenced by non-AI factors | No decisive evidence; Anthropic staff did not agree among themselves; detailed attribution evidence redacted |
| Self-reported staff uplift from AI use | Up to 12x by October 2026 | Slack polls; METR calls this very loose evidence | Low-effort responses; individual uplift is not an acceleration rate |
| Acceleration not caused by faster training-compute scaling | Confident | METR states confidence directly | The convincing evidence behind this judgment was redacted |
| No significant bottlenecks in the next 12 months | None observed | METR is not confident in this assessment | Understanding changed several times as more detail arrived; the field may be missing important bottlenecks |
What METR actually did
From August 12 to October 29, 2026, METR staff performed a high-access evaluation at Anthropic to assess the degree to which AI is accelerating AI research and development. The team was small, 2.5 full-time equivalents, and the access was, in METR's word, unprecedented: internal data on pretraining, reinforcement learning, training compute in each domain, benchmark scores for internal models, AI speedup analyses, forecasts of a future model's metrics, and interviews with relevant staff, about fifteen of them. Anthropic took no payment from METR in the sense that METR reports it took no payment, and Anthropic provided laptops and free tokens for the work.
The main measure is AECI, Anthropic's version of the Epoch Capabilities Index, which turns benchmark test results into a single score. A single score built from many benchmarks is how the field now keeps score, and the leading index rewrote its own methodology in October, which is why the choice of index matters as much as the number it produces. METR dated each model snapshot by when most Anthropic employees could use it, which creates a common basis for released and unreleased systems, and the higher METR figure versus the Haiku 5.5 system card comes mainly from including three unreleased models trained since July. That inclusion is both the strength and the weakness of the headline number: it captures progress the public cannot see, and it cannot be checked by the public for exactly that reason.
The redaction problem
The most important sentence in the report is not a number. It is in the redaction summary at the top: Anthropic had the right to redact any non-public information, and it "redacted a majority of the evidence we found most informative for these conclusions." Anthropic described the redactions as aimed at removing confidential information that could accelerate other AI developers. METR says it was able to share its highest-level conclusions about risk freely, and that all redactions important to its conclusions are covered by descriptions in the report, but it also says the report "does not achieve transparency about evidence," because much of the redacted evidence had multiple plausible interpretations and the public cannot judge it, reconcile it with other sources, or come to its own conclusions.
The consequence is stated by METR itself: the public has to rely on METR's judgment much more than would be ideal, given the weight of the questions answered here. That applies to the most consequential findings. The evidence that convinced METR the acceleration was not driven by training compute was redacted. The detailed estimates for the cumulative and instantaneous effects of AI on the rate of progress, at various stages of the training pipeline, were redacted. Some of the key findings that most informed METR's conclusions about risk were redacted. A reader who wants to verify the 1.9x figure can verify the method and the confidence interval, but not the underlying data, and should say so rather than pretend otherwise.
What the numbers do and do not prove
The 1.9x figure is a statement about benchmark scores on verifiable tasks, and METR lists three ways it could overstate the truth. First, non-AI factors in Anthropic's research progress may deserve more weight, particularly for the late-2025 acceleration. Second, AECI focuses on tasks that are easy to grade automatically or with a language model, which may not represent broader capabilities. Third, Anthropic's early growth rate, from 2023 to 2025, may have been unusually low, which would make any later rate look faster by comparison. METR also lists three ways it could understate the truth: some growth rates have increased further recently, further automation of research tasks could accelerate progress again, and AECI growth has sometimes understated growth in real-world impact.
Two cautions deserve emphasis because they are easy to lose in the headline. The 12x self-reported uplift figure, from Slack polls in which staff vote with reactions on their estimated speedup from the most recent model, is not an acceleration rate, and METR says the responses reflect low-effort consideration. It shows that staff feel dramatically faster, not that capabilities are improving twelvefold. And the robustness range, 1.1x to 2.9x under alternative modeling choices, is wide enough to contain both a modest acceleration and a near-tripling. METR says it would be surprised by no acceleration or by acceleration above 3x, which is a real constraint, but the difference between 1.1x and 2.9x is the difference between a fast lab and a different era.
The threshold dispute
METR published a companion review alongside the assessment, of the automated research and development section of Anthropic's public August 2026 Risk Report, and it disagrees with the public report in a way that matters. Anthropic's Responsible Scaling Policy sets two capability thresholds for automated research and development in key domains. One is full substitution, models substituting for humans, which both sides agree has not been crossed. The other is the 2x Acceleration Threshold, defined in the policy as dramatic acceleration of the pace of AI progress for reasons that likely relate to the automation of AI research and development, and described in the Risk Report as an early warning rather than evidence of high risk. The August public report argued the threshold had not been crossed, on the grounds that capability metrics and leading indicators showed less than a twofold acceleration, with factors other than AI driving it.
METR's review agrees that catastrophic risks from the models as of July 15, 2026, were low, but says the public report's argument that the threshold had not been crossed is overstated, that it is plausible the threshold was crossed by July 15, and that the public report overall understates the level of acceleration that was occurring. METR also records the honest split: most relevant researchers at METR and Anthropic lean toward the threshold not having been crossed on July 15, but METR thinks it plausible it had been. The dispute is about a judgment call on an early warning, not about whether catastrophe had arrived, and both sides say so. That distinction is the point, and it is why the companion review is worth reading alongside the headline.
Attribution: assistance, not autonomy
The report's most quoted line, that perhaps 80 percent of the current acceleration is due to AI, carries the most uncertainty in the document. METR's judgment is that a majority of the acceleration today is likely attributable to Anthropic's extensive use of AI to do research and development, but that other causes likely played a large role in late 2025, that there was no consensus among Anthropic's own staff, and that no decisive evidence was found. Interviews and surveys are consistent with AI use causing a doubling in the pace of capabilities progress, with estimates centering around 2x, but the range was wide.
What METR was told, consistently, is that research is accelerated primarily through AI assistance rather than fully autonomous contributions, with some signs that fully autonomous contributions will play a larger role soon. That is the correct frame for the recursive self-improvement question. The report is evidence that AI is speeding up the humans who build AI, not evidence that AI has started building its successor alone. The one judgment METR states with confidence is a negative: the acceleration was not due to an acceleration in the scaling of training compute. The evidence behind that confidence was redacted, which is the report's central tension in miniature.
On bottlenecks, the report is careful in a way that should not be smoothed over. METR did not see evidence for significant bottlenecks that would prevent further capabilities progress due to limits on algorithmic progress, training compute, experiment compute, or data in the next 12 months, but its understanding of different bottlenecks changed several times as it received more detailed information, and it is not confident in its current assessment. The lack of observed bottlenecks or strong diminishing returns means, in METR's words, that there is a significant risk further capabilities progress could lead to even faster acceleration over the coming year. That is a statement about risk, not a prediction, and the report's own uncertainty is part of it.
What to watch next
Five questions separate a serious reading of this report from a headline. First, is the number total acceleration or AI-attributed acceleration, because the two are different measures and Anthropic's own system cards put the AI-attributed figure lower. Second, what was redacted behind the number, and does the report say so. Third, does the estimate carry a confidence interval and a robustness range, or is it a point estimate with the uncertainty stripped out. Fourth, what would falsify it: a second lab assessed the same way, a public benchmark trend that refuses to follow, or a system card that revises the figure down. Fifth, does the policy consequence follow: Anthropic's own framework treats a doubling as an early warning, and the company said in its November 2 securities filing that future models may imminently meet its definition of dramatically accelerated progress attributable to AI, which is the designation this report's evidence now informs. A lab that answers all five with numbers is doing the work. A report that answers them with adjectives is not, and this one, to its credit, mostly answers with numbers.
The METR assessment is the clearest evidence yet that AI is accelerating its own development, and it is also a demonstration of how hard that evidence is to verify when the data sits inside one company. The 1.9x figure is real, it is uncertain, and it is partly redacted, and all three of those things are true at once. The next assessment, whether from METR or from another evaluator with the same access, is the event that will tell us whether the doubling holds, and the audit table above is the baseline it will be judged against.
By ByteHerald Staff. METR, November 5, 2026, published "Independent Assessment of AI R&D Acceleration at Anthropic in October 2026," reporting an 11-week assessment from August 12 to October 29, 2026, with 2.5 full-time equivalents, finding benchmark scores growing 1.9 times faster since October 2025 with a 95 percent confidence interval of 1.5 to 2.8 times, a robustness range of 1.1 to 2.9 times, annual AECI growth rising from 15 to 29 points, and perhaps 80 percent of current acceleration attributable to AI, while stating Anthropic redacted a majority of the evidence METR found most informative. METR, November 5, 2026, published a companion review of the automated research and development section of Anthropic's August 2026 Risk Report, agreeing catastrophic risks from July 15 models were low but finding it plausible the 2x Acceleration Threshold had been crossed by July 15 and that the public report understates the acceleration occurring. Reuters, November 5, 2026, first reported the assessment, describing Anthropic's models improving nearly twice as fast with AI playing a bigger role, and noting METR's caution that AI was not yet capable of fully autonomous contributions to the most important training stages. An aistockwire explainer, November 2026, tabulated the system-card figures behind the assessment, including the Haiku 5.5 card at 1.7x and the Opus 5.5 trend-break fit at 1.53x with a 95 percent range of 1.20x to 1.82x.
Comments
Post a Comment