63% of AI Benchmarks Are Already Solved, Mollick Finds

Of 173 AI benchmarks cited by the major labs since 2023, 63 percent are now saturated, effectively solved, according to Ethan Mollick's SatBench count published on November 14, 2026, and new benchmarks are saturating faster than the old ones did. Mollick, a professor at the University of Pennsylvania's Wharton School who writes widely on AI at work, posted the finding on Bluesky with a blunt summary: "AI is getting better faster than our measurements are." The count is his own, built from benchmarks the labs themselves cite, and it puts a number on a problem the industry has talked around for two years: the tests we use to judge AI models are being finished faster than we can write new ones.

Editorial graphic: the headline The Benchmarks Are Solved, November 2026

What SatBench counts

SatBench measures how quickly major AI benchmarks become saturated, meaning effectively solved by AI, and it counts 173 benchmarks cited by the labs since 2023. Sixty-three percent of them are now saturated, which works out to about 109 of the 173, by ByteHerald arithmetic from Mollick's figures. The other 64 are still open, but Mollick adds that saturation of new benchmarks is also happening faster, which means the window in which a fresh test can tell models apart is shrinking. The methodology is Mollick's own: the benchmark list is the set the labs cite, the saturation judgment is his, and the count is presented as a quick measurement, not a peer-reviewed study. He says so himself in the post: "I threw together SatBench."

That informality is part of the point, and it is also the limit. SatBench is one researcher's count, and no public SatBench site with the underlying table could be found at the time of writing, so the per-benchmark verdicts cannot be checked independently from the post alone. What can be checked is the direction, and the direction matches what the labs' own scoreboards show: benchmark after benchmark that once separated the best models now shows the leading systems clustered at the top, and the tests that still separate them are the newest and hardest ones.

The replacement wave is much harder

The clearest evidence that the replacements are harder comes from a new benchmark published two weeks earlier. Co-Design Bench, a preprint posted on arXiv on October 26, 2026 by researchers at Fudan University, Alibaba Token Hub and Alibaba Group, Zhejiang University, the Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University, builds 273 tasks from 37,463 deduplicated queries across 26,314 real user coding sessions, and grades them against 2,331 hidden expert-authored acceptance criteria. Instead of asking a model to write code from scratch, it asks an agent to revise a working application in response to a real user's request, with the original app, its context and its attachments in hand.

Two-panel chart: 109 of 173 benchmarks saturated in SatBench, and the strongest model on Co-Design Bench meets 69.9 percent of criteria but fully passes only 26.4 percent of tasks

The gap between partial and complete success is the finding. Under a shared agentic harness, the strongest of ten model endpoints, GPT-6 Astra, passes an average of 69.9 percent of criteria per task but fully satisfies only 26.4 percent of tasks. On the repository track, where edits span multiple files and change types, the strongest model fully passes only 17.9 percent of tasks. The paper's own framing is that substantial coverage of individual requirements can coexist with limited complete-task success, which is a precise description of why a 99 percent score on an old benchmark and a useful coding agent are different things.

The same paper shows how much the test conditions matter. On the same 150 WebCompass-edit tasks, switching one model from single-shot to agentic execution raises mean graded feature completion from 73.6 percent to 95.1 percent, and another leading model reaches 99.4 percent with agentic execution. A test that looks nearly solved under one execution regime can look hard under another, which means a benchmark score is not a property of the model alone. It is a property of the model, the harness, and the grading, and changing any of the three moves the number.

How to read a benchmark score now

Benchmarks die for three ordinary reasons, and all three are at work here. First, capability growth: models get better, so a test written for last year's models stops separating this year's. Second, training exposure: once a test is public, it can leak into training data, and a model can score well by remembering rather than reasoning. Third, Goodhart's law: once a score becomes the target, labs optimize for it, and the test stops measuring what it was built to measure. Saturation is what those three forces look like when they finish the job, and Mollick's 63 percent says they have finished it for most of the tests the labs still cite.

For anyone buying or choosing a model, the practical rule follows from the Co-Design numbers. A headline score on a saturated benchmark proves almost nothing, because the test no longer separates good from great. The numbers that still carry information are the ones from tests the model has not seen, that require finishing a whole task rather than most of it, and that mirror the work the buyer actually does. Ask for the full-task pass rate, not the average criterion score, and ask whether the test was run with the same harness the buyer will use. A model that meets 69.9 percent of requirements and finishes 26.4 percent of tasks is not a 70 percent model. It is a model that leaves three quarters of real jobs unfinished, and that is the number that matters. Our earlier audit of benchmark credibility, Who Grades the Graders?, sets out the five questions to ask before trusting any AI score, and saturation is the first reason those questions exist.

What Mollick's count does not prove

Three limits belong in the open. SatBench is a single researcher's count from a social media post, and without the underlying table the 63 percent cannot be audited benchmark by benchmark. Co-Design Bench is a preprint, not peer-reviewed, and its grading is done by AI judges under a judge-then-refute protocol, with a separate language model challenging positive verdicts and audits requiring screenshots for visual claims. That design is stronger than a single automated judge, but it is still machine grading, and the authors say so. And saturation is not the same as capability: a benchmark can be solved because the test is narrow, because it leaked, or because the models genuinely mastered the skill, and SatBench does not separate those causes. What it does establish is that the measurement system is running behind the thing it measures, and that the tests replacing it are, for now, much harder than the ones they replace.

By ByteHerald Staff. Ethan Mollick posted on Bluesky on November 14, 2026, that he built SatBench, which measures how quickly major AI benchmarks become saturated, and that of 173 benchmarks cited by the labs since 2023, 63 percent are now saturated, and that saturation of new benchmarks is happening faster. Co-Design Bench, a preprint posted on arXiv on October 26, 2026, by researchers at Fudan University, Alibaba Token Hub and Alibaba Group, Zhejiang University, the Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University, introduces a benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions, with 2,331 hidden expert-authored acceptance criteria, and reports that the strongest of ten model endpoints passes an average of 69.9 percent of criteria per task but fully satisfies only 26.4 percent of tasks. The same paper reports that on 150 WebCompass-edit tasks, switching one model from single-shot to agentic execution raises mean graded feature completion from 73.6 percent to 95.1 percent, and that another leading model reaches 99.4 percent with agentic execution.

BH
Written by