Agent benchmarks decide which coding assistant, computer-use model, or customer-service agent a company buys, and the numbers look precise. They are not. The seven benchmarks the industry quotes most use different tasks, different harnesses, and different scoring rules, and in one case a grading bug counts an empty answer as a pass.
We audited the seven, not by re-running them but by reading what their makers and their critics disclose: who runs each test, what it measures, how it handles contamination, who reports the score, and where the number is known to distort. The result is the scorecard below. The pattern is consistent: the score measures the harness as much as the model, and the headline number is the least informative part.
The scorecard
| Benchmark | Who runs it | Tasks | What it measures | Contamination control | Who reports the score | Known distortion |
| SWE-bench Verified | Princeton, human-screened with OpenAI | 500 GitHub issues, Python | Real issue resolution | None; all 500 predate training cutoffs | Vendors and the official board | Saturated: official board 79.2 percent vs vendor-reported 95.0; audits found 32.67 percent leakage and 31.08 percent weak tests |
| SWE-bench Pro | Scale AI | 1,865 tasks, 731 public | Large-repo engineering, 4 languages | Fresh repos, larger diffs, 10-line minimum | Scale AI | New; the public set is the comparable slice |
| FrontierCode 2.0 | Cognition (independent); reported by Anthropic | Agentic coding tasks (count not disclosed) | Agentic coding | Not disclosed | Vendor (Anthropic) | Effort differs by model (Xhigh vs Medium); vendor's own model leads |
| OSWorld 2.1 | OSWorld team; scores reported by Tencent | Desktop operating-system tasks | GUI computer use | Not disclosed | Tencent-reported | Partial vs binary scoring changes the order (Ember 69.5 partial, 33.8 binary) |
| Terminal-Bench 4.0 | Terminal-Bench team | Terminal tasks | Command-line agent work | Not disclosed | Tencent-reported | Measurer gap of 1.1 points (Kimi 65.7 vs 66.8) |
| tau-bench / Tau2 | Sierra | Customer-service dialogues (retail, airline, telecom) | Tool use plus policy following | Simulated users, written policies | Sierra and vendors | pass^k is strict; a grading bug counted empty answers as passes in some conditions |
| GAIA | Meta, Hugging Face, AutoGPT | 466 questions | General assistant, web and tools | Human-designed questions | HAL leaderboard and vendors | The wrapper is worth about 30 points (bare 44.8 vs scaffolded 74.6) |
Three things to notice before reading any single number. First, only two of the seven publish a contamination control you can check. Second, three of the seven are reported by the same company that built the model being graded. Third, the task counts range from 466 questions to 1,865 engineering tasks, so a one-point gap on one benchmark and a one-point gap on another are not the same size of claim.
One benchmark, two numbers
SWE-bench Verified is the oldest and most quoted coding-agent test, and it is the one whose number has drifted furthest from what it claims to measure. It is a human-screened subset of 500 tasks drawn from the original SWE-bench set of 2,294 real GitHub issues across 12 popular Python repositories, validated by human annotators working with OpenAI in August 2024. The model is given a real bug report and the surrounding codebase, and it passes only if its patch makes the repository's own hidden tests pass. OpenAI stopped publishing Verified scores in February 2026, saying all 500 tasks come from public Python repositories that predate every frontier model's training cutoff, so the test may measure recall rather than problem solving. The official leaderboard now defaults to a bash-only setting where every model runs the same mini-SWE-agent environment, and that board tops out at 79.20 percent, set by Claude 4.5 Opus at Medium effort in December 2025. Vendor-reported numbers reach 95.0 percent, claimed for Claude Fable 5 in August 2026. Same test, two numbers, 15.8 points apart, and the gap is harness quality, effort settings, and contamination, not model capability.
The audits put numbers on the drift. A SWE-Bench+ study screened the patches a GPT-4 agent produced on issues the benchmark marked as resolved and found 32.67 percent of the successful patches involved solution leakage, where the fix or something close to it was already stated in the issue report or the comment thread, and a further 31.08 percent passed only because the accompanying tests were too weak to catch an incorrect implementation. Filtering those cases out dropped the measured resolution rate for that model and scaffold from 12.47 percent to 3.97 percent. A second study, using a differential testing tool called PatchDiff, estimated that 11.0 percent of patches marked as resolving an issue were in fact behaviorally incorrect, inflating reported resolution rates by roughly 6.4 percentage points, and manual review of a 77-patch sample confirmed 28.6 percent were certainly wrong despite being scored as passes. A December 2025 paper tested the contamination question directly: models performed three times better on SWE-bench Verified than on comparable fresh benchmarks, and six times better at finding the edited files with only the issue text as context, a gap that points to the tasks having been seen in training. The same paper notes 94 percent of the instances were created before the training cutoffs of the most common models. Small tasks make the problem worse: 161 of the 500 tasks are one-to-two-line changes, with an average of 11 lines changed and roughly one file touched per task. That is a spot-the-difference exercise, not software engineering, and it is the reason Scale AI built SWE-bench Pro, with 1,865 tasks across 41 actively maintained repositories in four languages, where a task averages 107.4 changed lines across 4.1 files and every task is at least 10 lines. We audited model benchmarks in October and found the same pattern: disclosure is the difference between a measurement and a press release. Verified discloses the least and gets quoted the most.
The wrapper is worth 30 points
GAIA, the general-assistant benchmark from Meta, Hugging Face, and the AutoGPT team, asks 466 questions that are conceptually simple for a human but require several correct steps, chained together without error, with web browsing and tool use. At publication, human annotators reached 92 percent while GPT-4 with plugins reached 15 percent. The current numbers show the gap that matters most in agent evaluation: on the HAL leaderboard, a scaffolded agent running Claude Sonnet 4.5 scores 74.6 percent, while a bare model, GPT-5 Mini with no agent stack, scores 44.8 percent. The 30-point difference is the wrapper, the tools, the memory, and the retries, not the model. A startup picking the right framework can beat another running a stronger model with a weaker wrapper, and a buyer who reads only the headline score cannot tell which one they are buying.
The harness is part of the score everywhere you look. FrontierCode 2.0, the agentic coding test Cognition runs independently and Anthropic reported with Fable 5.2, put Fable at 49.1 percent against 45.5 for Opus 5.5, 42.8 for GPT-6 Astra, and 41.1 for Gemini 4 Carbon, but the effort levels differed, Xhigh for Fable and Fable 5.1 against Medium for Gemini, and effort changes the result. Our computer-use scoreboard from November 5 carries the OSWorld 2.1 and Terminal-Bench 4.0 numbers, including the 1.1-point gap between two independent runs of the same Kimi model, which is the noise floor for the whole table. OSWorld's partial-credit scoring changes the order too: Muse Ember scores 69.5 on partial credit and 33.8 on binary pass-fail, so whether the model looks competitive depends on which column you read. None of this makes the benchmarks useless. It makes the single number useless without the harness, the effort, and the scoring rule attached to it.
The reliability test tau-bench gets right
tau-bench, built by Sierra, measures the thing most agent benchmarks ignore: whether the agent follows the rules while doing the job. An agent holds a realistic multi-turn conversation with a simulated user, calls tools against a database, and must follow a written domain policy, and it is graded by comparing the final database state to the annotated goal. The metric it introduced, pass^k, counts a task as solved only if all k independent trials succeed, rather than pass@k, which counts it if any one of k attempts succeeds. A strong function-calling agent built on GPT-4o succeeded on fewer than half its tasks, and pass^8 fell below 25 percent in the retail domain, worse than independence would predict, which means failures cluster rather than spreading randomly. Tau2, the April 2026 update, added airline and telecom domains and voice full-duplex measurement; the top pass^4 rates were 0.765 in airline, about 0.71 in retail, and about 0.62 in telecom, the hardest domain because of dependency chains. An agent with pass@1 of 0.85 but pass^4 of 0.50 fails roughly one run in six (about 15 to 16 percent of single runs), which is undeployable in a customer-facing flow, and that is the number a buyer actually needs. The benchmark is not clean either: an audit of widely used agentic benchmarks found that flaws in task setup and reward design distort the reported scores, and that tau-bench's grading logic counts an empty agent response as a success in some conditions. The difference is that tau-bench's flaw is documented and its metric is honest about reliability, while the benchmarks that report a single optimistic percentage hide both.
The five-question test
Read any agent benchmark score with these five questions, in order, and do not trust a number that cannot answer all of them. First, who ran it, and who paid for it: a vendor grading its own model on its own test is a claim, not a measurement. Second, was the test set public before the model was trained: if the answer is yes, or unknown, the score may be recall. Third, did every model run on the same harness, at the same effort, with the same tools: if not, the gap is scaffolding, not capability. Fourth, one attempt, or the best of many: pass@k flatters, pass^k tells the truth about reliability. Fifth, does the score measure your job, or someone else's: a coding benchmark says nothing about customer service, and a computer-use score says nothing about policy following. A benchmark that cannot answer all five is not a measurement. It is a claim, and claims need a second source.
By ByteHerald Staff. Solangle reported on August 19, 2026, that SWE-bench Verified is saturated, that OpenAI stopped publishing scores in February 2026 citing contamination, and that the official bash-only board tops out at 79.20 percent against vendor-reported figures reaching 95.0 percent. EduFabTech reported that a SWE-Bench+ audit found 32.67 percent of successful patches involved solution leakage and 31.08 percent passed only on weak tests, and that a 2025 PatchDiff study found 11.0 percent of patches marked as resolved were behaviorally incorrect. A December 2025 arXiv paper found models performed three times better on SWE-bench Verified than on comparable fresh benchmarks and six times better at locating edited files with minimal context. Anthropic announced on November 9, 2026, that Claude Fable 5.2 scored 49.1 percent on FrontierCode 2.0 against 45.5 percent for Claude Opus 5.5, 42.8 percent for GPT-6 Astra, and 41.1 percent for Gemini 4 Carbon. Prefactor reported on October 28, 2026, that tau-bench measures tool use and policy following with the pass^k reliability metric, and analyses of agent benchmarks by Anhtu and Absolute Digital Publishers reported that GAIA has 466 questions and that a scaffolded agent scored 74.6 percent against 44.8 percent for a bare model.
Comments
Post a Comment