Who Grades the Graders? The AI Benchmark Credibility Audit

Every month, AI labs announce record benchmark scores. Almost none of them tell you who set the exam. ByteHerald audited four claims from October 2026 and found the same pattern each time: the headline number depends on the harness, the grader, or both. NVIDIA says its Nemotron Bolt ranks first on PinchBench while NVIDIA's own model card scores a rival 0.25 points higher. Google's Gemini 4 Argon scores 97.5 percent on ARC-AGI-3 with Google's provider adapter and 49.8 percent on the neutral standard harness. Meta's training-efficiency figure moved between its paper and its public telling. And a viral X post declared DeepSeek's model the winner of a cybersecurity benchmark its own authors built. The scores came from vendor-run harnesses, not independent grading.

The disclosure scorecard

One table, every claim in this article, graded on the five things that make a benchmark score trustworthy. "Harness" is the test rig the model ran on. "Independent replication" means someone other than the claimant reproduced the number.

Claim	Who ran the test	Harness disclosed	Cost disclosed	Independent replication	ByteHerald verdict
Gemini 4 Argon: 97.5% on ARC-AGI-3	ARC Prize, with Google's provider adapter	Yes, two harnesses	Yes, $4,800	Yes, ARC Prize re-ran it on the standard harness: 49.8%	The score is harness-dependent. ARC Prize publishing both numbers is the honest part.
Nemotron Bolt 74B: #1 on PinchBench	NVIDIA announcement	No scores in the announcement	No	NVIDIA's own model card shows Qwen3.6 at 88.07 vs Bolt 87.82	The #1 claim and the company's own table disagree.
Meta GEM: 20-25% training efficiency	Meta	Not disclosed	Not disclosed	Meta's own paper says 37%, as reported by Rundowns AI on Oct 18	Meta published two different efficiency figures with no reconciliation.
DeepSeek V4.1-Flash 1017 beats GPT-6 Sol on a cyber task	The benchmark's authors, via X post	Their own simulated-network harness	No	None	The benchmark's authors ran, scored, and announced their own test; this is a claim, not a verified result.

Case 1: the 48-point harness gap

ARC Prize's Gemini 4 Argon results are the cleanest demonstration in the industry of why the harness matters more than the model. Under the Provider Adapter harness, a setup using vendor-specific features like retained reasoning and context compaction, Argon scored 97.5 percent at a cost of about $4,800. Under ARC Prize's standard harness, the provider-neutral version that only lets a model carry forward notes it chooses to keep, the same model scored 49.8 percent at low reasoning effort and cost about $5,800, more money for half the score. At medium effort it scored 46.3 percent; at high effort, API problems stopped it from finishing all 55 games, so ARC Prize published no score.

GPT-6 Astra shows the same pattern: 99.9 percent with its provider adapter, 62.7 percent on the standard harness, at a cost of about $26,100. This is not a Google-specific phenomenon. It is the current state of frontier benchmarking: the headline number is a joint product of the model and the scaffolding around it.

Credit belongs where it is due. ARC Prize published both tables, with per-effort breakdowns, game-completion counts, and costs. That dual reporting is the gold standard, and it is what lets anyone see the gap. Independent context helps too: Epoch AI's Capabilities Index scored Argon at 165, tied with Claude Fable 5.1 and behind Claude Opus 5.5 at 167 and GPT-6 Astra at 166, noting that Argon trails rivals in math and software engineering even as it leads on the general puzzle-solving test. Without the harness disclosed, a benchmark score cannot be compared to anything. ARC Prize disclosed both harnesses, which is why the gap is visible at all.

Case 2: NVIDIA's #1 and its own table

NVIDIA's October announcement for the Nemotron 3.5 family says the Bolt 74B-A9B model ranks first among open models in its class on PinchBench, and second on BrowseComp. The announcement gives no scores and does not define the comparison class beyond "comparable open models." NVIDIA's own research page adds detail: Bolt is compressed from the 120B Super VL model to 74B total with 9B active parameters, quantized to about 2 bits per element, fits in 24GB of memory, and hits 84 tokens per second on the AIPerf SPEED-Bench coding test on an RTX Spark system. On public leaderboards among models with a similar memory footprint, NVIDIA states Bolt is #1 on PinchBench, #2 on WebArena, and #2 on BrowseComp.

The awkward part is in NVIDIA's own model card, as spotted by independent readers: it lists Qwen3.6-35B-A3B at 88.07 against Bolt's 87.82, a 0.25-point gap in the rival's favor, inside the same table. The qualifiers do the heavy lifting here: "comparable open models," "similar memory footprint." Those are not disclosed as explicit inclusion lists, so the reader cannot check who was excluded. The honest reading: Bolt is a genuinely impressive efficiency story, a 74B model squeezed into 24GB with competitive agentic scores, wrapped in a headline the company's own numbers do not quite support. We have graded vendor claims this way before, in our open-weight benchmark comparison: the score is data, the ranking is marketing.

Case 3: Meta's moving efficiency number

As reported by Rundowns AI on October 18, Meta puts the training efficiency of its GEM system at 20 to 25 percent in its public telling, while its own paper says 37 percent. We present this as reported: we have the secondary report, not the paper text in front of us, and efficiency definitions vary with what gets counted as training. The pattern is the point. When the same company ships two numbers for the same system with no reconciliation, the reader is grading on vibes. A serious efficiency claim names the denominator: 37 percent of what, measured how, against which baseline. Until then, both numbers are press.

Case 4: the self-graded X benchmark

The weakest disclosure of the four arrived as an X post on October 20: "We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task," tested with Sol via a trusted-access program, on "a cybersecurity benchmark based on simulated corporate networks" the authors are building. What is disclosed: the task family, simulated networks, the access path for the rival model. What is not: the number of tasks, the scoring rubric, the harness code, whether anyone else can run it, or whether the benchmark's own authors have any incentive beyond a good story. When the benchmark's own authors ran, scored, and announced the test, it is a claim, not a verified result. Treat it as a lead for independent testing, not as evidence.

Contamination: the other way grades get inflated

Harness choice is not the only lever. The other is contamination: training on the test. If a benchmark's questions appear in a model's training data, the score measures memorization, not capability. The honest labs disclose decontamination procedures and hold out private test sets; the quiet ones report the number and move on. Any claim that does not address contamination is incomplete by default, because the base rate of leaked benchmarks in this industry is high and everyone grading knows it.

The 5-point benchmark-reading checklist

Use this on every record-benchmark headline, including ours:

1. Who ran the test? The vendor, an independent evaluator, or the benchmark's own authors. Each has different incentives.
2. Is the harness disclosed? Standard versus provider adapter can move a score 48 points. No harness, no trust.
3. Is the cost disclosed? Compute spend is a proxy for effort. A $26,100 run and a $4,800 run are not the same achievement.
4. Was it independently replicated? One lab's number is a press release. Two labs' numbers are a finding.
5. Is the comparison class defined? "Comparable models" and "similar footprint" are doing quiet work. Ask for the list.

Scores without prices are half the story, which is why we also track what the models cost to run. Our open-versus-closed value analysis put 28 models on one dollars-and-cents scoreboard: open weights win value by roughly 60x while closed models hold the capability crown. The benchmark tells you what a model can do. The price tells you what it costs to find out. Demand both numbers, and the harness they were measured on, before you believe a leaderboard.

By ByteHerald Staff. ARC Prize figures via The Decoder (Epoch AI), OfficeChai, and Golden Era, October 13-19, 2026. NVIDIA figures from NVIDIA Research and its model card, cross-checked via CellCog. Meta GEM figures as reported by Rundowns AI, October 18, 2026. The X cybersecurity-benchmark claim from the October 20 radar window.