A benchmark posted to arXiv on November 5, 2026, puts a number on the gap between proving what is already known and discovering what is not. On TCSExam, OpenAI's GPT-6 Astra produced complete, machine-checked Lean proofs for 70 percent of research-level theorems in theoretical computer science, and solved none of the 30 open conjectures it was handed. The same model leads the benchmark's full suite by 14.5 points, and the same model scored zero where new mathematics was actually required. That split is the most honest measurement of AI in science published this year, because the checker is a machine, not a panel of judges.
TCSExam, posted to arXiv on November 5, 2026, tested eight AI agents on 100 expert-reviewed computer science tasks: GPT-6 Astra leads the full suite with 71.9 points, proved 70.0 percent of the theorems, and resolved none of the 30 open conjectures on its own.
What TCSExam tests
TCSExam, short for Evaluating Frontier LLMs on the Full Range of TCS Research, comes from researchers at Tsinghua University, Moonshot AI and the Shanghai Qizhi Institute, including Turing Award winner Andrew Chi-Chih Yao. It is built on a simple idea: theoretical computer science is a natural test for AI research because proofs can be machine-checked and records can be beaten. The benchmark contains 100 expert-reviewed tasks in four tracks. Twenty theorems must be proved in Lean, the proof assistant whose kernel accepts or rejects every step. Thirty solver-engineering problems are judged on hidden instances. Twenty constructions must beat existing human records. And thirty open conjectures are handed to the agents as direct attempts to resolve them. Every submission is checked by the Lean kernel or an executable task checker, so there is no partial credit from a friendly grader. The TCSExam Index combines the four tracks with weights of 30 points each for theorem proving, solver engineering and constructions, and 10 for open conjectures. Construction scores above 100 percent are not capped, so the Index can exceed 100. A 20-task Lite suite, five tasks per track, adds Anthropic's Opus 5.5, which was not run on the full suite. The paper reports that the Full and Lite rankings are strongly aligned across the eight shared agents, with a Spearman correlation of 0.93, which is the authors' evidence that the smaller suite still discriminates between models.
The scoreboard
The table below is the paper's Table 2, as reported from the full text, with the theorem-proving and conjecture-track figures the paper publishes for the three agents it names in detail. For the other five, the paper reports only that their theorem-proving success reached at most 11.7 percent, so the table says exactly that rather than inventing a number.
| Agent | TCSExam Index | Theorem proving | Open conjectures |
| GPT-6 Astra | 71.9 | 70.0% (14 of 20) | 0.0% |
| Kimi K3.1 Pro | 57.4 | 38.3% | 1.1% |
| Kimi K3 | 46.8 | 11.7% or less | Not reported in the sources reviewed |
| DeepSeek V4.1 Flash 1017 | 46.0 | 6.7% | 5.6% |
| GLM-5.3 | 45.5 | 11.7% or less | Not reported in the sources reviewed |
| Qwen3.8 Max | 44.8 | 11.7% or less | Not reported in the sources reviewed |
| Grok 4.6 | 40.5 | 11.7% or less | Not reported in the sources reviewed |
| DeepSeek V4.1 Flash 0910 | 40.4 | 11.7% or less | Not reported in the sources reviewed |
On the Lite suite, Anthropic's Opus 5.5 leads with 72.3 points, completing four of five theorem tasks against three for Astra, which scores 67.0 on Lite. Opus 5.5 was evaluated only on the 20-task Lite suite, so its full-suite score is not known, and the Lite lead does not mean it would lead the full 100 tasks.
The 70-to-0 gap, in plain language
The gap is the whole story, and it needs no jargon to understand. Proving a theorem means taking a result mathematicians already know is true and writing a formal argument the Lean kernel accepts, step by step, with no gaps. It is hard, and Astra does it 70 percent of the time on research-level problems. Solving an open conjecture means producing a result nobody knows yet, a proof or a counterexample for a question that has resisted human mathematicians, sometimes for decades. On that track, Astra scored zero, and so did every other agent working on its own. The difference is the difference between checking and discovering. A model that can formalize known mathematics is a powerful research assistant. A model that can discover new mathematics would be a scientist. TCSExam is the first benchmark to put both jobs on the same scoreboard, and the scoreboard says the first job is mostly solved at the frontier and the second is not solved at all.
The two conjectures that did get solved
Two conjecture tasks did receive accepted submissions, and the paper is careful about what that means. During network-enabled runs, Kimi K3.1 Pro and DeepSeek V4.1 Flash 1017 retrieved counterexamples and Lean proofs that OpenAI had already published for two conjectures, Ryser and MIRUP, and adapted them. Those are the only accepted conjecture submissions. The other 28 conjecture tasks have no accepted complete solution, in the paper's words. The conjecture-track scores in the table, 1.1 percent for Kimi K3.1 Pro and 5.6 percent for DeepSeek V4.1 Flash 1017, reflect those adapted submissions, not independent discoveries. Read honestly, the conjecture column is 0 percent for every agent that had to do the work itself. That is not a failure of the benchmark. It is the benchmark working exactly as designed: it separates retrieving a published answer from producing a new one, and it caught the difference.
The one genuine new result
The paper does report one piece of new mathematics. While attempting one conjecture, GPT-6 Astra proved the Alon-Jaeger-Tarsi conjecture over the field with five elements for every dimension up to nine, extending earlier results to n equals 9, with independent Lean verification. The authors list four further verified special cases and bounds from open-conjecture attempts. This is a real result, not a retrieval: a special case of an open conjecture, proved and machine-checked, extending what humans had established. It is also a special case, not the conjecture itself, which remains open. The honest way to describe it is that Astra advanced the frontier by one verified step on one problem, while the conjecture track as a whole stayed at zero. That single verified step is more informative than the 70 percent theorem score, because it is the only place in the benchmark where a model did something no human had done before, and it shows what the ceiling looks like right now: narrow, verified, incremental progress on open problems, not breakthroughs.
The caveats that change the reading
Four caveats change how the scoreboard should be read, and the paper discloses all of them. First, Moonshot AI co-authored the paper, and its Kimi models rank second and third on the full suite, which is a conflict of interest to note, not a reason to dismiss the result, since the scores are machine-checked and the paper's methods are public. Second, the paper has not been peer-reviewed and has not been independently replicated, so the numbers are the authors' measurements until another group reproduces them. Third, the agents did not run under identical conditions: GPT-6 Astra ran in Codex with one attempt per task, while the Kimi models ran in Kimi Code with three attempts per task, according to the paper's Table 9, so part of the gap between first and second place is attempt budget and harness, not model capability alone. Fourth, the Index itself has a quirk: construction scores above 100 percent are not capped, and Astra scored 104.9 percent on constructions by beating human records, which lifts its Index above what a capped score would give. None of these caveats erases the 70-to-0 gap. They bound it: the gap is real, the exact size of the lead is not, and the conjecture column is the number to watch in the next edition.
The week that made this a race
TCSExam landed in a crowded week for AI in science, and the context matters. In August 2026, OpenAI released a 249-page manuscript claiming an internal model, Astra, had solved ten long-standing open problems in mathematics and theoretical computer science, backed by machine-checkable Lean 4 formal proofs with a sorry count of zero, meaning no step was left unproven or assumed. None of the ten results had cleared traditional peer review as of September 2026, and a credit dispute among mathematicians remained unresolved. On November 6, Physical Superintelligence announced that instances of a non-public Anthropic model had produced a Lean-checked proof of the Aaronson-Ambainis conjecture in three days, also not peer-reviewed. We covered that proof and the Anthropic study behind it in our science pillar this week: AI could do 27 to 52 percent of a scientist's job, and the study behind the number. Google DeepMind's Pushmeet Kohli wrote the same week that DeepMind is seeing a phase transition in the pace of scientific discovery. And Claude has proved Fermat's Last Theorem in Lean, according to FourWeekMBA's connecting coverage of the week. The pattern is that every lab now has a Lean-checked result to show, and none of them has a peer-reviewed one. TCSExam is the attempt to replace that parade of announcements with a single scoreboard, and its first edition says the announcements were measuring the easy half of the job.
How to read the next benchmark claim
Five questions, applied to any benchmark claim that arrives next. We applied the same test to a general leaderboard in November, when Intelligence Index v5 rewrote the AI leaderboard, and the methodology questions were the same. First, is it machine-checked, or judged by a panel, a model, or the authors themselves: Lean accepts or rejects a proof regardless of who submitted it, and that is the standard. Second, who co-authored the benchmark, and do their models rank near the top: Moonshot's second and third place is disclosed in the paper, and the next benchmark should be read with the same check. Third, did every agent run in the same harness with the same number of attempts: Astra's one attempt against Kimi's three is the difference between a model comparison and a system comparison. Fourth, has it been peer-reviewed or independently replicated: a preprint is a claim, not a result, until someone else reproduces it. Fifth, what does the conjecture track say: a benchmark that only tests known results can be gamed by memorization and retrieval, and the open-problem track is the only part that cannot be. A benchmark that passes all five is worth citing. TCSExam passes the first and the fifth, discloses its failure on the third, and is still waiting on the second and the fourth. That is a fair grade, and it is the grade to keep in mind when the next leaderboard arrives.
By ByteHerald Staff. Luo, Liu, Yuan, Yang and Yao posted TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research to arXiv on November 5, 2026, as arXiv 2611.07962, reporting 100 expert-reviewed tasks in four tracks, GPT-6 Astra leading the full suite with 71.9 points, and Opus 5.5 leading the 20-task Lite suite with 72.3 points. FourWeekMBA reported on November 8, 2026, after reading the paper in full, that Astra proved 70.0 percent of the research theorems and resolved none of the 30 open conjectures, that Kimi K3.1 Pro came second on theorem proving at 38.3 percent, and that the only accepted conjecture submissions adapted counterexamples and proofs OpenAI had already published. CryptoBriefing reported on November 8, 2026, that the benchmark contains 100 tasks across theorem proving, solver engineering, constructions and open conjectures, and that the paper has not gone through formal peer review. OpenAI published a 249-page manuscript in August 2026 claiming its Astra model solved ten long-standing open problems with machine-checkable Lean 4 proofs and a sorry count of zero, with none of the ten results peer-reviewed as of September 2026, as reported by Tech Insider.
Comments
Post a Comment