An AI agent just finished 41.3 percent of 240 real freelance projects at a quality a client would approve. The result comes from the Remote Labor Index, a benchmark built by Scale Labs and the Center for AI Safety from work real freelancers were paid to do on Upwork, and it was set by Anthropic's Claude Fable 5.2, released November 9, 2026. Twelve months earlier, the leading system scored 2.5 percent.
Claude Fable 5.2 completed 41.3 percent of 240 real freelance projects on the Remote Labor Index to a client-acceptable standard, about twice the runner-up's score, Claude Opus 5.5 at 21.3 percent, and up from 2.5 percent for the best agent when the test launched in October 2025.
The scoreboard
The leaderboard is the whole story in six rows. Fable 5.2 leads at 41.30 percent, nearly twice the second-place score, and the gap between first and second is larger than the gap between second and sixth. The board below is the complete published leaderboard as of November 10, 2026, with the launch baseline included, because the baseline is what makes the top number mean something.
| Rank | Model | Automation rate | Projects finished (of 240) |
| 1 | Claude Fable 5.2 | 41.3% | 99 |
| 2 | Claude Opus 5.5 | 21.3% | 51 |
| 3 | GPT-6 Astra | 20.8% | 50 |
| 4 | Claude Fable 5.1 | 17.9% | 43 |
| 5 | Gemini Argon 4 | 17.1% | 41 |
| 6 | Claude Fable 5 | 15.8% | 38 |
| Baseline | Best agent at launch (Manus, Oct 2025) | 2.5% | 6 |
The project counts are ByteHerald arithmetic from the published rates and the 240-project set, and they are labeled as such. One setting matters for reading the table honestly: Scale AI ran Fable 5.2 at high effort with a 200,000-token context window and 32,000 tokens per response, while Anthropic's own launch table reports its results at maximum effort. Different effort settings change the result, so the 41.3 percent is Scale's number from Scale's run, not a number every user will reproduce at default settings. Models from the previous generation still sit below 3 percent, a measure of how far the newest frontier systems have moved from the last generation.
What the test actually measures
The test starts from a different question than most benchmarks. Instead of asking whether a model can solve a puzzle or answer a question, it asks whether an agent can finish a job a client paid a freelancer to do. Scale Labs and the Center for AI Safety gathered 240 such jobs from Upwork, spread across 23 kinds of work and supplied by 358 freelancers whose identities were verified. The combined value of the work is about 144,000 dollars. The jobs are substantial: a typical one took a freelancer close to 29 hours and paid around 632 dollars, while the middle project in the set took 11.5 hours and paid 200 dollars. The agents do the work in a virtual machine stocked with the tools freelancers actually use, Blender for 3D, GIMP for images, Audacity for audio, alongside AI tools that generate images, speech, and video.
The score that matters is the automation rate, and the bar for passing is deliberately practical. Three separate human reviewers look at each finished project, and it passes only when they agree the result is good enough for a client to approve. The reviewers agree with each other 94.4 percent of the time, which is high enough to trust the score and low enough to show the judgments are real. The paper behind the benchmark, written by Mantas Mazeika and 46 co-authors and submitted October 30, 2025, says the point is to put numbers under the debate about AI and work, so that claims about automation can be checked against evidence instead of argued from anecdotes. Measuring paid work instead of test questions is what gives the result its weight.
The doubling cadence
What matters most is how fast the number is moving, and it moves fast enough to check with arithmetic. The test opened in October 2025 with Manus, the best system then, at 2.5 percent. Fable 5 scored 16 percent in July 2026. Fable 5.1 reached 17.9 percent. Fable 5.2 now sits at 41.3 percent. The jump from 2.5 percent to 41.3 percent in about 13 months is about 16.5 times, which is about four doublings, or about one doubling every three months, by ByteHerald arithmetic from the two endpoints. Scale AI says the same thing in its own words: the automation rate has doubled about every three months. The two readings agree, and the agreement is the point. A rate that doubles every three months does not need a forecast to be startling. It needs a check on whether the next doubling is possible, and the test itself sets the limit on that check, because the grading is done by humans.
Where the gains came from
The improvement is not spread evenly across the work. Scale Labs reports that the largest gains came where the deliverable is a single file that can be opened and judged on its own: audio and music edits, architecture and interior designs, 3D models, and long projects that earlier models could not finish at all. Roughly 50 of the 99 were projects no agent had ever completed, which is the clearest sign that something changed between generations rather than everything improving a little. The two categories where the model still struggles, industrial product design and animation, are the ones where the deliverable has to stay consistent across views, parts, and motion. Finishing a file is one skill. Holding a design together is another, and the gap between them is where the frontier still sits.
Why most jobs still fail
Even at 41 percent, most of the work still defeats the model, and the pattern of failure is instructive. The single most common problem is work that looks like a beginner made it, followed by deliveries that are simply unfinished: a video that stops halfway, files that never arrived. Then come the technical failures, files saved in the wrong format or corrupted, and designs that fall apart when you look at them twice, a brand kit where the colors drift from one page to the next. The reviewers also found that automated judges are easy to fool: a deliverable can look polished and still be broken underneath, and an AI grader will pass it. That is why the grading is still done by human experts, and why running the test is slow and expensive. Every new model has to be scored by people opening files and comparing them with the freelancer's original work, and there are only 240 projects to grade. The doubling cadence says the next jump is coming. The grading bottleneck says it will take time to prove it.
What 41.3 percent does not mean
The question everyone asks first is whether people are losing jobs to AI, and the test does not answer it. It measures project completion, not employment. Client calls, physical work, and results that take months to show up, like search rankings, are all outside the test's scope, so the 240 projects are a sample of remote work, not a picture of the whole market. A rate of 41.3 percent across 240 selected projects is not 41.3 percent of freelance jobs, and it is not 41.3 percent of anyone's working week. What the test does say is narrower and more useful: for well-defined projects with a clear deliverable, the best current agent finishes about two in five at a quality a client would approve, and the work it still fails is the work that needs a client on the phone, a relationship, accountability for the result, or a judgment that no file can carry. The jobs that survive, for now, are the ones the test does not cover. That is not a consolation. It is a description of where the frontier actually sits.
The launch that produced this number is worth reading alongside the test: Fable 5.2 was released November 9, 2026, at 8 dollars per million input tokens and 40 dollars per million output tokens, 20 percent below Fable 5.1, and the Remote Labor Index was one row of fourteen in Anthropic's benchmark table. The automation question is also being asked inside the labs, where we audited the claims about AI speeding up AI research itself, and the two measurements answer different questions: one asks how fast AI builds AI, and this one asks how much paid work AI can finish. And the index problem is the same one we mapped in the Artificial Analysis Intelligence Index: a single number hides the method, the effort setting, and the choice of what counts, and the only honest way to use it is to read the method first.
How to use this number
The number is useful as a guide to where agents work and where they do not, not as a verdict on the future of work. First, check the deliverable yourself, file by file, because the test's own reviewers learned that a polished surface can hide a broken result, and an automated check will not catch it. Second, look at the spread between models before paying for one: 41.3 percent against 21.3 percent means the leader finishes roughly twice as many projects as the runner-up, a difference worth paying for when the work matches, and worth ignoring when it does not. Third, start with bounded jobs that end in a file, a podcast edit, a room layout draft, a first pass at a 3D model, because that is where the gains are. And keep the work that involves a client, a judgment call, or a result that only shows up months later in human hands, because that is the work the test does not cover and the work agents still fail at most.
What to watch
First, the next leaderboard update, because the doubling cadence predicts the next number and the human-grading bottleneck predicts it will arrive slowly. Second, whether other labs submit their models, since the board currently has six rows and the comparison is only as good as the field. Third, whether the test grows beyond 240 projects, because a bigger set would make the rate harder to move and easier to trust. And fourth, whether the weak spots close: industrial product design and animation are the two categories where the frontier is still visibly human, and the day an agent clears them is the day the test stops being a story about creative chores and becomes a story about the whole job.
By ByteHerald Staff. Scale Labs and the Center for AI Safety built the Remote Labor Index from 240 real Upwork projects collected from 358 verified freelancers across 23 work categories, worth about 144,000 dollars, with an average project taking almost 29 hours and paying about 632 dollars, and three independent evaluators voting on each project with 94.4 percent agreement, as reported by AI Parabellum on November 10, 2026, from the Scale Labs leaderboard. The paper behind the test, by Mantas Mazeika and 46 co-authors, was submitted to arXiv on October 30, 2025, as arXiv 2510.26787, and states that the highest-performing agent at the time achieved an automation rate of 2.5 percent. Anthropic announced Claude Fable 5.2 on November 9, 2026, and its system card reports that Scale AI evaluated Fable 5.2 on the Remote Labor Index at 41.3 percent, against 21.3 percent for Claude Opus 5.5 and 20.8 percent for GPT-6 Astra, as reported by AI Weekly on November 10, 2026. Scale AI reported on November 10, 2026, via X, that the automation rate on the Remote Labor Index has doubled about every three months, from 2.5 percent a year ago to 41.3 percent for Fable 5.2.
Comments
Post a Comment