A new Harvard study of 718 firms found that AI coding agents boosted code output by 30%, and shipped zero additional software. The bottleneck moved from writing code to reviewing it. Here is what the numbers actually say, and what they don't.
The study
"Artificial Intelligence in the Firm: Bottlenecks in Software Production, " by Harvard researchers Fiona Chen and James Stratton, is a working paper (version dated August 4, 2026) built on data from the Jellyfish engineering-analytics platform: roughly 300 million work events from about 726, 000 workers at 718 software firms, from January 2021 through March 2026. Using a staggered difference-in-differences design, comparing firms as they adopted AI tools at different times, it estimates what changed after adoption.
The headline split: AI coding assistants (autocomplete-style tools) raised lines of code by 12%, with only the effect on commits reaching statistical significance. Fully agentic tools had a much larger effect, lines of code up 30%, commits up 20%, pull requests up 23%, all statistically significant. But neither category produced a measurable change in resolved Jira issues or completed epics, the study's proxies for shipped software, or in employment.
The paper traces the gap to review. After agent adoption, the average time to review a pull request rose 49%, the share of pull requests sent back for changes nearly doubled, comments per pull request rose 35%, and firms shifted staff into review work, the share of employees doing code review grew 14%. The bottleneck held even at firms that had already adopted AI code-review tools, because humans kept final sign-off. Twenty-one developers, product managers, and engineering leads were interviewed; one senior engineer said a tool like Claude Code "can write a draft in two seconds that's correct 90 percent of the time" for routine work, but makes frequent mistakes on harder tasks, "and you end up spending more time reviewing."
Corroboration, and a fairness note
A second study tells the same story at a different scale. "Writing Code vs. Shipping Code, " by Mert Demirer of MIT, Leon Musolff of the University of Pennsylvania, and Liyuan Yang, tracked more than 500, 000 GitHub developers across successive waves of AI tooling: the largest gains came from autonomous coding agents, then the gains shrank moving down the chain, from commits, to projects, to actual software releases. Across four major app marketplaces, the number of new apps rose sharply while total usage did not. A great deal of new software is being made that essentially no one uses.
The fairness note: two of that study's three authors are paid research consultants for Microsoft, which sells the GitHub Copilot tools the paper studies. That does not invalidate the work, the finding is unflattering to the AI-boosts-everything story, not flattering, but it is a disclosed interest worth stating.
The 25–50% claim, handled honestly
Circulating alongside the paper is an industry estimate: that AI-led productivity gains, measured on an end-to-end product basis, are "at best" 25% to 50%, prototyping is much faster now, but the finished product needs substantially more human work, so the 10x and 100x claims are fantasy. We charted it in the figure above, deliberately hatched and separated, because it is a reported estimate, not a study finding. The Harvard paper publishes no end-to-end productivity figure. Treat 25–50% as informed opinion, not evidence.
The honest bottom line from the evidence: individual coding got dramatically faster, and the firm-level gains got eaten by the review queue. This is consistent with our own scorecard work on five AI coding assistants, which measured what the tools do well at the individual level, the firm-level story is a different, harder problem. And the contrast with 17 AI agents rebuilding a game engine's source code is instructive: agents are spectacular at bounded, verifiable tasks, and firms' work is mostly neither.
ByteHerald's reading
The study's real contribution is a measurement correction: the industry has been grading AI coding tools on code volume, which is the metric most flattered by the tools themselves. Measure shipped software instead, and the "10x developer" story becomes a story about a moved bottleneck, from writing code to deciding whether to trust it. The firms that win the next phase won't be the ones that generate the most code; they'll be the ones that rebuild their review pipelines to match the new production speed.
By ByteHerald Staff. The author did not fetch the paper's PDF directly; figures are drawn from seven independently converging secondary reports of the paper. It is a working paper, not yet peer-reviewed; the design is observational (staggered difference-in-differences), and the 49% review figure is elapsed submit-to-merge time, not active reviewing time. Figures may be revised.
Comments
Post a Comment