Two physicists at George Washington University say they have found a formula, derivable with high-school arithmetic, that predicts when a chatbot will flip from helpful answers to harmful ones. Tested on seven open-weight models, it called 18 of 19 tipping outcomes correctly. Here is what the math actually claims, what it proved, and what is still hype.
What the formula says
Neil Johnson and PhD student Frank Yingjie Huo published the work in Patterns, a Cell Press journal. The formula outputs a single number, n: roughly how many good, sensible responses a chatbot will give before it flips and produces something harmful. Huo traces the crack to "one unit of its attention", the smallest working part of the machine, the same attention mechanism we walked through in our visual explainer. Johnson's line that traveled everywhere: it floored the team that a machine with billions of moving parts "obeys a formula you can derive with pen, paper, and arithmetic taught in high school."
The key insight is the tug-of-war: everything said earlier in a conversation feeds the competition over what comes next, so the order of a conversation matters as much as its content. The team asked the same set of questions, about vaccines, hurting people, self-harm, in two different orders. In one order, the AI gave an undesirable answer to every single question. In the other, it gave an acceptable answer to each one. And once the flip happens, the slope takes over: every undesirable answer drags the next one further down. Notably, "undesirable" is definition-agnostic, misinformation, a medical or legal duty breach, a dangerous instruction. It is not true-versus-false. A bad answer can be factually correct yet dangerous.
How they tested it
The team ran the formula against seven openly available transformer models, ranging from 124 million to 12 billion parameters, built by three independent groups. The model called the right outcome in 18 of 19 tipping cases. Independent checks of the big commercial chatbots showed the same behavioral patterns the formula predicts, immediate flips in some conversations, delayed runs of good answers before the flip in others.
The warning light
The practical pitch is disarmingly simple: a few calculations before each output, and phones of the future could carry a built-in warning light that fires at the tipping point, before the bad answer, not after. Johnson says his lab has already inserted the check into their open-source models, adding: "obviously we cannot get inside OpenAI and Anthropic's closed AI models to do this." The motivation is concrete: phones now run small chatbots offline with little safety oversight, and those are exactly the deployments nobody is watching.
Proven vs hype, the honest version
Proven: the formula fits observed tipping behavior across the seven tested models; the order-of-questions effect replicated; the flip-after-first-bad-output dynamic matches what independent testers see.
Not proven: generalization to frontier closed models, the biggest systems were not and cannot be tested this way. The warning light is a lab prototype, not a deployed product. The number n is statistical, not a guarantee for any single conversation. The math detects drift into "undesirable" however you define it, it does not distinguish deception from honest error. And the most dramatic claim, the "Lord of the Fl(AI)es" scenario, one tipping agent's output tipping other agents it talks to, with no human in the loop, is a hypothesis, not a demonstrated attack. Anthropic's own warning about self-replicating AI worms this week shows the labs are thinking about propagation too, but thinking about it is not the same as measuring it.
How it compares
Four approaches to the same problem, side by side. The Johnson-Huo formula is predictive and pre-output, it forecasts the flip before it happens, but it needs model-internal access and is still lab-stage. METR-style evaluations are behavioral and third-party, but post-hoc: they measure what already happened. Anthropic's threshold alarms are forecast-based tripwires, but they live inside one lab. Red-teaming finds failures adversarially, but only after the failures exist. The formula's novelty is real: it is the only approach that predicts before the output. It is also the least independently verified.
ByteHerald's reading
If it holds up, the safety conversation changes shape: from "did the model misbehave" to "how many turns until it does." The honest version is narrower than the headlines, 18 of 19 on small open models is a genuine signal, not a solved problem. The open question that matters: does n scale to frontier models, or does the tipping behavior wash out at a hundred billion parameters? Nobody can answer that without the labs' cooperation, which is exactly why the warning-light pitch is aimed at open models and phones, not at anyone's data center. Watch for independent replication on larger models. That is the number that decides whether this is a curiosity or a tool.
By ByteHerald Staff. Based on Johnson & Huo, Patterns (2026), via SWNS, SecurityWeek and LiveScience reporting. Diagram is illustrative, not measured model data. The single Johnson quotation is brief; all else paraphrased.
Comments
Post a Comment