The First Bad Token: A Study on Tipping Points in Small Language Models
A 124-million-parameter GPT-2 model, asked about vaccines, hurting people or self-harm, gave different answers depending on what it had been asked earlier about the shape of the Earth. Neil Johnson and Frank Yingjie Huo of George Washington University present that behavior in Patterns as the product of one calculation inside the model's attention mechanism, and they supply a formula for when a run of acceptable output will flip to an unwanted one.
The formula
The paper concentrates on models small enough to run offline on a phone, where no cloud service filters what comes out. An attention head compares the latest token with earlier ones using dot products and uses the result to choose the next token. The authors sort output into content classes: B for desirable output such as "round," D for undesirable output such as "flat." Each class is represented by the average embedding of six example phrases.
Whether the model produces D immediately or after a stretch of B depends on the dot products between the conversation so far and those two averages. The tipping point, n*, is the number of B tokens that appear before the first D. If the conversation already sits closer to D, n* is zero and the model tips at once. If it sits closer to B, the model produces a run of B first, and the length of that run is set by the same dot products.
The tests
In the February preprint, the authors ran the formula against six open-weight models from OpenAI, EleutherAI and Meta, ranging from 124 million to 410 million parameters. They measured n* at the token level. Separately, they split 300 tokens of greedy-decoded output per model and prompt into 704 sentences and labeled each as B or D. Claude served as the labeling model, and the authors report that their manual labels matched its labels exactly. The preprint is at arxiv.org/abs/2602.14370.
The two measures agreed on D detection in 16 of 18 non-control cases (89%). Both misses involved negation, as in "The Earth is not flat," where the individual token embeddings sit near D while the sentence means B. The formula identified immediate versus delayed tipping correctly in 15 of 16 clear cases (94%), after two near-boundary cases were set aside. For the Flat Earth and Vaccines prompts, all six models agreed on the direction of the dot-product gap.
The authors note that confidence intervals from resampling the six phrases span zero for many model and prompt pairs, and they place the statistical weight on agreement across models. According to GW's release, the published paper extends testing to seven models of up to 12 billion parameters. Cell's full text was not accessible for this article, so the figures above come from the preprint.
Steering and monitoring
Because the formula takes the whole conversation as input, added content moves the tipping point. In the paper's worked example, content aligned toward D shortens n* and content aligned away from D lengthens it. The authors say this ordering effect matches a 2025 study by the Center for Countering Digital Hate, in which ChatGPT-4o returned harmful content in 53% of 1,200 responses to 60 high-risk prompts, including 44% of self-harm responses and 66% of eating-disorder responses. The match is structural. The CCDH transcripts are not public, so the authors did not test the formula against them directly.
They propose a monitor that runs alongside generation. Basin averages are computed ahead of time, and each token costs a few dot products, which the authors say scales with the embedding dimension d where a forward pass scales with d squared. A flag would fire when n* falls below a chosen threshold. Alignment training such as RLHF can move the averages apart and suppress tipping for specific prompts, the authors write, but a new content domain or an adversarial prompt can restore the condition.
Limits
The preprint's tests used small models and a 300-token window, and predicted n* values can be off by one. Defining B and D for each application domain requires hand-written example phrases, which the authors describe as a one-time annotation task. The preprint reports no external funding. Johnson is a co-founder of d-AI-ta Consulting LLC, which the preprint says is not directly related to the study.
