A Dictionary-Lookup Test for Long-Horizon LLM Execution
Asked to keep a running sum of integers between -99 and 99, with every value supplied in the prompt and one lookup key given per turn, Qwen3-32B got the first turn right nearly every time. Its accuracy on the full task dropped below 50% within 15 turns. The result comes from Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab and Jonas Geiping, whose paper, The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs, was published at ICLR 2026 and is available on arXiv. They set out to measure how many steps a model can carry out when it already has the knowledge and the plan.
A task with no planning
Shojaee and colleagues had reported that reasoning models eventually fail as puzzles get longer. Sinha and his coauthors argue that those models follow the correct plan for many steps before slipping, so the failures occur in execution. Their test removes planning and knowledge demands. Each turn, the model receives one or more five-letter English words, looks up the integer assigned to each in a dictionary included in the prompt, and adds it to a running total. The number of turns and the number of keys per turn set the task length.
They tested Qwen3 at 4B to 32B parameters and Gemma3 at 4B to 27B, with 100 rollouts per model and no intermediate thinking. Every model except the two 4B models scored near-perfect on the first step. Larger models held up for more turns, with no sign of the gains leveling off, though with at most four sizes per family the authors declined to fit a scaling law. Stateless retrieval and stateless addition were both near-perfect, while a prefix-sum task that required carrying a running total declined slowly, which points to state tracking as where errors enter.
The authors also work through what step accuracy implies. If a model succeeds at each step with probability p and never recovers from an error, the task length it completes with 50% probability is ln(0.5)/ln(p). By my arithmetic, 99% step accuracy gives a horizon of about 69 steps and 99.9% gives about 693. The paper takes this to mean that short benchmarks may understate what small gains do over long tasks, and it treats the link between task length and economic value as a premise it does not test.
Errors that feed on themselves
Per-turn accuracy declined as the task went on, which could reflect longer context or the model reacting to its own earlier mistakes. To separate the two, the authors rewrote the chat history to contain wrong answers at rates of 0, 25, 50, 75 and 100 percent, then measured accuracy at turn 100. With a fully correct history, turn-100 accuracy was still below the turn-one level, which they attribute to context length. As the share of injected errors rose, accuracy fell further. Models above 200B parameters (Kimi-K2, DeepSeek-V3 and Qwen3-235B-Instruct-2507) reached near-perfect accuracy on a corrected history through 100 turns and still declined as errors were added. The paper's summary figure states that larger models are more prone to the effect.
Qwen3 thinking models did not self-condition, and their turn-100 accuracy stayed flat across error rates. The authors suggest reinforcement learning, which moves models away from plain next-token continuation, or the removal of earlier thinking traces from the history, as possible reasons. For non-thinking models, keeping only recent turns in context helped Gemma3 12B, though this suits only tasks without long-range dependencies. Prompting models to verify earlier answers gave mixed results, with Gemma3 exhausting its context and Qwen3 thinking models improving little.
Steps in a single turn
Without chain of thought, Qwen3-32B, Gemma3-27B, DeepSeek-V3 and Kimi K2 failed to handle more than six keys in one turn. The authors ran a binary search for the largest number of keys each frontier model could sum correctly with at least 80% accuracy. GPT-5 reached 2,176, Claude 4 Sonnet 432, Grok 4 384 and Gemini 2.5 Pro 120. DeepSeek-V3 without chain of thought failed at even four steps, while its thinking version R1 exceeded 100. A repeat with chains of 2x2 matrix multiplications, run because adding numbers can in theory be parallelized inside a transformer, found that models without thinking tokens could not multiply even two matrices.
What the authors caution
The task is synthetic, and the authors write that gains on it are needed for real-world long-horizon work but do not guarantee it. They also examined annotated failures from agent benchmarks released by other researchers and estimated that roughly 20% of GAIA, 48% of ALFWorld and 33% of WebShop failures resembled self-conditioning, while noting that judging step correctness there is subjective. The accuracy metric counts any mistake as failure, so self-correction goes unmeasured, and the authors note that in tasks where errors are easy to undo, recovery could extend how long a model keeps going.
