The Replay Trick: Google DeepMind's Path to Cheaper Recursive Self-Improvement
Getting an AI coding agent to discover a faster algorithm, a better mathematical construction, or a more efficient GPU kernel usually means running it thousands of times: propose a candidate, test it, learn from the result, propose again. That loop works, but it is slow to improve, because the thing actually worth improving, the search strategy itself, is expensive to evaluate. Figuring out whether a new way of allocating search effort is any good requires watching it play out across an entire discovery run, which can take hundreds or thousands of proposal-and-test cycles before the answer comes back.
A paper released September 14, 2026 by a team of Google and Google DeepMind researchers, working with academic collaborators at the University of Maryland and the University of Virginia, proposes a way around that bottleneck. The system, called Dream-RSI, treats the record of everything a search process has already tried as a reusable simulator. Instead of running a new search strategy for real to see how well it performs, the system replays it against history that has already been collected, for free.
The problem with improving how you search
Recursive self-improvement systems generate candidate solutions, score them, and use the results to guide the next round. Systems like AlphaEvolve and its successors have shown this loop can produce genuine mathematical and engineering discoveries. But as these systems scale to longer horizons and larger search spaces, the strategy that governs the search, which branches to pursue, how many to run in parallel, when to give up on a dead end, becomes the actual lever worth pulling. Most existing systems keep that strategy fixed for the entire run. A fixed strategy cannot learn from what has already happened, and it can keep pouring compute into directions that stopped paying off rounds ago.
Recent work has tried to optimize the search strategy itself while discovery is underway, and the paper identifies two reasons that is hard. Feedback on a strategy only appears after an entire long-horizon rollout finishes, so testing one idea for how to search can be as expensive as running the whole discovery process from scratch. And the space of possible search strategies is large, so a team may need to try many candidates before finding one that works. Put together, evaluating a single proposed improvement to the search process can cost as much as the discovery task it is meant to speed up.
History as a stand-in for the real world
The paper's central move borrows an idea from model-based reinforcement learning, the Dreamer line of research in particular, where an agent learns a compact model of its environment and then improves its behavior by imagining outcomes inside that model rather than acting in the real world every time. The researchers argue that a completed discovery run already provides something equivalent. Every proposal an agent tried, every score it received, and every dead end it hit gets recorded in a tree structure. Because those outcomes are already known, an alternative search strategy can be run against that tree and scored on how it would have performed, without generating a single new candidate or calling the underlying coding agent again.
Dream-RSI operationalizes this as a three-part loop. In the online phase, the current search strategy directs a real discovery run, producing a fresh tree of attempts and outcomes. That tree is then folded into a growing pool of replay simulators built from every run so far. In the offline phase, the system generates and evaluates a large batch of candidate revisions to the search strategy, each one scored purely by replaying it against the accumulated simulator pool, at no execution cost. The best-scoring version is deployed for the next real discovery run, which produces more history, which expands the simulator pool again. The loop closes on itself: each round of real search feeds a growing archive that makes the next round of strategy improvement cheaper and better informed.
The underlying coding agent, the evaluator that scores candidates, and the models involved all stay fixed throughout. Only the code governing how search effort gets allocated changes.
Testing it across three very different domains
The researchers evaluated Dream-RSI on eight tasks spanning three domains, comparing it against a controlled baseline that uses the identical setup but never updates its search strategy, plus domain-specific published systems.
In algorithm engineering, the target task was discovering an efficient solver for the Lasso regularization path, a common tool in high-dimensional statistics. Using Google's Gemini 3.1 Pro as the coding agent, Dream-RSI cut the number of agent calls from 550 to 317 across five recursive rounds while also improving average runtime on six held-out datasets, from 3587.1 milliseconds down to 2931.0 milliseconds. With the smaller Gemini 3.7 Flash model, calls dropped from 3,200 to 1,879 with a similar runtime improvement. Measured against SimpleTES, a previously published system that used 51,200 generations with an open-weight model to reach a comparable solver, Dream-RSI reached faster downstream performance using roughly 162 times fewer discovery-agent calls. The solver it landed on combines strong-rule feature screening with an adaptive pruning check based on the Cauchy-Schwarz inequality, a different mechanism than the LARS-and-coordinate-descent switching that SimpleTES's solver relies on.
In mathematical optimization, covering the sum-difference problem, autocorrelation inequalities, and circle packing, Dream-RSI matched or edged out a field of comparison systems that includes AlphaEvolve and its successor AlphaEvolveV2, while using under 1,000 generations. On the autocorrelation task specifically, SimpleTES posted the best published score, but it needed 51,200 generations to get there.
In GPU kernel engineering, using four tasks from the KernelBench benchmark, Dream-RSI reached comparable final performance to the fixed-strategy baseline using 2.43 times fewer generations on a VGG16 kernel and 1.79 times fewer on a LayerNorm kernel. On two other kernels, ConvDiv and ConvMax, it reached 2.09 times and 1.44 times higher performance respectively when both methods were given the same generation budget.
A finding that cuts against the obvious fix
One might expect that feeding a search agent more explicit guidance, telling it directly which directions look promising based on past rounds, would help it search better. The researchers tested this by injecting high-level directional hints straight into the discovery agent's prompt, applying the same treatment to both Dream-RSI and the fixed-strategy baseline. In both cases, the version with explicit guidance performed worse than the version without it. The paper's interpretation is that strong semantic steering, layered on top of a process already running many parallel search threads, tends to narrow the field of exploration rather than sharpen it, which works against the diversity that long-horizon search depends on. Learning to navigate history through recorded outcomes, rather than being told in words what history implies, was the better path in these experiments.
A separate analysis of one kernel task tracked how the learned search strategy changed over eight recursive rounds. As performance climbed early on, the strategy cut back on how many candidates it evaluated per round, from 110 down to 50. Once progress leveled off, it pushed evaluation effort back up, and further gains followed. The strategy was, in effect, learning when to conserve compute and when to spend more.
What this does and does not show
Dream-RSI is a preprint, posted to arXiv on September 14, 2026, and it has not gone through peer review. The comparisons against SimpleTES, AlphaEvolve, and the other domain-specific baselines rely on results reported in each of those systems' own papers rather than figures the Dream-RSI team reproduced independently on matching hardware. The GitHub repository accompanying the paper currently hosts the paper and a project page, with the full codebase, reproduction scripts, and the discovered programs themselves still listed as in progress.
What the paper does show clearly is a specific mechanism for cutting the cost of improving a search strategy: build a simulator out of what has already happened, and test new ideas against it before spending real compute on them. Whether that mechanism generalizes past the eight tasks tested here, to messier real-world discovery problems with less clean scoring functions, is the open question the released code and datasets will eventually help answer.
About the Author
David Borish is a writer covering AI research and its practical implications for business and technology. He is the author of The Tony Hawk Paradox: When Video Games Predict Reality and publishes The AI Spectator, a weekly newsletter on AI developments. More at davidborish.com.