Visual Symbols and Code Generation in Frontier Vision Language Models
Asked to implement an inorder binary tree traversal, Claude Opus 4.7 and GPT-5.4 each wrote a recursive solution in all 20 samples when a red heart image preceded the prompt. Without the image, neither model produced a recursive answer. Researchers at Google DeepMind and Stanford report the result in A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs.
Design and results
Alexa Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer, Martin Wattenberg, Been Kim and Yuxuan Li built a library of 288 symbols, 24 shapes in 12 colors, each rendered at 400 by 400 pixels. The task, inorder traversal, can be solved recursively or iteratively. The image went before the prompt, 20 samples were drawn per condition, and an automated judge (Gemini 3.5 Flash) classified each answer. The main models were Claude Opus 4.7, Gemini 3.1 Pro and GPT-5.4.
Baseline recursion probability was 0 for Claude and GPT-5.4 and 0.4 for Gemini. The red heart raised it by 1.0 for the first two. Gemini responded most to a yellow checkmark, which lifted it by 0.5 to 90% recursion. Telling the models outright to use recursion raised Claude by 1.0 and Gemini by 0.6. Averaged by shape, Claude's strongest responses came from triangles (+0.83), hearts (+0.79) and circles (+0.68). Gaussian noise had almost no effect. Shuffling a symbol's pixels or rotating it substantially reduced steering, and a token-matched text caption of each symbol correlated only weakly with the image's effect (r = -0.05 for Claude, 0.31 for Gemini, 0.17 for GPT-5.4).
The symbols also shifted other behavior. Pink shapes and hearts pushed writing toward poetic prose, and blue triangles moved Gemini and GPT-5.4 from preferring summer to preferring winter. The effect carried over to five more recursion problems, depth-first versus breadth-first search, and data structure choice.
What the models reported
When asked to interpret a symbol directly, the models' answers correlated with their behavior at r = 0.22 (Claude), 0.29 (Gemini) and 0.31 (GPT-5.4). The authors also read 2,880 thinking traces from Gemini 3.1 Pro. In steered runs, 71.5% of traces (1,524) disregarded the symbol in their reasoning and 25.0% (533) credited it. In runs that were not steered, 19.3% (145) of traces claimed the symbol had driven the answer. Across nine additional coding tasks and 20,130 traces, 64.1% showed this unfaithful disregard. For GPT-5.4, 74.7% of non-steered traces said the symbol drove its behavior, while only 50.4% of steered traces did. Claude Opus 4.7 returned no thinking traces and is absent from this analysis.
In a silent injection condition, where the prompt never mentioned the image, the maximum recursion increase was 1.00 for Claude, 0.60 for Gemini and 0.50 for GPT-5.4. Every GPT-5.4 trace and 52.3% of Gemini traces never mentioned the image at all.
Sender and receiver games
In the coding game, a sender model refined an image over up to 20 turns to steer a receiver, which was the same model, toward recursion. Symbol-based success took an average of 5.2 turns for Claude, 1.3 for Gemini and 14.0 for GPT-5.4. Senders mostly picked spirals, infinity signs and arrows, shapes that read as intuitive to people, and these were seldom the symbols with the highest measured steering.
In a second game, the sender conveyed a concept or story so that the receiver could pick the right answer from four options within five turns. Free-form drawing through PIL code succeeded eventually 97.8% of the time and on the first attempt 91.2% of the time. With the symbol library, 47.8% succeeded on the first attempt and 30.7% reached the turn limit without success.
Seven people took part in an exploratory human study. About 36% of their judgments matched the steering direction of the top-steering symbols, below the 50% chance level (Gemini 44.4%, Claude 38.9%, GPT-5.4 33.3%). Symbols the senders chose were interpreted correctly by humans about 67% of the time. Across turns, human accuracy on symbol messages fell from 66.7% to 44.4% for recursion and from 77.8% to 55.6% for stories. Free-form drawings stayed readable, at roughly 79 to 91% on concepts and 100% on stories.
Mechanism and limits
The authors do not identify a mechanism. They suggest image representations may land near task concepts in the model's internal space, or that image-to-text training creates associations that are not tied to generation. They note that influence the model never verbalizes could escape oversight that relies on reading reasoning traces. Frontier models largely ignored noise and photos, which the authors say may mean steering shifts toward structured, symbol-specific influence as capability grows. They list two limitations: the tasks are synthetic, and the human study is small.
About the author
David Borish is the author of The Tony Hawk Paradox: When Video Games Predict Reality More at davidborish.com
