Inside Jalapeño: How OpenAI's First Chip Targets Memory & How Kimi K3 is Tackling the Same Problem
- David Borish
- 10 hours ago
- 6 min read
OpenAI has released its first full set of performance data for Jalapeño, the custom inference accelerator it developed with Broadcom, and the numbers describe a chip built almost entirely around one problem: getting data to the compute cores fast enough that they stop sitting idle. Tested on the InferenceX benchmark suite from SemiAnalysis against three model families, GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T, Jalapeño delivered 1.5 to 1.9 times more mixed-token throughput per kilowatt at peak operation and cut end-to-end request latency by 1.7 to 3.6 times compared with the leading commercially available systems OpenAI tested against. For the most latency-sensitive, interactive workloads, the gap widened to 2.1 to 4.1 times higher performance.
The most striking figures come from matching decode speed exactly to what the comparison hardware could do at its fastest setting, then measuring how much throughput per kilowatt each system could still deliver. At that matched point, Jalapeño produced 53.7 times more mixed tokens per kilowatt than the comparison system on GPT-OSS, 104.3 times more on DeepSeek R1, and 56.1 times more on Kimi K2.5. Those multiples describe the widening advantage at the fastest, most demanding operating points. OpenAI's own charts show the gap narrowing considerably at less demanding settings, so this ceiling case should be read as the extreme rather than the typical result.
How the test was built
OpenAI evaluated performance the way an operator actually experiences it: how much useful work a system completes per unit of power while holding to the latency an interactive agent needs, rather than raw throughput alone. That framing matters because agentic workloads chain many model calls together, so a small per-step delay compounds across a full task. The company normalized results using each accelerator's published chip power rating, with Jalapeño rated at 700 watts against comparison chips rated at 1,200 and 1,400 watts, and noted that Jalapeño's measured sustained power stayed at or below 550 watts on the workloads tested. On the largest model in the test set, Kimi K2.5's one trillion parameters, Jalapeño delivered about 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency than the comparison system, a smaller gap than on the other two models but still enough to sit on what OpenAI calls the Pareto frontier for the tested operating range.
A chip designed around keeping data close
Jalapeño's architecture starts from a specific observation about how language models actually run. Processing a prompt is compute-intensive, generating a response token by token is limited mainly by memory bandwidth, and communication delays between chips can leave compute sitting idle while it waits for data to arrive. OpenAI designed Jalapeño to minimize that movement, keeping model state, including the key-value cache used while generating a response, explicitly placed and local, while a large networking domain built around Broadcom's Tomahawk technology lets an entire workload stay inside one connected system rather than shuttling data across chip boundaries. Tom's Hardware's analysis of the released chip imagery backs up that emphasis on memory: the package shows one large compute chiplet surrounded by six HBM modules, a layout consistent with an accelerator built to keep memory access local rather than one optimized purely for raw compute density.
Broadcom's role went beyond manufacturing. The company contributed the silicon implementation and the networking fabric, while Celestica handled board, rack, and system integration, an arrangement OpenAI has framed as a template for how an AI operator with deep knowledge of its own models and serving patterns can pair with an experienced silicon partner rather than building fabrication capacity itself. Broadcom CEO Hock Tan has said compute demand across the company's AI customers is effectively without limit through the end of the decade, and OpenAI President Greg Brockman told CNBC that the company "cannot get compute fast enough," a framing that lines up with OpenAI's decision to keep expanding rather than substitute one supplier for another.
AI building the hardware it will later run on
Jalapeño moved from initial design to tapeout in nine months, a timeline OpenAI attributes partly to using its own models to explore chip implementations and shorten verification loops, including work on the chip's arithmetic circuits. That same logic extended past the chip itself: OpenAI says it also designed Jalapeño as a predictable programming target, one where engineers describe work as local tensors and explicit communication so that AI systems can handle the harder problem of scheduling and mapping that work across the full system. Using Codex with GPT-Astra, the team brought three open-weight models that were not part of Jalapeño's original production plan to high performance within two months, and for selected attention and mixture-of-experts blocks in GPT-OSS, the AI-generated implementations ran 1.5 to 1.8 times faster than the versions written by human engineers. It is a small, contained example of a pattern that shows up across frontier AI development more broadly: capability proven first in a constrained, verifiable setting, chip design and kernel optimization in this case, tends to migrate outward into systems that ship at scale once the underlying approach is validated.
The same bottleneck, seen from Beijing
Five weeks before OpenAI's Jalapeño results, Moonshot AI released Kimi K3, a 2.8 trillion parameter model that activates only 104 billion parameters per token across 896 experts, and the reaction on trading floors echoed the shock that followed DeepSeek's debut in early 2025. Semiconductor stocks fell on the news, with Bloomberg reporting that shares of Chinese AI rivals dropped sharply, Z.ai down 28 percent and MiniMax down 16 percent the same day. But Bloomberg's own analysis argued the comparison to DeepSeek was incomplete. DeepSeek's breakthrough centered on training and serving models more cheaply. Kimi K3 improves computing efficiency too, but it does so at a scale that places far heavier demands on memory infrastructure, the kind of demand that continues to support the case for high-bandwidth memory from SK Hynix and Nvidia's latest systems rather than undercut it.
Moonshot's own technical framing supports that reading. The company built Kimi K3 around Kimi Delta Attention, a hybrid linear attention design intended to reduce the memory pressure that standard transformer attention creates as context windows grow, paired with a change to how information moves between layers that Moonshot calls Attention Residuals. Independent technical writeups of the model's hosting requirements describe decode as the harder problem to solve: generating each new token uses the model's weights exactly once, so compute cores spend much of their time waiting on high-bandwidth memory to feed them data, regardless of how sparse the model's routing is. Moonshot's own deployment guidance recommends supernode configurations of 64 or more accelerators for production use, and third-party estimates of the released checkpoint put its minimum practical VRAM footprint above 1.6 terabytes.
China's memory supply chain remains the binding constraint on that architecture. Huawei has led a domestic consortium working toward producing high-bandwidth memory chips, but advanced HBM production still sits concentrated in SK Hynix, Samsung, and Micron, and China's most advanced domestic memory maker has not yet demonstrated it can manufacture HBM at the density modern accelerators require, let alone at volume. A Bank of America research note on Kimi K3's release argued that large-scale pretraining combined with architectural efficiency can still produce major gains for Chinese labs despite persistent compute constraints, though that framing describes a workaround for a memory shortage rather than a solution to it.
Two labs, one constraint
Set next to each other, Jalapeño and Kimi K3 describe the same underlying pressure from opposite starting points. OpenAI has effectively unlimited access to leading-edge chips and is building custom silicon anyway, in pursuit of power efficiency and cost at a scale where even a 1.5 times improvement in tokens per kilowatt compounds into meaningful savings across a fleet of data centers. Moonshot is working around export controls that limit access to the newest accelerators and the memory that goes with them, and it has responded by designing a model architecture that asks for less memory bandwidth per token rather than assuming more will become available. Both point to the same conclusion: as models grow and inference workloads shift toward longer, more agentic tasks, memory bandwidth has become the resource labs on both sides of the Pacific are now racing to conserve, ahead of raw compute throughput.
What comes next
OpenAI says it plans to begin deploying Jalapeño within its own infrastructure by the end of 2026 while continuing to expand its use of Nvidia and other suppliers rather than replace them. The company describes Jalapeño as the first generation of a multigenerational platform, with a second generation already deep in development and a third taking shape behind it. Whether the efficiency gains OpenAI has published hold up once Jalapeño runs at the scale of a full data center, rather than in the lab conditions used for this report, is the test that will determine whether the architecture becomes a meaningful part of OpenAI's compute stack or a proof of concept that gets folded into later designs.
About the Author
David Borish is a journalist and analyst covering frontier AI, cybersecurity, and emerging technology, and the author of the forthcoming book The Tony Hawk Paradox, which examines how capabilities proven in controlled or simulated environments migrate into broader real-world systems. More of his analysis is available at davidborish.com.
