OpenAI’s Jalapeño Chip: Designed for Rapid Inference and Scalable Performance, Benchmarks Reveal

During the Hot Chips conference on Tuesday, OpenAI provided a comprehensive update on its new processing system, Jalapeño, showcasing the initial benchmark results. Evaluated using Semianalysis’s InferenceX benchmark, Jalapeño outperformed existing leading inference processors in terms of both token production per user and energy efficiency.

Richard Ho, OpenAI’s hardware chief, commented in a press call, “The key takeaway is that these results signify a remarkable leap in performance compared to the current standards. Jalapeño can handle greater AI workloads using less power while also delivering faster response times. It efficiently caters to numerous users and maintains low latency.”

It’s important to note that these results were compared to an Nvidia Blackwell system. However, by the time Jalapeño is fully operational, the competitive landscape may have evolved considerably. Ho projected that Jalapeño would be deployed in limited quantities by the end of 2026, with larger scale deployment anticipated in 2027.

First introduced last October, Jalapeño was created through a collaborative effort between OpenAI and Broadcom, utilizing OpenAI’s models throughout the development. The company aims to establish Jalapeño as a versatile platform that integrates AI products, models, chips, and memory in a cohesive manner.

This holistic approach allowed OpenAI to focus on specific stages of the inference process that can lead to delays. Specifically, Jalapeño is engineered to reduce lags during the prefill and communication stages, which are often the sources of inefficiencies.

The company explained, “We designed Jalapeño to reduce data movement and communication delays. This allows the model state, including the KV cache used for response generation, to be effectively localized while the system optimally engages the necessary compute, memory, and networking resources for each inference phase.”