OpenAI's Jalapeño Chip Boosts AI Inference

OpenAI has unveiled early results for its custom inference chip, codenamed Jalapeño, showcasing significant advancements in AI performance. The chip demonstrates industry-leading speed and power efficiency, promising faster and more responsive AI services for users.

The Jalapeño chip is designed to handle the demanding task of AI inference, the process of running trained AI models to generate outputs. OpenAI claims it delivers both higher throughput and lower latency within a single architecture, a notable achievement as existing hardware often requires trade-offs between these two critical metrics. This dual improvement means quicker responses for end-users and more reliable access as demand for AI services grows.

This development is significant for OpenAI, a company that has consistently pushed the boundaries of AI capabilities. With a Microsoft (NASDAQ:MSFT) backed valuation of $850 billion, OpenAI is investing heavily in its infrastructure to support its ambitious mission of ensuring artificial general intelligence benefits all of humanity. StartupHub.ai data shows OpenAI with a score of 86/100, positioning it among the top AI companies, including competitors like Google DeepMind (score 83/100) and Anthropic (score 76/100).

Architected for Speed and Efficiency

Jalapeño's architecture was conceived from the ground up with modern language models in mind. The design focuses on minimizing data movement and communication delays, crucial for handling the distinct phases of AI inference, such as prompt processing (prefill) and token generation (decode). By keeping model state local and optimizing compute, memory, and networking for each phase, the chip ensures that processing units remain active and efficient.

A key innovation is the integration of networking within the chip's architecture. This allows entire workloads to remain within a connected system, minimizing latency and enhancing overall efficiency from start to finish. This balanced approach makes Jalapeño a fungible accelerator, adaptable to evolving model architectures and capable of excelling in the dynamic demands of agentic workloads.

AI as a Design Partner

Remarkably, AI played a direct role in Jalapeño's creation. OpenAI utilized its own AI models to accelerate the chip design process, reducing the time from initial concept to tapeout to just nine months. AI also assisted in optimizing the chip's arithmetic circuits, enabling more performance to be packed into the silicon.

Furthermore, the chip was engineered to be a predictable programming target for both humans and AI. This clear structure allows AI systems to more effectively optimize the mapping, placement, scheduling, and coordination of tasks across the system. OpenAI reported that AI-generated implementations for specific blocks in open-weight models ran 1.5 to 1.8 times faster than human-expert written code, highlighting a new paradigm in hardware-software co-design.

Performance Across Models

Jalapeño's efficacy has been demonstrated across several large, open-weight models, including GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. Across these benchmarks, the chip delivered 1.5 to 1.9 times more AI work per watt and 1.7 to 3.6 times lower end-to-end latency compared to existing systems. For interactive workloads, performance saw an even greater boost, with 2.1 to 4.1 times higher output.

Specifically, on the Kimi K2.5 1T model, Jalapeño achieved approximately 1.5 times higher peak performance per watt and 3.4 times lower latency. OpenAI noted that internal testing on their own frontier models shows an even wider advantage, suggesting Jalapeño's value increases with model size and complexity.

The Path Forward

OpenAI plans to begin deploying Jalapeño within its compute infrastructure by the end of the year. This first-generation chip is part of a multi-generational roadmap, with second and third generations already in development. The company emphasizes that meeting the growing demand for AI will require a broad deployment of accelerators, including those from partners like Nvidia.

The full-stack advantage OpenAI possesses, designing models, products, serving software, and hardware together, is evident in Jalapeño. This integrated approach allows for continuous learning and improvement across all layers of the AI stack, ultimately aiming to make increasingly capable AI more affordable and accessible to everyone. The results so far indicate a future of more responsive, capable, and agentic AI delivered with unprecedented efficiency.