Jalapeño and the economics of inference

What changes when a customer builds its own chip?

A faster chip earns attention. A credible alternative changes the next purchase.

That is the useful way to read Jalapeño, OpenAI's custom inference accelerator. Its first public results give OpenAI a reason to investigate moving repeated serving work onto hardware it controls. They do not establish the cost of a production fleet, or tell us how much NVIDIA will charge OpenAI next year.

My argument is narrower. If custom hardware can reliably serve a meaningful share of demand, OpenAI gets another option when buying capacity. NVIDIA can continue selling more systems while facing harder negotiations over some of that business. Revenue growth and reduced pricing power can happen together.

This Congar edition adapts my original DEV article, with new diagrams and a closer look at what the evidence supports.

What has OpenAI actually built?

Jalapeño is an accelerator designed for language-model inference. OpenAI announced it with Broadcom in June 2026. Celestica contributes board, rack, and system expertise. At launch, OpenAI described engineering samples running workloads at the intended operating frequency and power, with initial deployment planned by the end of 2026. Those are different milestones from qualifying and operating a large fleet. OpenAI's launch announcement

The larger program predates the chip announcement. In October 2025, OpenAI and Broadcom set a 10-gigawatt custom-accelerator target, with rack deployments scheduled to begin in the second half of 2026 and finish by the end of 2029. The announcement describes development and supply agreements alongside a deployment term sheet. Its capacity figure is a plan, not a meter reading. The Broadcom collaboration

That distinction matters to the economics. A working sample answers whether the design can execute useful work. A factory ramp answers whether enough good chips can be built. An operating fleet answers whether they can deliver a service at an acceptable cost. Each step has its own failure modes.

Why does inference spend so much time moving data?

During prefill, a serving system processes the prompt and prepares attention state. During decode, it generates successive tokens using the model's weights and cached state. The balance of arithmetic, memory traffic, and communication changes between these phases, and with the workload.

One serving design puts prefill and decode on separate resources. Another keeps them together. Neither choice follows automatically from the word "GPU."

The DistServe research explains a reason to separate them. Prompt processing can interfere with ongoing token generation. Giving each phase its own resources allows separate allocation and scheduling, while making communication between the phases another cost to manage. The paper's placement strategy explicitly accounts for available bandwidth.

Transcript: Prefill processes a prompt and builds KV state. During decode, model operations repeatedly use weights and prior attention state to produce another token and extend that state. Placing useful state near the compute that consumes it can reduce movement. This is a conceptual serving explanation; GPUs can also use local state. Timing and distance in the animation are not measurements.

Open full-size static illustration ↗Download GIF

Read this animation as a conceptual explanation of placement choices. It is not a GPU wiring diagram or a measured latency comparison. A GPU serving system can also preserve local state; a large custom-accelerator deployment can still communicate across chips.

OpenAI describes Jalapeño as allowing explicit placement of model state, including the KV cache, while balancing resources across inference phases. Its technical results explain that design goal.

The operational question is where waiting occurs. A long prompt can occupy compute that another request needs. A growing cache can consume memory capacity. Moving state can consume network bandwidth. Optimizing one of these costs can make another more visible. The useful comparison is between complete serving configurations under the same demand.

What do the first numbers establish?

OpenAI reports peak throughput per watt improvements of 1.5 to 1.9 times, and end-to-end latency speedups of 1.7 to 3.6 times, across GPT-OSS 120B, DeepSeek R1, and Kimi K2.5. Its appendix compares GPT-OSS with GB200 and the other models with GB300, using nominal 8,000-token inputs and 1,000-token outputs. The published results and appendix

The power denominator deserves attention. OpenAI normalizes using published accelerator power ratings. Jalapeño's rating is 700 watts; OpenAI separately reports sustained draw at or below 550 watts in these tests. The headline ratios are not a measurement of the entire facility's electricity bill. OpenAI's measurement explanation

Open full-size diagram ↗

SemiAnalysis provides a useful first-person account of the tests. Its researchers say they observed InferenceX runs in OpenAI's lab, but OpenAI supplied the numbers. They did not run the complete suite and had not seen Jalapeño AgentX results at the time of that report. The lab account and caveats

AgentX matters because it tests long-context, multi-turn coding sessions, including cache behavior and subagent execution. Those demands differ from a fixed prompt and response length. The benchmark's own methodology separates these scenarios. InferenceX and AgentX methodology

A procurement team also needs failure rates, replacement time, utilization, and software maintenance costs. None can be calculated from a tokens-per-watt ratio. Even a real chip-level efficiency gain can shrink when the rest of the system consumes power or waits for work.

Can software make a new accelerator cheaper to adopt?

OpenAI says Jalapeño reached tapeout nine months after initial design, with its models helping parts of the development process. Tapeout is the handoff for manufacturing, not nine months of proven production service. The launch account

It also reports faster implementations for selected GPT-OSS blocks. That claim covers those blocks, not an equivalent speedup for the complete model. The software results

The practical question is how much engineering work remains each time a model changes. A fast kernel is useful. So are a debugger that explains a failure, a compiler that handles a new operation, and a rollback procedure that restores service without losing requests.

My reading is that automated optimization may reduce part of the adoption cost. The evidence does not price the whole migration. A team evaluating custom silicon should count the people needed to support it after the exciting benchmark runs are over.

Where could NVIDIA feel pressure first?

OpenAI has already described its purchasing logic. In her infrastructure strategy, CFO Sarah Friar says the company selects providers for different workload requirements and uses credible alternatives to maintain pricing discipline. That is a statement of intent, not evidence of a negotiated discount. OpenAI's infrastructure strategy

A plausible allocation follows. Predictable, repeated inference becomes a candidate for custom hardware when utilization and support costs justify it. Training, unfamiliar models, and spare capacity can favor a platform with broader software support. The boundary can move as either platform improves.

Transcript: A workload is matched to a platform. Repeated, predictable inference is a candidate for custom silicon such as Jalapeño. Training, changing workloads and extra capacity can continue to use broader platforms. Actual choices depend on production costs, software support, utilization and reliability. Both branches can grow together; this is a scenario, not a deployed fleet map or market-share forecast.

Open full-size static illustration ↗Download GIF

This allocation is an economic scenario, not OpenAI's disclosed deployment map. The animation gives no market-share forecast and assigns no percentage of traffic to either supplier.

For NVIDIA, the relevant exposure is the price of the next capacity purchase. A customer with an operational alternative can compare an external quote with its own cost to serve. A customer with only a prototype still has to discount that alternative for delivery and execution risk.

NVIDIA reported $96.2 billion in revenue and a 75.0 percent gross margin for its second quarter of fiscal 2027. These are company-wide figures. The filing does not identify a separate inference gross margin, so treating 75 percent as the profit available on every inference sale would be wrong. NVIDIA's quarterly filing

Nor does a supplier's gross margin become the customer's savings. A custom program still pays for development, manufacturing, memory, networking, spare systems, and operations. The comparison is total cost at the required service level.

Why would OpenAI keep buying NVIDIA systems?

Its NVIDIA relationship already includes a separate announced target of at least 10 gigawatts. The September 2025 announcement was a letter of intent, with a first phase targeted for the second half of 2026 on Vera Rubin. It described infrastructure for training and running models. It did not establish that the capacity was already deployed. The NVIDIA partnership announcement

Those parallel programs support a diversification argument. They do not establish a one-for-one replacement schedule.

Consider demand that grows faster than a custom-chip program can ramp. In that situation, every useful Jalapeño system adds capacity while GPU purchases continue. If demand steadies, competing options can matter more to allocation and price. The same benchmark can lead to different commercial outcomes under those two conditions.

What would make the economic case convincing?

The next useful disclosure would connect a production workload to its cost. It should state the model, serving configuration, latency target, utilization, and the accounting boundary. Include hosts, networking, cooling, maintenance, and idle capacity. Report sustained operation, not only the best interval.

For agent workloads, also measure successful task completion. Faster tokens are less valuable if the system needs more retries or fails during a long session. Compare equivalent quality and reliability before attributing savings to the hardware.

Jalapeño has made custom inference hardware a concrete option for OpenAI to test. The next proof is whether it can serve enough useful work, reliably enough, to change what OpenAI pays for that work.