For three years the AI conversation has been about who can build the biggest model. The less glamorous question — what it costs to keep that model answering, millions of times a day, without falling over — is where the money now leaks. llm-d, the open-source inference project started by Red Hat, IBM, Google Cloud, CoreWeave and Nvidia, has shipped its first stable release, and its pitch is aimed squarely at that leak.

The mechanism is worth stating plainly. An AI agent does not ask one question and leave. It keeps long sessions alive, rereads the same documents and spawns parallel helpers that hit the hardware in unpredictable bursts. In that pattern, handling the context can cost more than producing the answer. Every team serving models at scale, says Shadi Saba, CoreWeave’s vice president of engineering for AI/ML, runs into “context that often costs more to manage than the output”. llm-d’s answer is routing: send each request to the server that already holds its cached context instead of recomputing it from scratch.

Fisheye view of a server rack in the Datarmor high-performance computing and AI centre in Brest — Image: Gregory Rocher, Ifremer / Wikimedia Commons (CC BY 4.0)

Version 1.0 is less about new tricks than about making the existing ones safe to run for months. The project already offered cache-aware routing, KV-cache offloading and the splitting of the prefill and decode phases across machines, on top of model servers such as vLLM and SGLang. The new release adds priority and fairness controls so latency-sensitive requests and bursty agents can share one GPU fleet without starving each other, a highly available router for upgrades without downtime, a stable v1 configuration API, and session-aware routing that keeps multi-turn conversations next to their cached state.

The performance figures in the announcement come from the contributors’ own demonstrations, so read them as claims rather than audits. In one, researchers served GLM 5.2, an open-weight model of roughly 753 billion parameters, on widely deployed Nvidia GPUs, peaking above 6.6 million output tokens per minute with up to 3,000 concurrent coding agents; at current cloud rental rates, they say, that came out five to ten times cheaper per token than equivalent commercial API pricing. IBM Research reports an 11.5-fold lower inference cost than an all-proprietary setup on a quantum-error-correction chip design project, with a cache hit rate of about 98%.

More telling is who already leans on it. JetBrains says llm-d serves a significant share of the traffic behind its free Junie Lite coding-agent plan, DigitalOcean runs its serverless inference on it, and Mistral and Roblox describe building their serving layers on the project. Google Cloud is integrating 1.0 into Google Kubernetes Engine across its TPU and Nvidia GPU portfolio, while AMD is bringing it to its Instinct accelerators.

Founded in 2025 and now governed by the Cloud Native Computing Foundation, llm-d counts more than 500 contributors from over 25 organizations. Version 1.0 was announced on October 20, 2026, in San Jose, and is available now as open-source software.

The consequence is a shift in leverage. If the expensive part of AI is serving rather than training, whoever controls the serving layer controls the bill. llm-d is an attempt to make sure that layer stays shared and open, rather than locked inside a single vendor’s API.

Tags: nvidia, AI agents, Large Language Models, IBM, CoreWeave, Red Hat