What Is llm-d? How Distributed LLM Inference Works (2027)

Make AI inference cheaper, and teams will run more of it. Economists call this the Jevons paradox: when a resource gets more efficient to use, total consumption tends to rise. Token demand is already moving that way. DigitalOcean reported that early Inference Engine customers drove their total token consumption up about 30x in 60 days.

Every one of those tokens costs GPU time, and AI agents resend long, mostly repeated context with each turn. Standard load balancing wasn’t built for that pattern, and that’s the gap llm-d (sometimes written as “llm d” or “llmd”) is designed to close.

In a benchmark published in September 2026, IBM Research served up to 3,000 concurrent coding agents from 544 NVIDIA H100 GPUs, peaking at more than 6.6 million output tokens per minute (for context, that’s a lot). The software coordinating those GPUs was llm-d, and its cache-aware routing helped the stack serve 85% of input tokens from cache instead of recomputing them. Let’s break down how llm-d works, what changed in 1.0, and where it fits in production, including how DigitalOcean builds on it to power its own inference products.

Key takeaways:

• llm-d is an open source, Kubernetes-native layer that routes LLM requests across model servers like vLLM and SGLang, based on cache state and load, instead of treating every request the same.
• Its core techniques are prefix-cache-aware routing, prefill/decode disaggregation, tiered key-value (KV) cache offloading, and wide expert parallelism for mixture-of-experts models.
• llm-d 1.0 adds multi-tenant quality of service, zero-downtime operations, a stable v1 configuration API, and agent-aware serving.
• Published llm-d cost results are real but workload-specific. DigitalOcean builds its distributed inference stack on llm-d, and teams can also deploy the open source project themselves on DigitalOcean Kubernetes (DOKS).

What is llm-d?

llm-d is an open source, Kubernetes-native framework for distributed large language model (LLM) inference. It runs above model servers like vLLM and SGLang, deciding which server handles each request, splitting work across GPUs, and reusing cached computation across a cluster. Red Hat launched llm-d in May 2025 with founding contributors CoreWeave, Google Cloud, IBM Research, and NVIDIA. It’s now governed by the Cloud Native Computing Foundation (CNCF) as a Sandbox project.

DigitalOcean, the AI-Native Cloud, uses llm-d behind its Serverless Inference and Dedicated Inference endpoints and contributes features back to the project. DigitalOcean Kubernetes (DOKS) is a tested platform for teams that want to run llm-d themselves.

Why LLM inference needs its own scheduling layer

Most web traffic is short, stateless, and roughly uniform, so load balancing helps spread requests evenly. A standard Kubernetes Service, for example, picks a backend pod at random by default.

LLM traffic doesn’t work the same way:

• Requests vary widely in cost. To a standard load balancer, a one-line chat question and an agent’s long prompt full of tool output look the same, so both can land on the same server, even though the agent’s request takes far more GPU work. Each response also takes up GPU memory for seconds or minutes while the model generates tokens one at a time, which leaves less room for other requests on that server.
• Cached work lives on specific servers. A model stores the intermediate state for each processed prompt (the KV cache) on that server’s GPU. Under round-robin routing across N replicas, a repeated prompt has a 1 in N chance of landing on the server that already cached it.
• AI agents amplify all of it. Agents resend the same instructions, tool definitions, and history on every turn. That’s a cache reuse opportunity, but only if routing sends each turn back to the right server.

How llm-d works

llm-d makes a fleet of GPU servers act like one coordinated service instead of a set of interchangeable machines. It sits between your gateway and your model servers.

Every request passes through the llm-d Router, which picks the best pod (one running copy of the model server) for it. From there, llm-d can split the work across specialized GPU pools (groups of pods dedicated to one job, such as prefill) and cached memory tiers.

The project packages each technique as a well-lit path, which is a tested deployment recipe you can adopt, one at a time.

The llm-d Router: prefix-cache and load-aware scheduling

The llm-d Router (formerly referred to as the inference scheduler) scores each model server pod on queue depth, current load, and KV cache affinity-how much of the incoming prompt a pod has already cached. Sending a repeated prompt back to that pod (called prefix-aware routing) skips work the GPU would otherwise redo.

In a case study coauthored by Tesla and Red Hat engineers, one Llama 3.1 70B deployment on four AMD MI300X GPUs saw 3x more output tokens per second and 2x faster time to first token (TTFT) after enabling prefix-cache routing.

Prefill/decode disaggregation

Every LLM request runs in two phases:

1. Prefill processes the whole prompt and builds the KV cache, which is compute-heavy.
2. Decode generates the response one token at a time, which depends mostly on memory bandwidth.

When both share GPUs, one long prompt can stall everyone else’s token stream.

Prefill/decode disaggregation runs the phases on separate pools that scale independently. The router schedules each request twice, and the KV cache moves between pools over the NVIDIA Inference Xfer Library (NIXL).

The trade-off is network overhead-that’s why llm-d recommends remote direct memory access (RDMA) networking for production. vLLM’s documentation notes that disaggregated prefill “DOES NOT improve throughput” on its own. The payoff is separate control over time to first token and the gaps between tokens.

KV cache reuse and tiered offloading

The KV cache is the model’s working memory for a prompt, and GPU memory is limited.

llm-d can:

• Offload the KV cache to CPU memory and storage
• Track which cached prefixes live where
• Share cache between pods

llm-d’s v0.5 benchmarks found offloading the KV cache to shared storage delivered 13.9x higher throughput at 250 concurrent users on four NVIDIA H100 GPUs, while a GPU-only setup’s throughput collapsed once GPU memory filled.

Wide expert parallelism for mixture-of-experts models

Mixture-of-experts (MoE) models activate only a few of their many expert subnetworks for each token. Wide expert parallelism spreads a large MoE model’s experts across many GPUs and nodes so the cluster serves it as one deployment. llm-d’s reference guide runs DeepSeek-R1 across 32 NVIDIA H200 GPUs.

Where Kubernetes, vLLM, SGLang, and the Gateway API fit

Each layer of an llm-d deployment has a separate job:

• Kubernetes: Runs and scales the pods. On DigitalOcean, that’s DigitalOcean Kubernetes (DOKS).
• Gateway API Inference Extension: An official Kubernetes project that gives network traffic gateways like Envoy and Istio (the components that route incoming requests into a cluster) a standard way to hand routing decisions to an endpoint picker, such as the llm-d Router. DigitalOcean Dedicated Inference runs on it.
• Model servers: vLLM is the default engine, and SGLang and TensorRT-LLM are also supported.

Kubernetes isn’t strictly required, though. Since May 2026, the llm-d router can run on bare metal, Slurm, or Ray, with some specific features still tied to Kubernetes.

What’s new in llm-d 1.0

llm-d spent its first 17 months in 0.x releases, maturing the techniques above while early adopters (including DigitalOcean) put them into production. It also joined the CNCF as a Sandbox project in March 2026.

Version 1.0 is less about new techniques and more about making the existing ones dependable to run day to day.

The IBM announcement highlights four changes:

1. Multi-tenant quality of service: Priority- and fairness-aware flow control keeps latency-sensitive requests and bursty agent workloads from starving one another on a shared fleet.
2. Zero-downtime operations: A backup router waits on standby to take over instantly, so traffic keeps flowing during upgrades. Prefill and decode replicas also roll out together as one unit.
3. A stable foundation: The router configuration API graduates to v1, which the maintainers call “a stable, versioned surface you can standardize a platform on.”
4. Agent-aware serving: Session-aware routing keeps multi-turn conversations close to their cached context, and turn-priority scheduling helps agent harnesses reuse context across turns.

The project now counts more than 500 contributors from over 25 organizations, including DigitalOcean.

Does llm-d lower inference costs?

Using llm-d can lower inference costs, but the savings come from efficiency, not a discount. Higher cache hit rates mean less repeated prefill, and smarter scheduling means fewer idle or overloaded pods, so each GPU serves more tokens. How much that saves depends on your workload.

The most-cited numbers come from IBM Research and the llm-d community:

Figures are as published by each source as of November 4, 2026. They’re vendor-reported results from specific workloads, not guarantees.

Each of these results comes from a specific setup, so you’ll want to weigh these factors before applying any of them to your own workloads:

• Separate the model choice from llm-d: The biggest savings compare self-hosting an open-weight model with paying for an API. llm-d makes the open model practical at scale, but much of the gap comes from the model choice itself.
• Match the workload: Context-heavy agents that resend the same prefixes benefit most. Short, unique prompts give the cache little to reuse.
• Treat endorsements as a starting point: Analyst Jason Andersen cautions that vendor support isn’t necessarily the same as commitment, and DigitalOcean’s Salman Paracha notes that moving workloads between chip architectures is “not super interchangeable today.”

llm-d vs. vLLM, NVIDIA Dynamo, and Kubernetes load balancing

llm-d gets compared with vLLM, NVIDIA Dynamo, and Kubernetes load balancing, but most work at different layers. vLLM runs the model, Kubernetes runs the pods, and llm-d coordinates the engines in between-NVIDIA Dynamo is the one real overlap.

Ultimately, llm-d doesn’t replace vLLM or Kubernetes. It runs on top of both and plugs into the Gateway API Inference Extension like any compliant endpoint picker.

As for Dynamo, TechTarget reports “significant feature overlap between the two open source orchestration tools,” and they share parts: llm-d moves KV cache with NIXL, which NVIDIA contributed through its Dynamo project. The difference comes down to governance and hardware. llm-d is a CNCF project with paths for NVIDIA and AMD GPUs, Google TPUs, and Intel accelerators, while Dynamo is NVIDIA-led, with extras for rack-scale NVIDIA systems.

Who uses llm-d?

llm-d’s 1.0 release post counts more than 25 organizations adopting the project, and IBM’s announcement adds endorsements from Mistral, Cohere, and AMD.

A few examples:

• Roblox. Roblox says llm-d gives it “a Kubernetes-native foundation for disaggregated serving.”
• JetBrains: According to the IBM announcement, “llm-d now runs in production behind our new Junie Lite plan.”
• Red Hat: Delivers llm-d to enterprise customers through Red Hat AI Inference.

llm-d on DigitalOcean

DigitalOcean, the AI-Native Cloud, is one of llm-d’s production adopters. DigitalOcean built its distributed inference stack on llm-d because the project natively supports mixed GPU types. That support made it possible to bring the Kimi K3 model to DigitalOcean’s Inference Engine on both AMD Instinct MI350X and NVIDIA HGX B300 GPUs.

llm-d also sits behind DigitalOcean’s managed inference products. On Serverless Inference, requests flow through the DigitalOcean Inference Gateway, which DigitalOcean engineers describe as “a fork of llm-d.” The endpoint picker scores each vLLM pod on how much of the prompt it has already cached and how full its KV cache is. Dedicated Inference runs on the same building blocks: vLLM and llm-d on DOKS, with the Gateway API Inference Extension handling inference-aware routing.

How to try llm-d

Try llm-d for yourself with our tutorial on how to deploy llm-d on DigitalOcean Kubernetes.

Its automated scripts handle most of this setup in about 15 to 20 minutes:

1. Create a DOKS cluster with a GPU node pool (RTX 4000 Ada, RTX 6000 Ada, or L40S). The setup script also installs the NVIDIA device plugin.
2. Run the deployment script with your Hugging Face token to launch llm-d’s prefill, decode, and gateway services, plus Redis for KV cache storage.
3. Send a test request to the OpenAI-compatible endpoint to confirm the model is serving.
4. Optionally, add Prometheus and Grafana dashboards to track time to first token, KV cache hit rate, and GPU use.

Many teams will skip the cluster work entirely, though. Andersen told TechTarget that enterprises are most likely to get llm-d in packaged form from a cloud provider or software vendor. On DigitalOcean, Serverless Inference on the DigitalOcean Inference Engine already routes requests through the llm-d-based Inference Gateway, and Dedicated Inference offers managed orchestration and scaling without Kubernetes complexity for sustained workloads.

llm-d FAQ

What is disaggregated prefill and decode in LLM inference?

Disaggregated prefill and decode in LLM inference is the practice of running the two phases of LLM inference on separate GPU pools. Prefill processes the prompt and builds the KV cache, and decode generates output one token at a time. Each pool scales for its own bottleneck, at the cost of moving the KV cache over a fast network.

Which inference platforms offer prompt caching?
DigitalOcean Serverless Inference reuses cached prompt prefixes through the DigitalOcean Inference Gateway, which DigitalOcean engineers describe as a fork of llm-d. Its endpoint picker scores each vLLM pod on how much of the prompt it has already cached and how full its KV cache is, so a repeated prefix goes back to the pod that already holds it. That skips prefill work the GPU would otherwise redo.

How does caching reduce token costs for long agent and coding sessions?
Agents resend the same instructions, tool definitions, and history on every turn, so caching those shared prefixes avoids repeating the same prefill work. llm-d’s prefix-aware routing sends each turn back to the pod that already holds that context, and 1.0 adds session-aware routing for multi-turn conversations. Savings depend on the workload: an IBM Research chip-design benchmark reported about a 98% prefix-cache hit rate, while short, unique prompts give the cache little to reuse.

How do you get low time to first token for a coding agent with long sessions?
Send each turn to a server that already holds the session’s cached context so the GPU skips prefill work it would otherwise repeat. On an agentic coding workload on Ironwood TPUs, Google reports that GKE Inference Gateway, which runs the llm-d router, cut median TTFT by more than 85% compared with a plain Kubernetes Service. DigitalOcean Serverless Inference applies the same idea through its llm-d-based Inference Gateway, which scores each vLLM pod on cached prompt content.

Which AI platforms support both AMD and NVIDIA GPUs for inference?
DigitalOcean built its distributed inference stack on llm-d because the project natively supports mixed GPU types. That support let DigitalOcean bring the Kimi K3 model to both AMD and NVIDIA platforms quickly. llm-d itself has deployment paths for NVIDIA and AMD GPUs, Google TPUs, and Intel accelerators.

Run llm-d-powered inference on DigitalOcean

The DigitalOcean Inference Engine gives you the benefits of llm-d without building and maintaining the cluster yourself. Serverless Inference routes requests through the llm-d-based Inference Gateway, while Dedicated Inference runs vLLM and llm-d on DigitalOcean Kubernetes for sustained workloads.

Key features include:

• Serverless Inference for real-time APIs and agents, with built-in observability for tokens, latency, errors, and spend
• Dedicated Inference with dedicated GPU endpoints, Bring Your Own Model (BYOM) deployment, and managed orchestration and scaling without Kubernetes complexity
• Cache-aware routing that scores each pod on how much of a prompt it has already cached
• One OpenAI- and Anthropic-compatible endpoint across serverless, batch, and dedicated workloads

Get started with the Inference Engine ->

Hippocratic AI, which builds AI agents for patient care, reported 2x production inference throughput and a 2x reduction in prefill latency on long-context sessions on DigitalOcean’s AI-Native Cloud, supported by a cache-aware routing architecture that maximizes KV-cache hit rate.

Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.

Product and project information in this article is based on DigitalOcean and llm-d documentation as of November 4, 2026. Features, availability, and published results may change.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.