Prefill|Decode Disaggregation: Why Production LLM Inference Is Splitting Onto Separate Hardware

Prefill/decode disaggregation is a serving architecture that runs the two phases of LLM inference on separate GPU pools. Prefill (processing the input prompt) runs on one set of GPUs, and decode (generating the response token by token) runs on another. When prefill finishes, the model’s memory of the prompt (the KV cache) is sent over the network to a decode GPU, which then generates the response.

The split exists because the two phases stress different hardware limits: prefill is bound by compute, decode by memory bandwidth. Separating them stops long prompts from interfering with everyone else’s token generation. The price is extra hardware, network transfers, and operational complexity. Production systems using it today include Moonshot AI’s Mooncake, DeepSeek’s inference clusters, and NVIDIA’s Dynamo framework.

We’ve written before about the prefill tax, but here’s the short version in case you’re arriving fresh. Every LLM request has two phases: the model first reads and processes your entire prompt (prefill), then starts generating its answer one token at a time (decode). That first phase is the tax. It’s latency you pay before a single word comes back, and it grows with every token in your prompt.

The deeper problem, the one the prefill tax only hints at, is that the two phases want completely different things from your hardware.

Prefill happens when the model reads your prompt. It processes every token at once, in parallel, in one big burst of matrix math. GPUs love this; it’s exactly what they were built for. The compute cores run hot, the memory system is barely stressed, and the job finishes fast relative to its size.

Decode happens when the model writes its answer. It generates one token at a time, and for each new token, the GPU has to re-read a huge amount of stored data: the model’s weights, plus everything it remembers about the conversation so far (the KV cache). The computation for each token is small. Most of the time is spent moving data to and from memory, while the GPU’s compute cores sit mostly idle.

So the two phases need very different things one starves for compute, the other for memory bandwidth and we’ve been asking a single GPU to deliver both.

Here is how it works in real life. Modern serving engines batch many users’ requests together on the same GPU pool to keep utilization high. Decode steps are small and quick, so the GPU cycles through everyone’s next token over and over, and things feel smooth. Then a request shows up with a long prompt.

Say someone pastes a 60-page contract into your document-analysis app. The GPU drops everything to process that long prompt, and every other user has to wait. Their tokens just stop streaming. The serving world calls this head-of-line blocking, and it’s why a chatbot that feels fast in testing becomes slow in production the moment real traffic shows up with its messy mix of short questions and giant pasted documents.

This isn’t an edge case; it’s the default shape of production traffic. It’s arguably the open problem in inference serving right now, and every team running LLMs at scale hits it. The software fixes people reach for first (chunked prefill, smarter schedulers, priority queues) soften the blow, but they all run into the same ceiling: as long as prefill and decode compete for the same GPU, one of them loses.

The fix is surprisingly simple. Stop making them share. Disaggregated serving (you’ll also see it called P/D disaggregation) splits inference across two separate pools of GPUs:

• A prefill pool that does nothing but process incoming prompts. The prep kitchen.
• A decode pool that does nothing but generate tokens. The line.

A request arrives, a prefill GPU processes the prompt at full speed, and the request hands off to a decode GPU that streams the response out token by token. Long prompts no longer stall anyone’s generation, because the machines doing generation never see a prompt. Each pool can also be sized, scheduled, and even hardware-selected for its actual job: more raw compute for prefill, more memory bandwidth for decode.

There’s a catch, though, and it’s the part that makes this an engineering problem rather than a config flag.

When the model finishes reading your prompt, it has built up a working memory of everything it read. That’s the KV cache. In a traditional setup, this memory just sits on the same GPU that will generate the response. In a disaggregated setup, it has to physically travel across the network from the prefill machine to the decode machine before a single token can be generated. For a long prompt on a large model, the KV cache can run into the tens of gigabytes. Moving it is not a rounding error. It’s the new bottleneck you’ve signed up to manage, and any writeup that treats the transfer as a solved side detail is selling you something.

It’s worth being clear about what disaggregation gives up. The standard approach today, what you get with vanilla vLLM on a unified GPU pool, is continuous batching: every GPU handles both phases, and the scheduler weaves new prompts and ongoing generations together as tightly as it can. It’s simple, it’s mature, and for many workloads it’s excellent. Every GPU stays busy, nothing crosses a network, and there’s one system to operate instead of three (prefill pool, decode pool, and the transfer layer between them). Disaggregation deliberately trades that simplicity for predictable latency. Whether the trade is worth it depends entirely on your workload, which we’ll get to.

This isn’t a whiteboard idea. Several real systems have shipped it, and they’re worth knowing by name, partly because they’re the reference points for the whole field and partly because their differences tell you where the hard parts are.

DistServe (Zhong et al., OSDI 2024). The academic paper that introduced the idea explains it most clearly. The core insight: when prefill and decode share a GPU, you can’t optimize time-to-first-token and time-per-generated-token independently. Improving one degrades the other. Separate them and you can tune each pool for its own latency goal. DistServe reported serving several times more requests within the same latency targets compared to colocated serving.

Mooncake is the serving platform behind Kimi, the assistant from Moonshot AI, and it’s the most compelling production evidence to date. Mooncake goes further than just splitting the pools. It treats the KV cache itself as the center of the whole design, pooling spare CPU memory and SSD storage across the cluster into a giant shared cache, so repeated or overlapping prompts don’t need to be prefilled from scratch at all. Moonshot published the architecture and reported that it is handling real traffic at a serious scale. This is not a lab demo.

DeepSeek disclosed in its published technical report that production serving for DeepSeek-V3 and R1 runs prefill and decode on separate GPU clusters, with different parallelism strategies for each phase. When one of the most cost-efficient inference operations in the world tells you how it serves its own models, that’s worth reading twice.

NVIDIA Dynamo is the clearest signal that this is going mainstream. Dynamo is NVIDIA’s open-source distributed serving framework, the successor to Triton for LLM workloads, and disaggregated serving is its headline feature rather than a footnote. It sits above engines like vLLM, TensorRT-LLM, and SGLang, routes prefill and decode to dedicated worker pools, and ships a dedicated transfer library (NIXL) whose entire job is moving KV caches between machines as fast as the hardware allows. NVIDIA has reported throughput gains of up to several times on large models under latency constraints. vLLM itself has been building native prefill/decode disaggregation support too, so the capability is arriving in the default open-source stack.

These frameworks are still relatively new, and there isn’t much practical guidance available yet. Most of the teams using disaggregated inference in production are large organizations building their own infrastructure and scheduling systems. That’s also why you won’t find many detailed articles from cloud providers, many are still exploring or rolling out these capabilities.

The hardest part is this: once you disaggregate, interconnect bandwidth decides whether the whole thing works. The KV cache for a single long-context request on a 70B-class model can reach tens of gigabytes. Push that over ordinary datacenter Ethernet and the transfer takes long enough to wipe out everything you gained by splitting the phases. Push it over NVLink, InfiniBand, or high-bandwidth RDMA-capable Ethernet and the handoff can hide inside time the decode GPU was going to spend anyway. This is why every serious system on the list above invests heavily in the transfer layer (Mooncake’s cache-centric design, Dynamo’s NIXL), and why “just split the pools” without thinking about the network in between usually produces a slower system than the one you started with.

Disaggregation isn’t always the right choice

Many articles make disaggregated inference sound like the obvious next step. The reality is more nuanced.

First, it can require more hardware. In a traditional setup, every GPU can handle both prefill and decode work, so resources are shared more efficiently. With disaggregation, some GPUs are dedicated to prefill while others only handle decoding. If your workload changes throughout the day, for example, long documents in the morning and short chat requests in the evening, one group of GPUs may sit idle while the other becomes overloaded. That means you may end up paying for extra capacity that isn’t always being used.

Second, it makes the system more complex. Instead of a request staying on one machine from start to finish, it now moves between multiple machines. That creates more opportunities for things to go wrong, such as network delays, overloaded decode workers, or requests getting interrupted if a machine fails. As a result, monitoring, debugging, and operating the system becomes more challenging.

Finally, the benefits only appear at very large scale. If you’re serving a small or medium-sized application, disaggregation often won’t make a noticeable difference. Modern inference engines like vLLM already use techniques such as continuous batching and chunked prefill to improve GPU utilization without splitting the workload across separate GPU pools. For many teams, these optimizations provide most of the performance gains while keeping the system much simpler.

Our view is straightforward: disaggregated inference is an important direction for the future of large-scale AI serving, especially for organizations running thousands of concurrent requests. But that doesn’t mean every team should adopt it today. For many deployments, a well-optimized unified inference system is easier to operate, costs less, and delivers excellent performance. The best approach is to understand when disaggregation solves a real problem and only introduce it when you’ve reached that point.

A quick note before we start: this section describes how a disaggregated setup could work on DigitalOcean infrastructure. These are not benchmark results, and this is not a DigitalOcean product.

Conceptually, the mapping is clean. DigitalOcean’s GPU Droplets offer H100 and H200 configurations. The H100 delivers the raw compute that prefill uses. The H200 pairs similar compute with substantially more and faster memory — 141 GB of HBM3e at roughly 4.8 TB/s versus the H100’s 80 GB at about 3.35 TB/s — which is the profile decode wants: more room for KV caches, faster reads per generated token. A prefill pool on H100s feeding a decode pool on H200s is a workable design, and honestly, it’s the textbook illustration of why heterogeneous pools are one of disaggregation’s most interesting promises.

The determining factor, as always, is the wire between the pools. DigitalOcean’s multi-node GPU configurations connect 8-GPU nodes over a dedicated high-speed fabric with RDMA support, and Bare Metal GPU offerings list east-west bandwidth up to 400 Gbps with GPU interconnect speeds up to 3.2 Tbps — the class of networking that KV cache transfer needs. (DigitalOcean’s own serving platform has begun using this pattern: the Inference Tax post describes disaggregated serving over a high-speed RoCE network as part of the Serverless Inference stack.) The back-of-envelope math any architect should run: take your typical KV cache size per request (it grows with context length, and with the model’s size and architecture — a 70B-class model stores roughly a third of a megabyte per token), divide by realistic sustained bandwidth between nodes, and compare the result against your time-to-first-token budget. If the transfer fits comfortably inside the latency you were already paying for prefill, disaggregation has room to work. If it doesn’t, no orchestration framework will save you. Frameworks like vLLM’s disaggregated mode or NVIDIA Dynamo are the natural software layer on top; both are open source and both run on standard NVIDIA hardware of exactly this class.

Disaggregated inference isn’t something every AI application needs. Three properties of your workload do most of the deciding.

Traffic volume: the win appears when GPUs are saturated

The biggest advantage of disaggregation appears when your GPUs are constantly busy.

If you’re running a small deployment with only a few GPUs and requests arrive occasionally, a traditional setup, where every GPU handles both prefill and decode, is usually the better choice. It’s simpler, cheaper, and easier to manage.

However, if you’re serving thousands of users at the same time and your GPUs are always processing requests, prefill and decode start competing for GPU resources. In that case, separating them into dedicated GPU pools can improve overall throughput and reduce latency.

Latency consistency: interactive applications feel the interference most

Not every application needs responses in a fraction of a second.

For workloads such as batch processing, document analysis, or internal tools, users usually don’t notice if one request takes a little longer than another. Small delays aren’t a major concern.

But for interactive applications, such as chatbots, AI coding assistants, or voice assistants, users expect responses to begin almost immediately and continue streaming smoothly. A single long prompt can delay every other request sharing the same GPU, creating an inconsistent user experience.

If predictable latency is a priority, disaggregation becomes much more valuable.

Prompt length: long-context workloads gain the most

Prompt length is often the deciding factor.

If your application mostly handles short prompts, the prefill phase is quick, so there’s little benefit in separating it from decoding.

On the other hand, applications that process long documents, perform Retrieval-Augmented Generation (RAG), or maintain lengthy conversation histories spend much more time in prefill. Since prompt lengths can vary dramatically from one request to another, these workloads benefit the most from disaggregation.

If your workload sits somewhere in the middle — real traffic, moderate latency sensitivity, mixed prompt lengths — try the simpler optimizations first. Chunked prefill, better scheduling, and continuous batching often deliver most of the improvement without requiring you to redesign your serving architecture. Disaggregation is the step you take when those run out of headroom on all three signals at once.

Disaggregated inference isn’t a silver bullet. It’s just one technique for improving LLM serving.

Other optimizations solve different bottlenecks. For example, speculative decoding speeds up the decode phase by generating tokens more efficiently, while quantization reduces memory usage and continuous batching improves GPU utilization.

In practice, production inference systems combine several of these techniques rather than relying on just one. Think of disaggregation as another tool in your optimization toolbox — one that’s especially useful for large, latency-sensitive workloads, but unnecessary for many smaller deployments.

1. Zhong, Y. et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
2. Qin, R. et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. Moonshot AI.
3. DeepSeek-AI (2024). DeepSeek-V3 Technical Report — Section 3.4 covers the disaggregated prefill/decode deployment.
4. NVIDIA (2025). Introducing NVIDIA Dynamo: A Low-Latency Distributed Inference Framework. NVIDIA Technical Blog.
5. NVIDIA. Dynamo disaggregated serving documentation.
6. DigitalOcean (2026). The Inference Tax: How Prefix-Aware Routing Eliminates the Hidden Cost of LLMs at Scale.
7. DigitalOcean. GPU Droplets · What is the NVIDIA H200 · Multi-node GPU configuration docs · Bare Metal GPU features.