Continuous Batching Mechanics: The Scheduler Behind vLLM, TGI, and SGLang

• Continuous batching updates the batch every decoding step rather than once per batch. This eliminates the head-of-line blocking and idle slots that make naive static batching inefficient, and it is the scheduler design shared by vLLM, TGI, and SGLang.

• The freedom to admit work every step requires two supporting mechanisms. Preemption (recomputation or swapping) handles the case where the KV cache runs out mid-generation, and paged KV-cache management (PagedAttention) makes that memory stretch much further by eliminating fragmentation.

• Every major system shares this foundation, and increasingly a shared toolkit. They all face the same hard problem, integrating a new request’s compute-heavy prefill into the stream of ongoing decodes, and the main techniques for it (chunked prefill, prefix caching, paged memory) are now common to vLLM, TGI, and SGLang alike. The differences are mostly in emphasis and defaults.

Generating text with a transformer happens in two phases. The first phase is prefill. The model ingests the entire prompt in a single forward pass, computing attention across all its tokens at once and building up the key-value (KV) cache, the stored keys and values for every token that later tokens will attend to. Prefill is compute-bound. There is a lot of parallel math to do, and the GPU’s arithmetic units are the bottleneck. The second phase is decode. The model generates the response one token at a time, and each new token requires its own forward pass. Decode is memory-bound. Each step does only a sliver of computation but must stream the model’s entire weight set and the growing KV cache through memory to do it.

The KV cache is the resource that will dominate every decision later in this article. Every token the model processes, either the prompt or generated, adds an entry to the cache, and that entry lives in GPU memory for as long as the request is active. A long conversation means a large cache. Many concurrent requests mean many caches competing for the same finite pool of memory. The cache is what lets the model avoid recomputing attention over the whole sequence at every step, but it is also what makes memory, not compute, the thing you usually run out of first.

When running one request at a time, the GPU is almost idle during decode. A single decoding step moves gigabytes of weights through the chip to produce one token’s worth of arithmetic, leaving the compute units mostly unused.

If one request wastes the GPU during decode, the intuitive fix is to run many requests together. Because decode is memory-bound, this works well. Once you have paid to stream the model’s weights through the chip for one sequence, adding more sequences to the same step is nearly free. This is static batching (sometimes called request-level batching).

It collects a group of N requests and pads every sequence to the length of the longest one because tensors need a uniform shape. It then runs forward passes over the whole batch, step by step, until every sequence in it has finished generating. Then it returns all N responses together and starts over with the next group.

But notice that once the batch starts, it commits to a fixed set of requests, processed together, from start to finish. That single decision, freezing the batch’s “membership” for its entire lifetime, is where every problem with static batching comes from.

The first failure is head-of-line blocking. A static batch runs until its longest member finishes. If a request that needs 10 tokens is unlucky enough to share a batch with one that needs 500, the short request is held. It finished long ago but cannot be returned until the whole batch drains. Latency for short requests becomes hostage to the worst-case length in their batch.

The second and third failures compound this. As sequences finish early, their slots in the batch go empty, but they stay allocated, so you spend the rest of the batch’s life computing on a half-empty tensor. Then there is no mid-flight admission. A request that arrives one step after the batch starts must wait for the entire batch to finish before it can begin. Under steady traffic, this produces latency spikes.

The root cause underneath all three problems is that static batching makes its scheduling decision once per batch, but the workload changes every token.

The fix, introduced by the Orca system in 2022, is to move the scheduling decision from the batch to the individual token. Instead of picking a batch and running it to completion, the scheduler runs before every single forward pass and re-decides which requests are in the batch. The batch is no longer a frozen group. It is a living roster that changes with each token. This is continuous batching, also called iteration-level or in-flight batching.

The engine runs a loop. The scheduler picks the current batch, the model runs one forward pass over it, the sampled tokens are appended to each sequence, and the scheduler updates its bookkeeping, retiring any request that just emitted an end-of-sequence token and immediately freeing its KV cache. Because that freed capacity is available at the very next step, waiting requests can be admitted right away. Head-of-line blocking disappears, because a short request leaves the instant it is done rather than waiting for its batch-mates. Idle slots disappear, because freed capacity is backfilled immediately. And the mid-flight admission problem disappears, because a newly arrived request joins within a step or two instead of waiting for a whole batch to drain.

Continuous batching introduces two new problems. First, a newly admitted request cannot simply start decoding. Its prompt must be prefilled first, a compute-heavy forward pass that can stall everyone else’s decoding if you are not careful. Second, because you are free to admit requests every step, you can overcommit the KV cache and run out of memory mid-generation.

A new request’s prefill is a big, compute-bound burst, while the decode steps keeping existing requests alive are a stream of small, memory-bound updates. Each engine has to fit that burst into the ongoing decode stream without stalling it.

The three answers to this problem are prefill-first, chunked prefill, and disaggregation. Prefill-first pauses the decodes to run the prefill. It is simple, but it produces a visible hitch in every other request’s output whenever a new one arrives. Chunked prefill slices the long prefill into smaller pieces and weaves them into the decode steps, so the new request makes progress without any single step being dominated by its prefill. Disaggregation goes further, running prefill and decode on entirely separate pools of GPUs and transferring the KV cache between them, so the two phases never compete at all.

Chunked prefill has become the default in most modern engines because it smooths latency without extra hardware. Disaggregation is a newer, heavier approach that pays off mainly at very large scale.

The second problem is memory. Because continuous batching keeps admitting requests as long as there is room, and because every active request’s KV cache grows with each token it generates, a batch that fit a moment ago can overflow the GPU’s memory pool. To solve this, you need to use the memory you have far more efficiently, and, when that is not enough, take some back.

Early systems gave each request a single contiguous slab of memory sized for its maximum possible length, which fragmented the pool and left some of it wasted on over-reservation and gaps. PagedAttention, introduced by vLLM, borrows the operating-system idea of virtual memory and paging. The KV cache is broken into small fixed-size blocks, and a request’s logically contiguous sequence of blocks is mapped, through a per-request block table, onto physical blocks that can sit anywhere in GPU memory. The request sees a clean, contiguous view, but the physical reality is scattered blocks mixed with those of other requests and with free space. This eliminates fragmentation, lets memory be allocated on demand as sequences grow, and makes both preemption and prefix sharing cheap.

Paged memory stretches the pool a long way, but under enough load it can still fill. When it does, the scheduler needs the authority to take memory back, evicting or freezing a request it previously admitted so the rest can continue. This is called preemption, and there are two ways to do it. Recomputation drops the request’s KV cache to free memory instantly, then rebuilds it later by re-running prefill: cheap on memory, but paying again in compute. Swapping copies the KV blocks out to CPU memory and back when the request resumes, preserving the computation but paying in Peripheral Component Interconnect Express (PCIe) bandwidth and host RAM. vLLM defaults to recomputation, but both are widely used. Which request to preempt is a policy choice. Engines typically evict the newest or lowest-priority requests first and restore them once memory frees up.

The techniques above, chunked prefill, prefix caching, and paged memory, have been largely combined into a shared toolkit that the mature engines all implement. What still differs is less about which techniques an engine has and more about where each idea originated, what each engine emphasizes by default, and how it is built.

vLLM is the memory-first baseline and the reference implementation most others are measured against. PagedAttention is its signature contribution, and its scheduler shows the same lineage. The older V0 engine used the prefill-first approach, with the latency hitches that come with it, while the rearchitected V1 engine makes chunked prefill the default and simply decides how many tokens each request advances per step.

TGI (Hugging Face’s Text Generation Inference) has the same core techniques, chunked prefill and prefix caching included, but its distinctive choice is structural. It splits serving into two parts. First, a fast router, written in Rust, handles request queuing and all the batching decisions. Second, separate workers do nothing but run the model’s forward passes. Its scheduling knobs, such as max_batch_total_tokens and waiting_served_ratio, live in that router and control how aggressively it fills each batch. Hugging Face has leaned especially hard on prefix caching, reporting that TGI’s handling of it makes the server strong on very long, repeated prompts. The origin here is production serving.

SGLang pushed prefix reuse the furthest. Its RadixAttention organizes cached prefixes in a tree so that requests sharing a common prefix, such as a system prompt, a block of few-shot examples, or a set of tool definitions, reuse each other’s KV cache instead of recomputing it. Prefix caching is common across engines now, but SGLang’s radix-tree version was an early and influential form of it, and the engine remains especially strong on workloads with heavy prompt overlap, such as agents and structured-generation pipelines. Vendor-optimized engines push in another direction entirely. NVIDIA’s TensorRT-LLM compiles the model ahead of time and does “in-flight batching,” winning on raw speed at the cost of flexibility.

Is continuous batching the same thing as PagedAttention?

No. Continuous batching is a scheduling technique (which requests run each step). PagedAttention is a memory-management technique (how the KV cache is stored). They are complementary. PagedAttention is what makes the memory go far enough for continuous batching to admit many requests at once, but you can implement one without the other.

Does continuous batching hurt per-request latency?

It can, indirectly. Continuous batching improves throughput and keeps the GPU full, but larger, fuller batches mean each step does more work and memory pressure rises. Tuning the batch-size and token budgets, and using chunked prefill to avoid prefill stalls, is how you balance throughput against time-to-first-token and inter-token latency.

The single-GPU version of this problem is largely solved. Continuous batching keeps the hardware busy, paged memory keeps the cache from overflowing, and the major engines have settled on that design. Most of the current work has moved outward, across many GPUs, and downward, into the decode step itself.

Outward means running prefill and decode on separate pools of hardware (as in NVIDIA’s Dynamo) and treating the KV cache as a tiered resource shared across requests and machines. The downward push attacks the decode bottleneck directly. Techniques like speculative decoding and KV-cache quantization squeeze more useful work out of each memory-bound step.

• The LLM Inference Optimization: Quantization to Speculative Decoding Part 1
• The MoE-ification of the Open Model Ecosystem, and What It Means for Your Inference Bill
• Continuous Batching vs. Static Batching in LLM Inference