A Guide to Multi-Node Tensor and Pipeline Parallelism

Single-node serving stops being the right default once the entire serving footprint (model weights, KV cache, and runtime overhead) can’t comfortably fit on a single node with sufficient memory headroom. There’s an important distinction here: many teams approximate only the memory needed for model weights and assume the deployment is possible. In production, that calculation is incomplete.

A serving replica requires GPU memory reserved for model weights, KV cache, activation buffers, NCCL communication buffers, scheduler state, CUDA graphs, quantization metadata, speculative decoding buffers (if enabled), and memory reserved for fragmentation and safety margin. For deployments of many 70B-class models on modern high-memory 8-GPU systems, the model weights may easily fit on a single node. However, KV cache can become the dominant memory component for workloads with long prompts, large conversation history, high concurrency, or large output lengths. When deploying 100B+ dense models or large MoE models, it may not even be possible to store the model weights on a single node.

image

A good rule of thumb is to avoid designing around 100% of the nominal GPU memory. If the deployment appears feasible on paper but leaves little or no memory margin, it will probably be unstable under bursty traffic patterns, large prompts, or framework overhead. The practical threshold for multi-node inference isn’t a specific number of parameters, but rather the point at which the total serving footprint can no longer fit comfortably on a single node with reliable production headroom.

For this reason, multi-node parallelism is considered part of the same decision space as speculative decoding and prefill/decode disaggregation. Speculative decoding changes how tokens are generated. Prefill/decode changes where different inference phases run. Tensor, pipeline, and data parallelism determine how the model and workload are physically distributed across GPUs and nodes. These are all closely related levers you can tune in the same production inference system. Adjusting one will often impact the effectiveness of the others.

The default naive scaling approach is to continue to scale up the TP degree until the model fits. The logic is simple. If 8 GPUs are better than 4, 16 should be even better. This intuition often fails in practice with tensor parallelism because tensor parallelism is communication-intensive.

Tensor parallelism shards the computation of each transformer layer across many GPUs. After each GPU computes its assigned subset of the layer, it must perform collective communication to synchronize intermediate results. The collective could be all-reduce, all-gather, or reduce-scatter depending on the implementation. This synchronization happens layer after layer throughout the prefill and decode stages.

Within a modern multi-GPU node, this communication is relatively inexpensive. NVIDIA’s Hopper line of GPUs offers up to 900 GB/s of fourth-generation NVLink bandwidth per GPU. NVSwitch provides a mechanism for GPUs on the same node to communicate with one another at high bandwidth. Under these conditions, TP will scale well since all of the collective communication remains within a low-latency, high-bandwidth interconnect.

Once tensor parallelism spans nodes, collective communication leaves the intra-node fabric and traverses the external network. In decode, where per-token collectives are small and frequent, the added inter-node latency can quickly dominate and erase the benefit of more GPUs.

The diagram below illustrates a simplified ring all-reduce communication showing the reduce-scatter and all-gather phases, along with the communication cost model for distributed tensor parallelism.

As a practical starting point, the TP degree should generally not exceed the size of the fastest local GPU interconnect domain. On an 8-GPU NVSwitch system, that often means TP=8. Expanding to TP=16 across two nodes creates a fundamentally different communication topology rather than simply a larger version of the same system.

Pipeline parallelism addresses a different concern. Rather than splitting every layer onto every GPU, PP partitions ranges of layers onto pipeline stages. Stage 1 computes early layers and forwards activations to stage 2, and so on until the final stage emits logits.

This results in a more favorable communication profile: activations are only shuffled at stage boundaries rather than through every layer’s TP collective. This makes PP more appealing to use across nodes, particularly if the network fabric is materially slower than intra-node NVLink/NVSwitch.

PP isn’t without cost. The major downside is pipeline bubbles: idle periods where some stages wait while others work. During prefilling, one request must propagate through the first stage before later stages can perform any useful work. While decoding, each token still completes before the next can start. If the runtime doesn’t use micro-batching, “chunked” prefill, and smart scheduling, then pipeline stages will spend expensive GPU time idle.

The right TP/PP/DP split depends on four inputs: model size, interconnect, latency SLA, and traffic pattern. The table below gives starting configurations rather than universal answers.

Start by keeping tensor parallelism within the fastest local GPU interconnect domain. Leverage pipeline parallelism when the model is too large to fit on a single node. Consider introducing data parallelism once the per-replica topology is efficient. These rules of thumb are intended to provide good starting points, rather than one-size-fits-all optimal configurations. Always benchmark each configuration with production-like workloads.

Suppose you have a 70B+ dense model serving long-context traffic. The model weights may fit in memory on a single high-memory 8-GPU node, but high concurrency and long prompts may use enough KV cache to force either lower concurrency, quantization, or a multi-node layout.

DigitalOcean offers H100x8 GPU Droplets with 640 GB of GPU memory and H200x8 GPU Droplets with 1,128 GB of GPU memory. Remember that those are capacity facts, not serving-performance guarantees.

Let’s suppose that the workload requires more memory headroom than a single H100x8 node can offer. You could run TP=16 across two 8-GPU nodes. This configuration shards each layer across all GPUs, but also requires TP collectives to span the node boundary. That layout seems attractive because it makes the model fit, but it risks paying communication cost at every layer and every decode step.

You could also run TP=8, PP=2. Each node would contain a contiguous set of layers, keeping tensor parallelism mostly local to each node. Most communication crosses the node boundary mainly at the pipeline boundary. This layout often has better communication locality, but incurs pipeline bubbles and may impact TTFT.

Assuming you have enough capacity, you could also run TP=4, PP=2, DP=2 across four 8-GPU nodes. This configuration creates two independent replicas, each spanning two nodes. You might see higher aggregate throughput and lower queueing, but only if TP=4 gives enough memory per stage and avoids underutilizing the GPUs.

Each of those expected curves is not linear. Going from TP=8/PP=1 to TP=16/PP=1 will likely worsen latency if cross-node collectives are the bottleneck. Going from TP=16/PP=1 to TP=8/PP=2 will probably improve inter-token latency but worsen TTFT if the pipeline schedules poorly. Moving to TP=4/PP=2/DP=2 may improve aggregate throughput but at the cost of reducing per-replica capacity.

A meaningful comparison of these layouts would measure TTFT, p95 inter-token latency, output tokens per second, GPU utilization, network utilization, and cost per million tokens across all three. Without those curves, “TP=16 is faster” or “PP=2 is better” is not an engineering conclusion, and it is an untested topology assumption.

Network topology can become more important than raw GPU count as tensor parallelism scales. A smaller deployment that fits on a single node with adequate local interconnect can outperform a larger deployment with poor cross-node communication. This is particularly true of latency-sensitive decode, which incurs synchronization penalties on every token.

DigitalOcean’s general Droplet documentation mentions that GPU Droplets have maximum public and private bandwidth limits. It’s dedicated multi-node GPU documentation notes standard public/private interfaces and dedicated GPU-to-GPU interfaces starting at eth2. DigitalOcean’s Kubernetes GPU documentation also mentions using high-speed networking across multiple 8-GPU H100 worker nodes with 8× Mellanox 400GbE interfaces. Those infrastructure details matter because they define which deployment layout is even plausible.

TP-heavy layouts spanning nodes require a fabric that can support frequent collectives. PP-heavy layouts can more easily tolerate infrequent cross-node activation transfer, so long as scheduling keeps bubbles under control. DP-heavy layouts avoid sending models across nodes during inference but require enough memory for each full replica.

The practical question to ask of a benchmark is therefore not “Can this provider provision 16 GPUs?” It is, “Can this provider’s multi-node fabric keep up with the communication pattern imposed by this TP/PP split at my traffic levels?”

The choice of serving engine influences the right split. vLLM exposes tensor and pipeline parallelism configurations and enables distributed inference across GPUs and nodes. Ray Serve also supports cross-node TP and PP support for distributing LLM inference across GPUs and nodes.

SGLang frames PP as useful for distributing layers across nodes and solving serving constraints that arise as context windows and model sizes increase. NVIDIA’s Megatron lineage also remains especially relevant because many production organizations inherit both its parallelism vocabulary and implementation details; Megatron Bridge outlines configurable tensor, pipeline, and data parallelism options as “composable” strategies.

This means that for AI-native engineers, the choices will include both engine selection and topology selection. You can run the same model on the same GPUs and observe different behavior depending on the scheduler, the micro-batching logic, prefill chunking, communication overlap, and KV-cache strategies.

The rules for determining the right multi-node inference layout start with three questions. (1) Does model + KV cache fit on a node with production headroom? (2) What interconnect provides the bandwidth on the communication path created by your parallelism strategy? (3) Is your workload latency-sensitive, throughput-heavy, or long-context dominated?

If the model fits on a node, you can keep TP inside that node and deploy DP replicas for throughput. If the model doesn’t fit, evaluate quantization, KV-cache policy, and context length limits first. If you still need to go multi-node for serving, you can use TP inside nodes, and PP across nodes as a starting point. Avoid using TP across nodes unless you’ve empirically validated the fabric under that collective pattern. If the SLA is strict on TTFT, try to keep PP as shallow as possible. If the workload is batch-like or throughput-heavy, PP becomes easier to justify because bubbles can be amortized.

This is the missing decision logic in most public LLM inference content. TP, PP, and DP are not abstract parallelism labels. They are cost-shaping choices. TP decides how often GPUs must synchronize inside layers. PP decides how model stages communicate and how much idle time the pipeline absorbs. DP decides how many complete replicas can serve independent traffic. The production answer is the configuration that maps those costs onto the hardware topology with the least damage to latency, throughput, and budget.

Seen this way, multi-node tensor and pipeline parallelism is the natural sequel to speculative decoding and prefill/decode disaggregation. All three belong in the same inference optimization series because all three answer the same deeper question: where should the work happen, on which hardware, under which traffic pattern, and at what cost per token?

• Read the PagedAttention/vLLM paper
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
• Megatron-LM parallelism paper
• Introduction to NVIDIA DGX H100/H200 Systems
• Speculative decoding paper
• SGLang prefill/decode disaggregation guide
• SGLang multi-node deployment guide
• DigitalOcean GPU Droplet Configurations
• DigitalOcean multi-node GPU configuration guide
• DigitalOcean Droplet feature documentation
• Ray Serve LLM documentation