TechnologyInference & Serving
Tensor & Pipeline Parallelism
At a glance
How a model bigger than one GPU runs: split each layer across GPUs, split layers into stages, or both.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Inference & Serving
- Technology
In short
Tensor parallelism splits each layer's weight matrices across GPUs that exchange partial results at every layer, while pipeline parallelism places groups of layers on different GPUs. Bigcircle recommends choosing the layout from the interconnect before the GPU count, because a 70B model's 140 GB of weights splits easily on a diagram and the all-reduce traffic then decides latency. The inference guide says model parallelism splits weights across GPUs and introduces communication between those GPUs.
Key facts
Definition
Tensor parallelism shards the matrix multiplications inside every layer across several GPUs, and pipeline parallelism assigns consecutive groups of layers to different GPUs.
How Bigcircle does it
- Bigcircle recommends tensor parallelism inside one server with NVLink and pipeline stages across servers, because tensor parallelism communicates at every layer while pipeline stages pass activations only between stages.
- Bigcircle recommends checking that the attention head count divides by the tensor parallel size, because vLLM refuses to start otherwise, so a 64-head model runs at 2, 4 or 8 GPUs.
- Bigcircle recommends trying quantization before adding GPUs, because a model that fits on fewer devices pays less communication on every layer.
Comparison
Tensor parallelism splits each layer and needs fast links because it communicates at every layer, while pipeline parallelism splits the stack, tolerates slower links and leaves stages idle unless microbatches keep it full.
Key number
The vLLM documentation sets tensor parallel size to the GPUs per server and pipeline parallel size to the server count, for example 8 and 2 for two servers of eight GPUs.
Tensor parallelism is a way of running one model on several GPUs by splitting the weight matrices inside each layer between them. Each GPU holds a slice of every matrix, computes its part of the result, and exchanges partial results with the others before the next layer starts. Pipeline parallelism splits the model the other way, putting whole layers on different GPUs. Both exist because large models do not fit on one device.
A 70B model stored at 16 bits needs about 140 GB for its weights alone. One 80 GB GPU cannot hold it, and that is before the KV cache, activations and runtime overhead. Model parallelism is the family of techniques for splitting it.
Tensor parallelism#
Megatron-LM showed in 2019 how to split a transformer layer's matrix multiplications so that each GPU works on a shard and the shards combine with an all-reduce, a collective operation in which every GPU ends up with the summed result. Megatron's layout needs two of those per layer in the forward pass, one after attention and one after the feed-forward block. That traffic happens at every layer of every token, so tensor parallelism only pays on fast links between GPUs, such as NVLink inside one server.
The split also has to divide the model evenly. vLLM refuses to start when the number of attention heads is not divisible by the tensor parallel size, so a model with 64 heads runs at 2, 4 or 8 GPUs and not at 6.
Pipeline parallelism#
Pipeline parallelism puts groups of layers on different GPUs: the first GPU runs the early layers, the next runs the middle, and so on. Only the activations between stages cross the link, once per stage rather than at every layer, so pipeline parallelism tolerates slower connections. Its cost is the bubble. While the first batch is in stage 2, stage 1 sits idle unless more work is queued behind it. GPipe's answer was to cut each batch into microbatches that follow each other through the stages, which fills most of the pipe at the price of more scheduling.
Expert parallelism#
Mixture-of-experts models add a third axis. Different experts can live on different GPUs, and the router sends each token to wherever its chosen experts are. Large training systems combine tensor, pipeline, data and expert parallelism. Serving is simpler: the goal is to fit the model and keep latency acceptable, and for MoE models vLLM documents combining data-parallel attention with expert or tensor parallel expert layers.
Choosing the layout#
As of October 2026 the vLLM documentation gives the rule most deployments follow. If the model fits on one server, use tensor parallelism across its GPUs. If it needs several servers, set the tensor parallel size to the GPUs per server and the pipeline parallel size to the number of servers, for example 8 and 2 for two servers of eight GPUs. On GPUs without NVLink, the same documentation recommends pipeline parallelism instead of tensor parallelism.
Leave room beyond the weights. Two 80 GB GPUs hold a 70B model's 140 GB of weights with about 20 GB to spare, which is little cache for real concurrency, so practical serving of that size often uses 4 or 8 GPUs. Tensor parallelism across more GPUs also shortens each layer, which lowers latency, but every added GPU adds communication on every layer, and past the point where the links saturate it stops helping.
Count the communication before the GPUs#
Bigcircle recommends choosing the layout from the interconnect before the GPU count, because a split that looks balanced on a diagram can spend more time moving partial results than computing them. Measure latency at your real batch size and context length on the actual hardware, since the published rules assume NVLink inside the server and a fast network between them. If quantizing the weights lets the model fit on fewer GPUs, try that first: no split at all has no communication cost.