Tensor & Pipeline Parallelism

A 70B parameter model in FP16 needs about 140 GB just for weights. One 80 GB GPU cannot hold it, and that ignores KV cache, activations, and runtime overhead. Model parallelism is the family of techniques that split the model across devices.

Tensor parallelism#

Tensor parallelism slices the matrix multiplications inside each layer. Several GPUs hold different shards of the same weight matrix, compute partial results, then communicate through all-reduce or all-gather operations. It works best inside a node with fast links like NVLink because every layer communicates.

Pipeline parallelism#

Pipeline parallelism puts different layers on different GPUs. GPU 0 runs early layers, GPU 1 runs middle layers, GPU 2 runs late layers, and so on. The problem is the bubble: while the first microbatch is in stage 2, stage 1 may be idle unless more microbatches are flowing behind it. Microbatching fills the pipe but adds scheduling complexity.

Expert parallelism#

Mixture-of-Experts adds another axis. Experts can be placed on different GPUs, and a router sends tokens to the selected experts. In large training systems, tensor, pipeline, data, and expert parallelism are composed. In inference, the goal is usually simpler: fit the model and keep latency acceptable.

Practical sizing#

A 70B FP16 model usually needs at least two 80 GB GPUs for weights, but practical serving often uses 4 or 8 GPUs to leave memory for cache and improve latency. Tensor parallelism across 4 GPUs can reduce per-GPU weight memory and speed each layer, but it pays communication every layer. Pipeline parallelism can cross slower links but is sensitive to batch shape.

Practical takeaways#

Choose the layout around the bottleneck. For low-latency inference inside one server, tensor parallelism is common. For very large models across nodes, pipeline and expert layouts enter. Always include communication cost in the plan; splitting a model is easy on a diagram and expensive on the wire.