Scaling LLMs requires 3D parallelism. Learn how Data, Pipeline, and Tensor parallelism combine to optimize H100/B200 clusters. Full architectural breakdown.
Why one parallelism axis is not enough
Modern LLM training no longer fits on a single GPU. Model weights, activations, optimizer state, and communication all grow with parameter count and sequence length. Data parallelism alone replicates the full model on every device, which wastes memory once the model exceeds what one GPU can hold. Tensor and pipeline strategies partition the model itself, but each introduces its own communication pattern and scheduling cost. 3D parallelism is the practice of combining data, pipeline, and tensor parallelism so clusters of H100- and B200-class GPUs can train models that are both large and throughput-efficient.
The goal is not to maximize any single dimension. It is to keep GPUs busy, keep memory under the per-device limit, and keep interconnect traffic from becoming the bottleneck. That balance depends on model shape, batch size, sequence length, and the topology of the cluster—not on a fixed recipe that works everywhere.
Data, pipeline, and tensor parallelism in practice
Data parallelism (DP) shards the global batch across replicas. Each replica runs the same model on different samples, then all-reduces gradients. DP scales throughput when the model fits in memory and the interconnect can absorb the all-reduce. Pipeline parallelism (PP) splits layers into stages. Microbatches stream through the pipeline so stages stay occupied; the tradeoff is pipeline bubbles and careful balancing of stage compute so no stage becomes a straggler. Tensor parallelism (TP) shards individual layers—often matrix multiplies and attention projections—across devices in a tight group. TP reduces per-device memory for large layers but needs high-bandwidth links and frequent collective operations within the tensor-parallel group.
In a 3D layout you assign each axis a degree: how many ways you shard data, how many pipeline stages, and how many devices share each tensor-sharded layer. The product of those degrees should match the device count you intend to use. Order of composition matters for communication: tensor groups usually sit on the fastest interconnect, pipeline stages exchange activations less frequently, and data-parallel groups run larger, less frequent gradient reductions.
How the three axes interact on GPU clusters
Memory pressure usually sets the minimum TP and PP needed so weights, activations, and optimizer state fit. After the model fits, you grow DP to use remaining devices and increase effective batch size—subject to learning-rate and stability choices, not just hardware. If TP is too high, collective latency inside each layer can dominate compute. If PP is too deep without enough microbatches, bubble time rises and utilization drops. If DP is large while per-step communication is slow, gradient sync can stall the critical path.
- Prefer higher TP only where layer size forces it and NVLink-class bandwidth is available within the group.
- Use PP to cut per-device layer memory when TP alone would oversubscribe the fast fabric.
- Scale DP last for remaining capacity once the model is sharded and the pipeline is filled.
- Match group placement to topology: keep TP on the tightest links; allow PP and DP to span broader fabric when needed.
Practical guidance for training runs
Start from a memory-feasible (TP, PP) pair for your model and sequence length, then raise DP until batch size or cluster size caps you. Profile step time by component: compute, TP collectives, PP send/recv, and DP all-reduce. Tune microbatch count and gradient accumulation so the pipeline stays full without overflowing activation memory. Prefer configurations that keep utilization high and communication predictable over ones that look larger on paper but leave devices waiting.
3D parallelism is a scheduling and partitioning problem as much as a systems one. Treat Data, Pipeline, and Tensor axes as knobs you co-adjust for your model and H100/B200 cluster, remeasure after each change, and lock a layout only when memory headroom, step time, and stability all clear your bar for production training.