Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText
Google DeepMind and Google Research implemented Streaming DiLoCo in MaxText, cutting cross-slice LLM training traffic by over 60x across TPU clusters.
By Dillip Chowdary • Nov 19, 2026 • Source: Google Developers Blog

At a glance
- Product
- Streaming DiLoCo
- Framework
- MaxText
- Author organization
- Google DeepMind and Google Research
- Target hardware
- Google Cloud TPU v5p
- Evaluation model
- Qwen3-32B
- Bandwidth reduction
- 64x to 65x
In this post
Google DeepMind and Google Research introduced an implementation of Streaming DiLoCo inside MaxText, Google's open-source JAX LLM training framework, enabling distributed model training across geographically separated datacenters without communication stalls. The technique partitions model parameters into staggered fragments that synchronize continuously over wide-area networks, as detailed in Google Developers Blog's report. By avoiding full-model transfers on constrained links, the system preserves training throughput on multi-slice accelerator clusters.
This report examines how asynchronous parameter streaming replaces monolithic data-parallel communication, the mathematics behind inter-slice bandwidth requirements, and benchmark results recorded across Google Cloud TPU v5p topologies. Machine learning engineers and infrastructure architects managing cross-region training clusters can use these findings to size network links and scale workloads across multiple facilities without hitting throughput bottlenecks.
01Inner and outer optimization cycles
Standard data parallelism requires accelerators to run an all-reduce collective on gradients across all devices at every single step, which causes severe network stalls over wide-area links. DiLoCo solves this by splitting training across independent compute groups called learners, which typically consist of individual TPU slices or intra-datacenter multi-slice setups. Each learner runs a local optimizer on its own data shard for a set cadence of local steps without communicating across groups.
During these local iterations, intra-replica interconnects continue operating normally within each compute group. Once the local interval finishes, learners calculate parameter deltas relative to the global model weights established at the beginning of the cycle. A distributed all-reduce then averages these deltas across the network, and an outer optimizer updates the shared global weights before the next local phase begins.
The algorithm reduces global communication frequency by orders of magnitude while matching the convergence and quality of standard synchronous training. However, basic DiLoCo still transmits weight deltas for the complete model simultaneously at the end of each local interval. On constrained network pipes, that bulk payload creates a recurring transmission pause across all participating sites.
02Staggered parameter fragmentation and load balancing
Streaming DiLoCo removes periodic transmission pauses by splitting the model into disjoint parameter fragments and transferring them continuously across iterations. In MaxText, the model is partitioned into a set number of fragments equal to the number of scanned Transformer decoder layers plus one additional fragment. That initial fragment contains non-scanned parameters including token embeddings, pre-logits layer normalization, and the output unembedding head.
Instead of halting computation to exchange the entire model at once, MaxText schedules a single fragment to transfer on staggered training steps. When the synchronization cadence matches the fragment count, one fragment moves on each step, converting a massive periodic bandwidth burst into a uniform data stream. MaxText also interleaves fragment selection across network depth rather than transferring consecutive layers, keeping parameter updates balanced throughout the architecture.
Because token embedding and output head matrices scale with vocabulary size, the non-scanned fragment can be substantially larger than individual transformer layers. To prevent that fragment from creating an outsized traffic burst, MaxText splits large parameter matrices from the non-scanned group and merges them evenly across the decoder layer fragments. This structural load balancing ensures uniform payload sizes across every execution step.
03Cross-slice bandwidth requirements and roofline constraints
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Network link requirements depend directly on parameter size, precision, and backward-pass execution time. Standard data-parallel training must overlap gradient reduce-scatter operations with the backward pass to prevent hardware stalls, demanding heavy sustained bandwidth across cluster slices. The following table contrasts network requirements for a 32-billion parameter model in bfloat16 precision running across distributed slices with a 6.2-second step time:
| Strategy | Per-step transfer volume | Required bandwidth | Behavior on 10 Gbps link |
|---|---|---|---|
| Data Parallel | 65.5 GB | ~85 Gbps | Stalls over 50 seconds per step |
| Vanilla DiLoCo | 0 GB (65.5 GB on step 65) | ~85 Gbps periodic peak | Runs 64 steps, then stalls |
| Streaming DiLoCo | ~1.0 GB | ~1.3 Gbps | Smooth execution, no stalls |
Under standard data parallelism, exchanging 65.5 gigabytes of gradient data at every step requires roughly 85 gigabits per second of dedicated throughput to finish within step compute limits. Restricting that workload to an inter-slice 10 gigabit-per-second link triggers stalls exceeding 50 seconds per iteration. Vanilla DiLoCo reduces aggregate network volume by 65 times over the full interval but still halts compute on the synchronization step.
Streaming DiLoCo divides the 65.5-gigabyte transfer into 65 single-gigabyte fragments exchanged once per step. This smooth transmission profile lowers minimum bandwidth to approximately 1.3 gigabits per second, fitting comfortably inside throttled wide-area connections. Accelerators maintain uninterrupted execution without waiting for massive gradient batches to clear network buffers.
04Hardware benchmarks across throttled network links
To test resilience under simulated wide-area bottlenecks, engineers evaluated a 64-layer Qwen3-32B model partitioned into 65 fragments across two TPU v5p-32 slices. The team applied Linux traffic-control utilities to constrain inter-slice network pipes across three distinct regimes: line rate, a moderate congestion setting of 30 gigabits per second, and severe congestion capped at 10 gigabits per second.
Over a 65-step test sequence, Streaming DiLoCo reduced total cross-slice network traffic from 4,258.8 gigabytes under standard data parallelism down to 67.0 gigabytes. Transferring roughly 1.0 gigabyte per step instead of 65.5 gigabytes delivered a 64-fold to 65-fold reduction in total inter-slice data transmission. The continuous fragment streaming maintained consistent communication rates without triggering packet bursts.
Note: Hardware evaluation was conducted on Google Cloud TPU v5p accelerator slices using Linux traffic control to shape inter-slice bandwidth limits across unthrottled, 30 Gbps, and 10 Gbps test configurations.
05Multi-datacenter scaling and WAN resilience
Distributing monolithic training clusters across multiple datacenters removes physical campus power limits and facility space constraints. However, long-haul metropolitan connections introduce higher latency and limited bandwidth compared to local datacenter fabrics. Traditional data-parallel frameworks experience severe model FLOPS utilization collapse when global gradient synchronization blocks local matrix operations.
Streaming DiLoCo eliminates communication stalls by decoupling local inner-loop steps from global weight averaging while keeping cross-slice transfers small and constant. By spreading synchronization across every step, high-latency wide-area networks can accommodate large-scale parameter reconciliation without stalling accelerator cores. This allows organizations to pool compute clusters situated in separate metropolitan facilities into unified training jobs.
The framework provides a production path for scaling foundation models beyond single-building infrastructure envelopes. By implementing low-overhead parameter streaming within MaxText, engineering teams can maximize accelerator utilization across geographically separated facilities while preserving standard model convergence and quality metrics.
Frequently asked questions
What is Streaming DiLoCo?
Streaming DiLoCo is a distributed optimization implementation in MaxText that partitions model weights into fragments and synchronizes them across steps to eliminate network stalls over wide-area connections.
How does Streaming DiLoCo differ from vanilla DiLoCo?
Vanilla DiLoCo exchanges deltas for the entire model all at once every H steps causing periodic stalls, whereas Streaming DiLoCo transfers one fragment per step to maintain a uniform, low-bandwidth data stream.
How much cross-slice network bandwidth does Streaming DiLoCo save?
In benchmarks with Qwen3-32B across two TPU v5p-32 slices over 65 steps, it reduced cross-slice traffic from 4,258.8 gigabytes to 67.0 gigabytes, cutting communication volume by over 60 times.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement