Home / Blog / Distributed LLM Training Across Metros: Streaming DiLoCo in…
Tech News

Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

Google DeepMind and Google Research implemented Streaming DiLoCo in MaxText, cutting cross-slice LLM training traffic by over 60x across TPU clusters.

By Dillip Chowdary • Nov 19, 2026 • Source: Google Developers Blog

Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText
Image: Google Developers Blog

At a glance

Product
Streaming DiLoCo
Framework
MaxText
Author organization
Google DeepMind and Google Research
Target hardware
Google Cloud TPU v5p
Evaluation model
Qwen3-32B
Bandwidth reduction
64x to 65x

Google DeepMind and Google Research introduced an implementation of Streaming DiLoCo inside MaxText, Google's open-source JAX LLM training framework, enabling distributed model training across geographically separated datacenters without communication stalls. The technique partitions model parameters into staggered fragments that synchronize continuously over wide-area networks, as detailed in Google Developers Blog's report. By avoiding full-model transfers on constrained links, the system preserves training throughput on multi-slice accelerator clusters.

This report examines how asynchronous parameter streaming replaces monolithic data-parallel communication, the mathematics behind inter-slice bandwidth requirements, and benchmark results recorded across Google Cloud TPU v5p topologies. Machine learning engineers and infrastructure architects managing cross-region training clusters can use these findings to size network links and scale workloads across multiple facilities without hitting throughput bottlenecks.

01Inner and outer optimization cycles

Standard data parallelism requires accelerators to run an all-reduce collective on gradients across all devices at every single step, which causes severe network stalls over wide-area links. DiLoCo solves this by splitting training across independent compute groups called learners, which typically consist of individual TPU slices or intra-datacenter multi-slice setups. Each learner runs a local optimizer on its own data shard for a set cadence of local steps without communicating across groups.

During these local iterations, intra-replica interconnects continue operating normally within each compute group. Once the local interval finishes, learners calculate parameter deltas relative to the global model weights established at the beginning of the cycle. A distributed all-reduce then averages these deltas across the network, and an outer optimizer updates the shared global weights before the next local phase begins.

The algorithm reduces global communication frequency by orders of magnitude while matching the convergence and quality of standard synchronous training. However, basic DiLoCo still transmits weight deltas for the complete model simultaneously at the end of each local interval. On constrained network pipes, that bulk payload creates a recurring transmission pause across all participating sites.

02Staggered parameter fragmentation and load balancing

Streaming DiLoCo removes periodic transmission pauses by splitting the model into disjoint parameter fragments and transferring them continuously across iterations. In MaxText, the model is partitioned into a set number of fragments equal to the number of scanned Transformer decoder layers plus one additional fragment. That initial fragment contains non-scanned parameters including token embeddings, pre-logits layer normalization, and the output unembedding head.

Instead of halting computation to exchange the entire model at once, MaxText schedules a single fragment to transfer on staggered training steps. When the synchronization cadence matches the fragment count, one fragment moves on each step, converting a massive periodic bandwidth burst into a uniform data stream. MaxText also interleaves fragment selection across network depth rather than transferring consecutive layers, keeping parameter updates balanced throughout the architecture.

Because token embedding and output head matrices scale with vocabulary size, the non-scanned fragment can be substantially larger than individual transformer layers. To prevent that fragment from creating an outsized traffic burst, MaxText splits large parameter matrices from the non-scanned group and merges them evenly across the decoder layer fragments. This structural load balancing ensures uniform payload sizes across every execution step.

03Cross-slice bandwidth requirements and roofline constraints

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Network link requirements depend directly on parameter size, precision, and backward-pass execution time. Standard data-parallel training must overlap gradient reduce-scatter operations with the backward pass to prevent hardware stalls, demanding heavy sustained bandwidth across cluster slices. The following table contrasts network requirements for a 32-billion parameter model in bfloat16 precision running across distributed slices with a 6.2-second step time:

StrategyPer-step transfer volumeRequired bandwidthBehavior on 10 Gbps link
Data Parallel65.5 GB~85 GbpsStalls over 50 seconds per step
Vanilla DiLoCo0 GB (65.5 GB on step 65)~85 Gbps periodic peakRuns 64 steps, then stalls
Streaming DiLoCo~1.0 GB~1.3 GbpsSmooth execution, no stalls

Under standard data parallelism, exchanging 65.5 gigabytes of gradient data at every step requires roughly 85 gigabits per second of dedicated throughput to finish within step compute limits. Restricting that workload to an inter-slice 10 gigabit-per-second link triggers stalls exceeding 50 seconds per iteration. Vanilla DiLoCo reduces aggregate network volume by 65 times over the full interval but still halts compute on the synchronization step.

Streaming DiLoCo divides the 65.5-gigabyte transfer into 65 single-gigabyte fragments exchanged once per step. This smooth transmission profile lowers minimum bandwidth to approximately 1.3 gigabits per second, fitting comfortably inside throttled wide-area connections. Accelerators maintain uninterrupted execution without waiting for massive gradient batches to clear network buffers.

04Hardware benchmarks across throttled network links

To test resilience under simulated wide-area bottlenecks, engineers evaluated a 64-layer Qwen3-32B model partitioned into 65 fragments across two TPU v5p-32 slices. The team applied Linux traffic-control utilities to constrain inter-slice network pipes across three distinct regimes: line rate, a moderate congestion setting of 30 gigabits per second, and severe congestion capped at 10 gigabits per second.

Over a 65-step test sequence, Streaming DiLoCo reduced total cross-slice network traffic from 4,258.8 gigabytes under standard data parallelism down to 67.0 gigabytes. Transferring roughly 1.0 gigabyte per step instead of 65.5 gigabytes delivered a 64-fold to 65-fold reduction in total inter-slice data transmission. The continuous fragment streaming maintained consistent communication rates without triggering packet bursts.

Note: Hardware evaluation was conducted on Google Cloud TPU v5p accelerator slices using Linux traffic control to shape inter-slice bandwidth limits across unthrottled, 30 Gbps, and 10 Gbps test configurations.

05Multi-datacenter scaling and WAN resilience

Distributing monolithic training clusters across multiple datacenters removes physical campus power limits and facility space constraints. However, long-haul metropolitan connections introduce higher latency and limited bandwidth compared to local datacenter fabrics. Traditional data-parallel frameworks experience severe model FLOPS utilization collapse when global gradient synchronization blocks local matrix operations.

Streaming DiLoCo eliminates communication stalls by decoupling local inner-loop steps from global weight averaging while keeping cross-slice transfers small and constant. By spreading synchronization across every step, high-latency wide-area networks can accommodate large-scale parameter reconciliation without stalling accelerator cores. This allows organizations to pool compute clusters situated in separate metropolitan facilities into unified training jobs.

The framework provides a production path for scaling foundation models beyond single-building infrastructure envelopes. By implementing low-overhead parameter streaming within MaxText, engineering teams can maximize accelerator utilization across geographically separated facilities while preserving standard model convergence and quality metrics.

Frequently asked questions

What is Streaming DiLoCo?

Streaming DiLoCo is a distributed optimization implementation in MaxText that partitions model weights into fragments and synchronizes them across steps to eliminate network stalls over wide-area connections.

How does Streaming DiLoCo differ from vanilla DiLoCo?

Vanilla DiLoCo exchanges deltas for the entire model all at once every H steps causing periodic stalls, whereas Streaming DiLoCo transfers one fragment per step to maintain a uniform, low-bandwidth data stream.

How much cross-slice network bandwidth does Streaming DiLoCo save?

In benchmarks with Qwen3-32B across two TPU v5p-32 slices over 65 steps, it reduced cross-slice traffic from 4,258.8 gigabytes to 67.0 gigabytes, cutting communication volume by over 60 times.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →