Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)

Google engineers published a systems engineering playbook for serving the **397B-parameter Qwen 3.5 Mixture-of-Experts (MoE)** model on **Ironwood (TPU7x)** TPUs. The work centers on a modular **JAX/Pallas** optimization stack built to make that large MoE practical under Ironwood’s sharding limits, with measured gains of up to a **4.7x** inference speedup on prefill-heavy workloads.

The stack’s core architectural move is a hybrid **Data Parallelism and Expert Parallelism (DP+EP)** topology. That layout was chosen specifically to work around severe hardware sharding constraints that would otherwise block efficient placement of a 397B MoE across devices. At the communication layer, the team added custom low-level fusions, including a **hierarchical reduce-scatter**, aimed at speeding cross-device token routing between experts rather than relying on stock collective patterns alone.

For engineers and builders, the write-up is less a model drop and more a concrete serving recipe: MoE at this scale is often limited by how you shard experts and move tokens, not by raw FLOPs. A modular JAX/Pallas path, DP+EP placement, and fused collectives show how software topology and kernel-level communication work together to unlock prefill throughput on TPU7x.

In market terms, the playbook sits at the intersection of open large MoEs (Qwen 3.5-class) and Google’s Ironwood TPU serving stack. It signals that competitive inference on hyperscaler silicon now depends on hybrid parallelism and custom communication as much as on model architecture, especially for prefill-bound product paths such as long-context batch and interactive first-token work.

Practical takeaway: if you are targeting large MoE inference on Ironwood or similar tightly sharded accelerators, evaluate hybrid **DP+EP** before forcing pure expert or pure data parallel layouts, and treat hierarchical reduce-scatter–style fusions as first-class levers for token routing cost. Watch for follow-on detail from the Google Developers Blog on the incomplete “finally, by executing…” path in the stack—whatever closes that loop will complete the full optimization sequence beyond the 4.7x prefill result already reported.