Home / Blog / Unlocking Stable On-Policy RL on TPU with Bitwise-Zero…
Tech News

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference

Google engineers eliminated floating-point train-inference logprob drift for on-policy RL across 64 Cloud TPU v5p chips using a four-part recipe in Tunix.

By Dillip Chowdary • Nov 16, 2026 • Source: Google Developers Blog

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference
Image: Google Developers Blog

At a glance

Product
Tunix Zero-TIM
Hardware
64 Cloud TPU v5p chips (DP8×TP8)
Release date
November 16, 2026
Test model
Qwen3-8B
Test benchmark
200-step FrozenLake
Precision target
Bitwise-zero logprob mismatch

Google software engineers Yuxuan Zhang, Lin Chai, Lance Wang, Tianshu Bao, and product manager Weiren Yu published a zero train-inference mismatch system in Tunix. As detailed in Google Developers Blog's report, the open-source implementation eliminates logprob divergence across Cloud TPU v5p clusters between rollout sampling and gradient updates.

This report covers the mechanics of floating-point drift, the four-part JAX and Pallas recipe, and multi-host validation results. Distributed systems engineers training large models will find concrete methods to diagnose divergence, order tensor reductions, and achieve true on-policy training without combining separate serving and training engines.

01Floating-point reduction order in bfloat16

Reinforcement learning for large language models assumes that gradient updates evaluate actions generated by that exact policy. In production, incremental decoding and batched training backward passes execute through separate runtime graphs. Even under identical weights, differing paths assign distinct log probabilities to identical tokens, introducing a hidden off-policy gap across multi-turn trajectories.

This discrepancy arises because floating-point addition is non-associative, causing reduction trees to round differently depending on accumulation order. In bfloat16, adding increments of 0.0078125 near 2.0 under round-to-nearest-even rules discards inputs when summed individually. In contrast, grouping those identical inputs together first accumulates enough magnitude to reach the next representable value.

Similar discrepancies occur when matrix products accumulate partial sums, normalization layers compute statistics, or tensor shards combine outputs. The original TPU stack ran sampler decoding and trainer passes with different token-array shapes. Because JAX specializes computations to input shapes and differentiation graphs, subtle arithmetic variations produced conflicting logprob bytes from identical equations.

02Three execution paths and the alignment contract

The alignment contract requires sampler decode, serving prefill, and trainer forward passes to yield bitwise-identical logprobs for every sampled action token under matching weights. Rather than allowing floating-point tolerances, this specification demands raw byte parity across compilation and differentiation boundaries. Meeting this contract guarantees numerical on-policy stability without merging separate inference and training codebases.

The architecture introduces serving prefill as an intermediate diagnostic checkpoint between generation and training. A disagreement between sampler decode and serving prefill proves the serving engine disagrees with itself during full-sequence re-scoring. Divergence between serving prefill and trainer forward reveals numerical mismatches introduced across JAX compilation or reverse-mode automatic differentiation boundaries.

Execution pathRuntime roleOutput contract
Sampler decodeIncremental action generationBehavior logprob per action
Serving prefillFull-sequence re-scoringIntermediate verification baseline
Trainer forwardBatched forward-backward passBitwise-identical policy gradient

Every path evaluates the exact token identifiers recorded during rollout generation rather than sampling alternate choices. Preserving sampled token IDs across turns ensures that all execution branches inspect an identical token history.

03Four-part Pallas recipe for TPU shards

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Google structured the zero-mismatch implementation in Tunix around a four-part recipe that bridges separate serving and training pipelines. Within each tensor-parallel rank, custom Pallas kernels lock accumulation order by traversing fixed contraction blocks regardless of token batch sizes. Partial products accumulate in float32 scratch registers before explicit output casting across projection layers.

The systematic recipe addresses cross-shard communication, scoring logic, and automatic differentiation through four distinct operational stages:

  1. Within-shard arithmetic Batch-invariant Pallas kernels process fixed contraction blocks while accumulating partial sums in float32 registers.
  2. Cross-shard reductions Programmed ppermute primitives communicate partial outputs so each rank executes additions in deterministic source-rank order.
  3. Action-logprob scoring A standardized Pallas log-softmax construction evaluates recorded action tokens while preserving historical token identifiers.
  4. Custom differentiation rules Custom vector-Jacobian products retain serving forward kernels and provide dedicated backward pullbacks for gradient passes.

Dedicated Pallas kernels also standardize arithmetic reductions for RMSNorm layers and intermediate precision casts inside SwiGLU activations. The implementation retains the existing Pallas paged-attention forward pass while unifying trailing logprob scoring calculations. By separating communication from addition, tensor-parallel ranks execute collective reductions in explicit code order rather than relying on collective library defaults.

04Custom differentiation rules and compiler flags

The training engine reuses serving prefill computations by wrapping Pallas kernels with custom vector-Jacobian products defined via the jax.custom_vjp interface. The forward evaluation function executes the serving Pallas kernel to return numerical outputs and cache required tensors. The backward function evaluates an analytical differentiable replica that mirrors accumulation blocking exclusively to compute the pullback.

Note: Training execution requires setting the compiler configuration flag --xla_allow_excess_precision=false through XLA_FLAGS to suppress variable intermediate precision rounding in compiler operations surrounding Pallas kernels.

The differentiated JAX program processes extended trajectories in fixed-size chunks while carrying paged key-value cache states between iterations. Reverse passes propagate parameter gradients across these cache connections before optimizer updates occur.

05Multi-host verification across sixty-four chips

Google verified the zero-mismatch implementation across a multi-host cluster containing 64 Cloud TPU v5p chips arranged in a DP8xTP8 topology. This scale demonstrates that the four-part Pallas recipe eliminates floating-point drift across accelerator boundaries and communication fabrics. The reusable kernel and collective patterns provide distributed JAX developers with a blueprint for stable reinforcement learning.

The team conducted a 200-step numerical verification benchmark on the synthetic FrozenLake environment using a public Qwen3-8B model checkpoint as a baseline. The benchmark confirmed bitwise-identical logprob generation across sampler decode, serving prefill, and trainer forward passes. Eliminating train-inference mismatch prevents subtle off-policy degradation, enabling stable on-policy convergence during demanding training schedules.

This Tunix framework brings on-policy alignment capabilities to the TPU ecosystem alongside concurrent open-source efforts developed for GPU hardware. Parallel projects, including vLLM integrated with TorchTitan and RadixArk LMSys Miles, have similarly adopted batch-invariant kernels to resolve train-inference discrepancies. Google's release provides modular JAX and Pallas implementations that let distributed training teams maintain mathematical fidelity across clusters.

Frequently asked questions

What causes train-inference mismatch in reinforcement learning on TPUs?

Floating-point addition is non-associative, so differing reduction trees and intermediate rounding in bfloat16 create divergent logprob outputs between incremental sampling and batched training passes.

How does the Tunix framework achieve zero train-inference mismatch?

Tunix uses batch-invariant Pallas kernels with float32 accumulation, ordered ppermute cross-shard reductions, shared log-softmax paths, and custom vector-Jacobian products to maintain bitwise-identical calculations.

How was the zero train-inference mismatch implementation tested?

The engineering team verified the implementation across 64 Cloud TPU v5p chips in a DP8×TP8 topology using a 200-step test on the FrozenLake environment with a Qwen3-8B model.

What compiler configuration is required for numerical alignment?

The training process requires the XLA_FLAGS option --xla_allow_excess_precision=false to constrain excess precision in compiler operations surrounding the Pallas kernels.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →