Taming Outlier Tokens in Diffusion Transformers

Apple Machine Learning Research studied outlier tokens in Diffusion Transformers, or DiTs, used for image generation. Earlier work on Vision Transformers, or ViTs, had already shown that a small number of high-norm tokens can draw disproportionate attention while carrying limited local information. That pattern was better known in recognition-style ViTs than in generative stacks. The new work targets modern Representation Autoencoder DiT pipelines, often shortened to RAE-DiT, and reports that the same outlier behavior shows up there rather than remaining a pure encoder-side curiosity.

On the technical side, the phenomenon is not confined to one stage of the pipeline. Pretrained ViT encoders in these systems can produce outlier representations, and the same high-norm token pattern appears in the denoiser as well. That is the architectural claim: outlier tokens are present both where latents are formed and where the DiT does denoising. In practical terms, a few tokens with unusually large norms can dominate attention while contributing little localized content, which is the same structural imbalance prior ViT work described, now observed inside a generative DiT stack.

For engineers building or shipping image generators, that matters because attention cost and signal quality are both sensitive to a small set of tokens. If a few high-norm tokens attract most of the attention mass, compute and capacity can be spent on tokens that carry weak local detail. Anyone training, fine-tuning, or debugging RAE-DiT style models has a concrete failure mode to inspect: encoder-side outliers that enter the latent path, and denoiser-side outliers that shape how attention is allocated during generation.

Competitive and market context is still largely about representation choice, not branding. DiT-based image models increasingly sit on top of learned latent or representation autoencoders rather than raw pixels, and many of those encoders are pretrained ViTs. Outlier tokens were already a known pathology in ViT recognition models; extending that diagnosis to both encoder and denoiser in RAE-DiT pipelines puts pressure on teams whose stacks look similar, whether they are optimizing quality, latency, or training stability. The paper frames the gap as underexplored in generative models even after the ViT literature had flagged the issue.

What to watch next is whether training and inference recipes start treating encoder outliers and denoiser outliers as separate control points. If the encoder is the source of high-norm, low-information tokens, representation design and encoder finetuning become first-order levers. If the denoiser recreates or amplifies the pattern, attention or normalization changes inside the DiT itself become the more direct fix. The immediate practical takeaway is to measure token-norm distributions and attention concentration in both stages of an RAE-DiT pipeline, not only in the backbone ViT used for encoding.