Taming Outlier Tokens in Diffusion Transformers

Apple Machine Learning Research published work titled Taming Outlier Tokens in Diffusion Transformers. The study focuses on outlier tokens in Diffusion Transformers (DiTs) used for image generation, a setting where the role of such tokens has remained underexplored compared with discriminative vision models.

Prior work on Vision Transformers (ViTs) established that a small number of high-norm tokens can attract disproportionate attention while carrying limited local information. The Apple research extends that observation into generative pipelines and shows the same pattern is not confined to classification-style ViTs.

Technically, the phenomenon appears in both the encoder and the denoiser of modern Representation Autoencoder (RAE)–DiT pipelines. Pretrained ViT encoders can produce outlier representations that then feed into the DiT denoiser, so high-norm, attention-heavy tokens can shape the latent path the model uses to generate images rather than only how it classifies or encodes them.

For engineers building image generators on DiT or RAE–DiT stacks, that matters because attention budgets and latent quality depend on how tokens are distributed. If a few high-norm tokens dominate attention while adding little local detail, training dynamics, representation quality, and generation fidelity can be skewed without any change to the nominal architecture.

In competitive and market terms, DiT-style diffusion transformers sit at the center of current image-generation research and product pipelines. Characterizing and controlling outlier tokens in both encoder and denoiser stages is a concrete lever for teams comparing RAE–DiT designs against other latent diffusion setups that rely on pretrained ViT-style encoders.

The practical takeaway is to treat outlier-token behavior as a first-class diagnostic in RAE–DiT systems: inspect encoder and denoiser token norms and attention concentration, and watch for follow-on methods that tame those tokens without discarding useful global structure.