The Design Space of Tri-Modal Masked Diffusion Models

Apple Machine Learning Research introduced the first tri-modal Masked Diffusion Models, pretrained from scratch on text, image-text, and audio-text data, in a NeurIPS paper published in October 2026. Apple Machine Learning Research's report sets that pretraining against recent work that initialized and finetuned a base unimodal model for bi-modal generation.

This account covers multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects, optimized inference sampling defaults, and the stochastic differential equation reparameterization of batch size. The preliminary model, at 3B model scale on 6.4T tokens, is shown for text generation, T2I tasks, and T2S tasks. It is for researchers and engineers who train discrete diffusion models.

Design Space of Tri-Modal Masked Diffusion: what actually changed

Bi-modal predecessors initialized a base unimodal model and finetuned it for bi-modal generation. This October 2026 NeurIPS paper pretrains a Masked Diffusion Model from scratch on text, image-text, and audio-text, and the authors call it the first tri-modal MDM. They present that model as a unified design. The demonstration is a preliminary model at 3B model scale and 6.4T tokens, and the authors report strong results in text generation, T2I tasks, and T2S tasks. Apple Machine Learning Research places the paper in Methods and Algorithms.

The authors systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and they provide optimized inference sampling defaults. The batch-size analysis yields a novel stochastic differential equation (SDE) based reparameterization, which they say eliminates the need for tuning the optimal batch size as reported in recent work. They describe the work as "the largest scale systematic open study of multimodal discrete diffusion models conducted to date" and say it provides valuable insights into scaling behaviors across multiple modalities.

Design Space of Tri-Modal Masked Diffusion: how it works

The Masked Diffusion Model is pretrained from scratch on one mixture of text, image-text, and audio-text. The authors pretrain a preliminary model to showcase that unified design and state its size as 3B model scale and 6.4T tokens. Task coverage in the paper is text generation, T2I tasks, and T2S tasks. Discrete diffusion is the frame they set beside autoregressive language models. The study includes modality mixing ratios, noise schedules, and optimized inference sampling defaults among the settings for that model.

The batch-size analysis yields a novel stochastic differential equation (SDE) based reparameterization. The authors say it eliminates the need for tuning the optimal batch size reported in recent work. The reparameterization decouples physical batch size from logical batch size. Physical batch size is often chosen from compute: GPU saturation, FLOP-efficiency, and wall-clock time. Logical batch size is chosen to balance the variance of gradients during stochastic optimization. Recent work is the reference they give for the tuning step this split removes.

Design Space of Tri-Modal Masked Diffusion: why it matters now

Discrete diffusion is offered as a strong alternative to autoregressive language models, tested in the paper on text, image-text, and audio-text. Earlier work reached bi-modal generation by initializing and finetuning a unimodal base. This study pretrains the tri-modal MDM from scratch and publishes scaling behavior for that choice. The authors report strong results at 3B model scale on 6.4T tokens in text generation, T2I tasks, and T2S tasks. They say the study supplies valuable insights into scaling behaviors across multiple modalities.

Recent work, on their account, still required tuning an optimal batch size. The SDE reparameterization is how they remove that tuning, and they call the reparameterization novel. Physical batch size follows GPU saturation, FLOP-efficiency, or wall-clock time. Logical batch size follows the variance of gradients during stochastic optimization. The design space also records modality mixing ratios, noise schedules, and optimized inference sampling defaults. Those settings are reported with the preliminary 3B model trained on 6.4T tokens of text, image-text, and audio-text.

Design Space of Tri-Modal Masked Diffusion: who is affected

The author list is Louis Béthune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, Theo X. Olausson, Devon Hjelm, Paul Dixon, João Monteiro, Pierre Ablin, Vishnu Banna, Arno Blaas, Nick Henderson, Kari Noriy, Dan Busbridge, Josh Susskind, Marco Cuturi, Irina Belousova, Luca Zappella, Russ Webb, and Jason Ramapuram. Bruno Kacper Mlodozeniec is noted at the University of Cambridge, Theo X. Olausson at MIT, and Jason Ramapuram at Google DeepMind. Footnotes say that work was done while at Apple.

Labs that initialize a unimodal model and finetune it for bi-modal generation are the prior approach named in the study. People who still tune an optimal batch size are the audience for the SDE split between a compute-based physical batch size and a logical batch size set by gradient variance. Teams pretraining on text, image-text, and audio-text use the mixing ratios, noise schedules, and multimodal scaling laws. Reported tasks are text generation, T2I, and T2S at 3B model scale and 6.4T tokens.

Design Space of Tri-Modal Masked Diffusion: what to watch

The authors call the showcased model preliminary and report strong results at 3B model scale (6.4T tokens) in text generation, T2I tasks, and T2S tasks. It illustrates a unified design pretrained from scratch on text, image-text, and audio-text. Published beside it are multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects, and optimized inference sampling defaults. The SDE reparameterization separates physical batch size from logical batch size and, the authors say, removes the optimal-batch tuning step reported in recent work.

Related papers appear with it: Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion (July 11, 2025; ICML; Methods and Algorithms; Speech and Natural Language Processing) and How to Scale Your EMA (December 4, 2023; NeurIPS; Methods and Algorithms; equal contributors). TCSM is a novel and versatile objective for training and fine-tuning discrete diffusion models, a general framework with broad applicability that supports pre-training directly from data samples. Preserving training dynamics across batch sizes enables a trade-off between batch size and wall-clock time. Stochastic gradient descent scales the learning rate linearly with batch size, and a model EMA is a functional copy of a target model.