Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

Apple Machine Learning Research describes Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers as the architecture behind Siri Expressive Voices. The system synthesizes rich, configurable speech in real time and entirely on device, driven by AFM 3 Core Advanced, Apple’s most powerful on-device foundation model.

At the center of the work is a memory-efficient audio synthesis path built for the Apple Matrix Coprocessor (AMX). A detokenizer converts the semantic audio tokens emitted by the foundation model into high-fidelity audio while staying inside AMX’s tight compute and memory budget. That detokenizer design is the practical link between AFM 3 Core Advanced’s token outputs and the audible speech users hear.

For engineers shipping speech on phone-class hardware, the hard constraint is not only model quality but real-time decode under fixed memory. Converting semantic tokens on-device, rather than relying on cloud-side waveform generation, keeps latency and privacy inside the device boundary and forces the synthesis stack to fit AMX resources instead of server GPUs.

In competitive terms, this positions Siri Expressive Voices as an on-device stack: foundation-model token generation plus a specialized detokenizer tuned for Apple’s matrix coprocessor. That is a different product shape from cloud TTS pipelines that stream audio from remote inference, and it ties voice quality gains to silicon-aware architecture rather than to larger off-device models alone.

What to watch next is how the detokenizer and Decoupled Temporal Depth Diffusion Transformers behave under production voice configurations—richness, configurability, and fidelity—while still meeting AMX’s real-time budget as AFM 3 Core Advanced remains the on-device foundation model feeding semantic audio tokens into synthesis.