On-Device Speech Recognition and Text-to-Speech in Google AI Edge Gallery

Google launched on-device Automatic Speech Recognition and Text-to-Speech capabilities within its Google AI Edge Gallery application on November 10, 2026. According to Google Developers Blog's report, the update enables accurate, offline voice transcription and speech synthesis with low latency and a compact memory footprint. The release couples local audio processing with small language models such as Gemini Nano and Gemma 4 to transcribe speech and generate natural spoken responses without remote servers.

This report details the architectural components, benchmark metrics, and cross-platform libraries powering the local voice pipeline across Android, desktop, and web environments. Mobile software engineers, embedded artificial intelligence developers, and wearable hardware designers can examine how these tools eliminate application bundle bloat while handling multimodal interactions entirely offline.

01Offline visual grounding in XR Language Teacher

The XR Language Teacher showcase in Google AI Edge Gallery demonstrates complete on-device multimodal execution for travelers lacking cellular roaming or wireless connectivity. Users point a smartphone camera at an unfamiliar physical object or printed restaurant menu and speak questions directly into the device microphone. Live visual input and streaming audio travel together in a single processing pass directly to Gemini Nano on Android or Gemma 4 on cross-platform systems.

The underlying small language model relies on a structured output schema to separate primary explanations from foreign vocabulary words. This separation enables bilingual code-switching within the local speech synthesis engine, shifting voices and accents mid-sentence from a primary tongue to a local dialect alongside an on-screen vocabulary flashcard. The resulting audio pipeline provides instant feedback without relying on cloud translation infrastructure.

Google adapted the local tutoring setup to run across hands-free smart glasses paired with a host smartphone. Because inference runs entirely on the connected handset, wearable glasses receive ambient visual context and streaming voice queries without bulky processing hardware. The architecture supports natural audio-first interactions during transit through underground stations or noisy markets where network access remains completely unavailable.

02Sentence synchronization and floating chat playback

Google integrated local speech recognition and synthesis directly into the primary AI Chat interface within Google AI Edge Gallery. The updated interface provides on-device models with a hands-free conversational voice channel suited for mobile usage. A dedicated Model Management configuration pane allows users to select specific speech recognition pipelines, pick preset synthesis voices, and install offline language packs.

The chat system introduces a floating playback bar featuring synchronized live sentence highlighting during read-aloud sessions. Users can enable automatic speech playback or activate individual assistant responses using an on-screen speaker button. The synchronized visual display tracks spoken progress across individual sentences, allowing readers to tap any specific phrase on screen to jump playback immediately to that point.

The reference implementation provides software patterns designed to maintain responsiveness while small language models generate output tokens. The audio pipeline strips Markdown syntax, suppresses non-spoken code blocks or data tables, and schedules audio synthesis on a sentence-by-sentence basis. This granular scheduling prevents compute and memory contention between language generation and speech synthesis threads on constrained mobile chipsets.

03System speech acceleration inside Android ML Kit

Android developers access system-level speech models through ML Kit GenAI Speech Recognition and Text-to-Speech programming interfaces managed via AICore. Because the operating system bundles the underlying model weights, applications incur zero megabytes of package overhead. The platform framework provides automatic hardware acceleration across neural processing units and tensor processing units alongside seamless background updates.

The ML Kit speech recognition engine transcribes audio across more than 100 languages, achieving an 8.0 percent to 9.0 percent Word Error Rate on the FLEURS benchmark. The system transcribes speech at 10x real-time speed, requiring roughly 300 milliseconds of processing time for three seconds of audio. This rapid execution reduces end-to-end speech translation waiting periods by 35 percent to 44 percent.

Speech synthesis in ML Kit relies on a compact few-step diffusion model accelerated on neural processing units. The synthesis pipeline begins vocal playback within approximately 100 milliseconds and produces 24 kilohertz audio across 8 voices and 21 locales at 50x real-time speed. It synthesizes a 27-second paragraph in roughly 500 milliseconds, cutting active power consumption by over 50 percent compared to autoregressive architectures.

04Turnkey cross-platform pipelines with MediaPipe Tasks

MediaPipe Tasks supplies unified speech recognition and synthesis interfaces across iOS, macOS, Windows, Linux, and web applications. The cross-platform framework automates audio framing, configurable voice activity detection, and decoding routines that previously required custom low-level engineering. Developers can deploy standardized voice interactions across diverse operating systems without maintaining separate platform-specific audio streaming layers.

The MediaPipe Speech Recognition Task supports real-time streaming, full-sentence batching, and hybrid transcription modes. Hybrid operation streams provisional text while a user speaks, refining the full sentence once speech pauses. Google includes support for open-weight models such as Parakeet-TDT 0.6B out of the box, allowing developers to test local browser transcription directly without cloud infrastructure.

The MediaPipe Text-to-Speech Task delivers low-latency streaming synthesis and voice selection across multiple compact open-weight model architectures. Supported packages include multilingual models like PocketTTS 100M and Kokoro-82M for global applications. For ultra-lightweight English synthesis, the framework runs Phonon 31M, achieving a time-to-first-audio latency of under 100 milliseconds on standard desktop and mobile hardware.

05Modular multi-subgraph execution under LiteRT runtime

LiteRT serves as Google's foundational on-device inference runtime powering both ML Kit and MediaPipe Tasks. The engine accommodates custom model architectures and offers fine-grained execution control for developers building specialized cross-platform speech pipelines. By isolating execution steps into efficient modular routines, LiteRT runs audio models alongside small language models without compute resource starvation.

On-device speech architectures require coordinating diverse subgraphs and supplementary data files across multiple execution stages. Speech recognition demands encoder and decoder subgraphs paired with dedicated tokenizers, while speech synthesis coordinates phonemizer dictionaries, voice embeddings, and neural vocoders. LiteRT manages these interconnected dependencies inside unified model bundles to maintain smooth memory management on local hardware.

Google introduced optimized litertlm model speech bundles to streamline deployment across desktop, mobile, and web targets. These bundles package models such as Parakeet-TDT 0.6B, PocketTTS 100M, Kokoro-82M, and Phonon 31M into portable containers. The standardized format ensures compact storage, deterministic loading, and predictable inference speeds across client environments without remote API dependencies.