Cost-optimizing AssemblyAI speech-to-text inference on AWS with Amazon EC2

AssemblyAI has deployed an elastic inference architecture across Amazon Web Services to operate speech-to-text deep learning models on thousands of graphics processing units. Detailed in AWS Machine Learning Blog's report, the production deployment combines Amazon Elastic Container Service, Amazon EC2 Spot Instances, and Amazon Simple Queue Service to achieve seventy percent compute cost savings compared to standard On-Demand pricing. The design addresses the steep computational demands of continuous speech recognition, speaker detection, sentiment analysis, chapter detection, and personally identifiable information redaction while maintaining strict service level commitments.

This technical analysis examines how engineering teams running high-volume GPU inference can pair diversified compute pools with automated queue-based retries to survive spot interruptions. It is intended for infrastructure architects, machine learning engineers, and technical leaders evaluating cost reduction strategies for batch and near-real-time deep learning pipelines operating under strict data residency constraints.

01Multi-family GPU clusters and event routing

The inference infrastructure runs containerized models on Amazon ECS clusters hosted inside virtual private clouds across multiple Availability Zones. Each cluster deploys tasks running a single deep learning model per GPU across thousands of instances. Audio files are retrieved directly from Amazon S3 storage buckets, processed through acoustic and language model pipelines, and written to dedicated customer results stores.

Fleet orchestration relies on ECS capacity provider strategies that blend baseline On-Demand instances with Spot capacity to absorb demand spikes. Amazon EventBridge monitors the cluster environment and immediately triggers a dedicated AWS Lambda function when an EC2 Spot interruption notification arrives. This interruption handler intercepts the affected task and orchestrates recovery before compute resources vanish.

Jobs that cannot proceed or exceed maximum retry thresholds route into an Amazon SQS dead letter queue for engineering investigation. System health and interruption frequencies are monitored continuously through Amazon CloudWatch Metrics to maintain total operational visibility. This decoupled layout isolates container failures so that transient hardware reclaims never compromise end-to-end processing pipelines.

02Capacity provider strategies across three architectures

Rather than anchoring workloads to a single GPU type, the architecture spans three distinct NVIDIA hardware generations. The g4dn instance family uses NVIDIA T4 GPUs to provide an economical baseline for standard acoustic and language models. When workloads require greater throughput, the system deploys g5 instances equipped with NVIDIA A10G GPUs to accelerate compute-heavy pipelines.

Larger models and generative language pipelines run on g6 instances featuring NVIDIA L4 GPUs, delivering improved performance-per-dollar. Compute configurations span specific sizes including g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge, and g6.2xlarge. Spreading tasks across multiple instance sizes allows the platform to draw from substantially wider hardware pools across active AWS data centers.

Amazon EC2 provisions instances using the price-capacity optimized allocation strategy, selecting pools with deep spare capacity and low interruption probability. Amazon ECS managed scaling dynamically adjusts instance counts based on queue depth and assigned capacity provider weights. When incoming traffic subsides, the auto-scaling groups scale compute down gracefully without terminating active customer requests.

03Queue decoupling and two-minute interruption recovery

Customer transcription requests arrive through an asynchronous ingestion path that decouples job reception from execution using Amazon SQS. Individual transcription tasks take between fifteen seconds and several minutes depending on recording length. Worker containers polling the primary queue pull pending messages, retrieve matching audio recordings from Amazon S3, and execute processing stages.

When AWS issues a two-minute Spot interruption warning, tasks evaluate their remaining processing time. In-flight jobs capable of finishing within two minutes continue running and deliver their transcripts normally. Jobs requiring additional processing time immediately release their lock and return to the main queue so available workers can claim them.

A healthy GPU instance in another Availability Zone or instance family picks up the returned message to complete inference. Custom Amazon Machine Images containing pre-baked model weights and container images initialize in roughly two minutes to accelerate worker replenishment. Customers experience only a slight increase in processing duration rather than outright transcription failures.

04Regional isolation across London and Oregon

Enterprises in regulated sectors such as healthcare, financial services, and legal technology require that sensitive audio data never cross geographic boundaries. AssemblyAI addresses these compliance obligations by establishing independent infrastructure stacks within specific AWS Regions. The primary footprint operates across US West in Oregon (us-west-2) and Europe in London (eu-west-2).

Each designated region hosts an autonomous installation comprising local Amazon SQS queues, ECS clusters, capacity providers, and GPU fleets. Audio recordings uploaded within European boundaries are processed and stored exclusively inside the London region. This architecture ensures complete regulatory isolation without depending on cross-region data transfers or remote compute failovers.

Both the Oregon and London locations maintain competitive Spot pricing across g4dn, g5, and g6 instance families. Having access to multiple GPU architectures in European availability zones allows local fleets to absorb Spot interruptions reliably. Consequently, regulated institutions satisfy data residency mandates while benefiting from discounted compute economics.

05Sustainable unit economics across fourfold expansion

Operating inference workloads on EC2 Spot Instances reduced AssemblyAI's compute expenditure by seventy percent compared to equivalent On-Demand configurations. This efficiency gain proved critical as the company experienced a nearly fourfold year-over-year increase in aggregate compute demand. The multi-family architecture maintained predictable unit margins throughout this period of rapid platform expansion.

Production operations demonstrated that instance family diversification is the most reliable mechanism for suppressing Spot interruption rates. Because g4dn, g5, and g6 instances draw from isolated physical server pools, market pressure on one hardware type leaves alternative pools unaffected. Combining multiple GPU families ensures stable aggregate capacity even during constrained regional market conditions.

Pairing automated retries with fast container warm-up converts hardware interruptions into manageable background events rather than service outages. Organizations running batch or near-real-time deep learning models can adopt this architecture by decoupling ingestion and spreading compute across instance types. The pattern proves that fault-tolerant system design unlocks dramatic cost optimization without degrading enterprise availability commitments.