Exa boosts embedding throughput by 50% to index billions of documents
Exa optimized its document embedding pipeline on AWS Inferentia2 and Trainium2 chips, cutting costs by 47 percent to re-index 30 billion web pages.
By Dillip Chowdary • Nov 19, 2026 • Source: AWS Machine Learning Blog

At a glance
- Product
- AWS Neuron hardware
- Target chips
- AWS Inferentia2 and Trainium2
- Workload
- 30 billion document re-embedding
- Correctness gate
- 0.999 cosine similarity
- Throughput gain
- 47% performance and cost gain
- Primary region
- AWS US West (Oregon) us-west-2
In this post
Exa optimized its document embedding pipeline on AWS Neuron hardware to index tens of billions of web documents while cutting compute costs. According to AWS Machine Learning Blog's report, the search engine company achieved a 47 percent gain in performance and cost per million documents embedded on AWS Inferentia2 and AWS Trainium2 chips. The engineering effort combined custom Neuron Kernel Interface kernels, compiler layout adjustments, and an agentic optimization harness to accelerate high-volume semantic index ingestion.
The performance gains directly impact Exa's in-house vector database, which requires embedding every structured web document identified as high quality. By targeting the unit economics of embedding at web scale, the technical workflow demonstrates how search infrastructure teams can eliminate memory stalls and cross-core contention on specialized AI chips. This breakdown examines the hardware choices, memory layout optimizations, custom attention kernels, automated development workflows, and multi-region compute scheduling behind the migration.
01Thirty billion document backfill trigger
Exa relies on multi-tower semantic machine learning models to embed web data and queries into a unified vector space. The company ingests raw web data through crawlers into an asynchronous pipeline covering parsing, moderation, filtering, and embedding before serving data from its vector database. When Exa upgraded to a superior document model, the engineering team selected 30 billion high-priority documents for re-embedding.
The re-embedding initiative temporarily doubled overall inference load across the platform. Two parallel pipelines shared the crawl and parse stages, with one pipeline embedding live data into the active index and the other re-embedding priority documents for the new index. This doubled throughput demand prompted the team to evaluate AWS Neuron hardware family economics rather than scaling existing instances.
Exa benchmarked the inf2.xlarge instance type, which contains one Inferentia2 chip and two NeuronCores. The core embedding workload executes on a single NeuronCore, allowing Kubernetes pods to pack tightly onto larger instances without leaving cores idle. This architecture provides container homogeneity across the fleet while scaling through data parallelism across both Inferentia2 and Trainium2 chips.
02Progressive compiler and kernel milestones
The initial baseline deployment ran a data-parallel FP16 model representation on Inferentia2. Exa partnered with the AWS Trainium team to progressively raise documents-per-second throughput using the Neuron SDK. The first major enhancement implemented weight layout optimization, removing unnecessary transposes by aligning tensor memory structures with the model's broader execution scheduling.
The team subsequently enabled compiler autocast, allowing matrix multiplications to downcast to lower precision while preserving required dynamic range. Downcast outputs satisfied a strict correctness threshold of 0.999 cosine similarity against reference models. The team then integrated drop-in Neuron Kernel Interface library kernels for common layers, using a pre-built attention kernel to eliminate initial computational inefficiencies.
| Optimization milestone | Primary mechanism | Performance outcome |
|---|---|---|
| Baseline | Data-parallel FP16 | Initial baseline throughput |
| Weight alignment | Layout transpose elimination | Reduced layout transformation overhead |
| Autocast | Lower precision downcasting | Maintained 0.999 cosine similarity |
| Drop-in kernels | Standard NKI library layers | Rapid efficiency gains on common layers |
| Custom fused kernels | Weight-stationary SwiGLU and tailored attention | 47% performance and cost gain per million documents |
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Fused SwiGLU and attention customization
Neuron Explorer profiling revealed that high-bandwidth memory spill and reload cycles stalled the Tensor Engine during forward passes. Exa replaced pre-built implementations with a custom weight-stationary fused SwiGLU MLP kernel written in the higher-level NKI language. The team used high-level NKI for layout, loops, and memory operations, reserving lower-level NKI ISA instructions for specific tensor operations to keep code maintainable.
Benchmarking then revealed cross-core contention, as per-accelerator throughput with one idle core significantly outpaced runs with both cores active. Profiling demonstrated that the workload had not saturated the Tensor Engine, highlighting an opportunity to reduce total FLOPS. Exa rewrote the attention kernel around the exact tensor dimensions of its model head architecture, eliminating the wasted operations generic kernels spend on padding.
The customized attention kernel also folded multi-step operations into single matrix multiplications. This structural change eliminated intermediate inner-loop instructions and minimized high-bandwidth memory round trips. Together, the custom attention and fused MLP kernels delivered a 47 percent improvement in cost and throughput per million documents embedded over baseline compile.
04Multi-tier agentic development harness
Exa built an automated three-part harness to discover, validate, and profile optimizations across development iterations. Operating on the Flyte workflow orchestration platform, automated agents compiled NEFF Neuron executables and checked forward passes against a baseline validation dataset. Every candidate kernel was required to achieve a 0.999 cosine similarity against the canonical model before advancing to performance testing.
Candidate configurations that cleared the accuracy check entered an automated execution gate that ran a single forward pass profile followed by a 30-minute saturation benchmark on physical hardware. The harness deployed tiered models matched to specific tasks to control development costs. Lightweight models managed grid searches across compiler flags, reasoning models tested bottleneck hypotheses, and specialized code generation models authored custom NKI kernels using Neuron agentic development skills.
Note: Hardware availability and spot pricing fluctuate significantly across cloud regions, making single-region optimization vulnerable to capacity constraints.
05Cross-region Kubernetes compute scheduling
Exa initially focused its Inferentia2 optimizations within the AWS US West (Oregon) region, known as us-west-2. To avoid compute shortages caused by localized spot and on-demand capacity constraints, the company manages resources across multiple AWS regions using a centralized Kubernetes control plane. The scheduler evaluates real-time performance per dollar across instance types, shifting pods to lower-cost nodes as indexing demand fluctuates.
Because Exa designed its single-NeuronCore pod architecture for data parallelism, optimizations developed for Inferentia2 transfer directly to Trainium2 hardware. Dynamic scheduling enables the fleet to absorb unused reserved compute or release nodes needed by other internal services. This multi-region, multi-hardware deployment model protects long-lived embedding operations from cloud capacity bottlenecks while maximizing hardware cost efficiency.
Frequently asked questions
Why did Exa evaluate AWS Neuron chips for document embedding?
Exa needed to re-embed 30 billion high-priority web documents for a new semantic model, which temporarily doubled inference load and prompted a shift toward hardware with lower cost per document.
How did Exa resolve memory stalls on Inferentia2?
Profiling showed high-bandwidth memory spills, so Exa built a custom weight-stationary fused SwiGLU MLP kernel using the Neuron Kernel Interface to keep the Tensor Engine fully utilized.
How did Exa automate kernel optimization?
Exa used Flyte to run multi-model agent workflows that compiled NEFF executables, enforced a 0.999 cosine similarity correctness gate, and ran 30-minute saturation benchmarks on physical hardware.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement