Cut self-hosted LLM cost and latency with the distributed KV caching agent
Amazon Web Services launched a distributed KV caching agent skill that pairs ElastiCache with self-hosted LLMs to cut inference costs by up to 40 percent.
By Dillip Chowdary • Nov 21, 2026 • Source: AWS Machine Learning Blog

At a glance
- Product
- Agent Toolkit for AWS KV caching skill
- Cost reduction
- Up to 40%
- Latency reduction
- Up to 80% time to first token
- Engine support
- vLLM 0.11.1+ and SGLang 0.5.13+
- Connector version
- LMCache 0.5.8rc1 or later
- Cache hardware
- cache.i8ge.48xlarge ElastiCache nodes
In this post
Amazon Web Services launched a distributed key-value caching agent skill inside the Agent Toolkit for AWS to automate evaluating and setting up shared inference memory for self-hosted large language models. The software integrates with coding assistants to evaluate GPU clusters, model configurations, and repetitive token patterns, according to AWS Machine Learning Blog's report. Using an external cache fleet powered by Amazon ElastiCache, the architecture cuts inference operational costs by up to 40 percent and reduces time to first token by up to 80 percent by eliminating redundant prefill computation across distributed GPU workers.
This guide details how engineering teams operating private AI models on Amazon Elastic Compute Cloud GPU fleets evaluate, size, deploy, and debug distributed attention caches. It covers the prerequisite software versions, offline simulation tooling, and administrative workflows required across multi-tenant inference infrastructure. System architects running large context windows on vLLM or SGLang can automate cluster provisioning while enforcing security group and tenant isolation policies.
01Prefill latency and distributed memory tiers
Self-hosting large language models on graphics processing units forces inference engines to recompute key-value attention states whenever context exceeds local hardware capacity. Long-running conversations, coding sessions, and legal document workflows routinely accumulate hundreds of thousands of context tokens. When a user sends a follow-up query after a pause, the serving instance rarely retains the prior session state in on-chip memory.
The receiving GPU must process the entire conversation history through a prefill phase before generating the first new token. In tests involving 64,000-token prompts on p5 instances, cold prefill processing produced first-token response times of 13 seconds. Distributed key-value caching places a shared storage tier alongside the GPU cluster, allowing any node to retrieve stored attention states across routing shifts and server interruptions.
Note: Hardware configurations require GPU instances that support Elastic Fabric Adapter Remote Direct Memory Access write capabilities, such as p5 instance types, paired with ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.
02Agent skill checks and workload validation
The distributed key-value caching skill inside the Agent Toolkit for AWS guides assistants like Kiro, Claude Code, and OpenAI Codex through pre-deployment validation. Before modifying infrastructure, the assistant issues a read-only describe-instances command to inspect the GPU fleet and count Elastic Fabric Adapter network interfaces. It prompts operators only for missing parameters including engine type, model architecture, tensor-parallel size, key-value data type, and request repetition frequencies.
Short prompts do not always benefit from shared caching because network transfer latencies can exceed raw calculation times. The assistant calculates the specific prompt length cutover point where reading from ElastiCache outperforms local recomputation. It also checks local prefix caching settings, warning engineers that workloads maintaining a 90 percent local hit rate via session affinity do not need external cache nodes.
For multi-tenant applications like legal assistants, the agent structures cache configurations to prevent timing side-channel data leaks between organizations. Tenants sharing infrastructure receive isolated cache spaces, or their unique organization identifiers are appended directly to key-value hashing functions. The assistant enforces consistent hashing seeds and explicit data types across all workers to eliminate lookup mismatches.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Capacity estimation with offline trace analysis
Engineering teams evaluate potential cost savings using KVTrace, a zero-cost simulation utility that records operational events directly from running engines. For an example cluster of 8 p5 instances running 64,000-token prompts, the assistant analyzes GPU costs, cache node pricing, and benchmarked prefill throughput. In this scenario, the agent recommends provisioning two cache.i8ge.48xlarge ElastiCache nodes.
The tool generates empirical hit-rate projections without storing sensitive prompt contents or raw token identifications. KVTrace captures only block hashes and token lengths across representative timeframes like full business days, with an optional flag available to sanitize hash values. Sizing algorithms pinpoint the smallest cluster configuration that captures 90 percent of maximum theoretical savings, noting that freed compute time yields dollar savings only when rightsizing instance counts or absorbing extra traffic.
| Parameter | Specification |
|---|---|
| GPU fleet example | 8 p5 instances |
| Context prompt size | 64,000 tokens |
| Recommended cache | 2 cache.i8ge.48xlarge nodes |
| Runtime engines | vLLM 0.11.1+ or SGLang 0.5.13+ |
| Connector requirement | LMCache 0.5.8rc1 or later |
04Orchestrated provisioning across cache clusters
Executing cache deployment through the assistant requires explicit human approval before creating billable resources in target Availability Zones. The automated workflow constructs dedicated network rules and provisions an ElastiCache cluster operating Valkey 9.1 with eviction policies. The system writes dedicated credentials to AWS Secrets Manager and launches an LMCache background server for each GPU host in the inference fleet.
- Fleet inspection The agent runs read-only commands to inventory GPU nodes, count network adapters, and confirm engine compatibility.
- Cluster configuration The assistant requests operator confirmation and provisions cache.i8ge.48xlarge Valkey nodes with assigned parameter groups.
- Secret generation System credentials for the LMCache connector are generated and stored inside AWS Secrets Manager.
- Node initialization The agent initiates LMCache server instances on every GPU host, linking them to vLLM or SGLang engines.
- Bus verification The orchestrator writes a test payload across Remote Direct Memory Access channels to confirm read and write speeds.
Before handling live inference traffic, the assistant flushes local server tiers and conducts validation runs to confirm output accuracy. Identical prompt submissions must return consistent token outputs when loaded from the cache layer, preventing silent failures caused by misaligned memory layouts.
05Diagnostic workflows and transfer troubleshooting
When operational errors occur during cache transfers, the agent analyzes diagnostic telemetry to isolate root causes. In an operational example where LMCache successfully executed connection checks but logged increasing transfer errors under large-object monitoring, the skill identified missing intra-group permissions. The assistant diagnosed that while initial coordination succeeded, the underlying security group blocked direct Remote Direct Memory Access data transfers between cluster members.
The assistant generates targeted shell commands to update security group rules without requiring full environment teardowns. It also automates checks across serving frameworks including Ray Serve and llm-d, ensuring that all workers share identical tensor-parallel dimensions. Operators can systematically verify cache health metrics and memory consumption directly from the coding assistant interface.
Frequently asked questions
What software versions are required to run distributed KV caching?
Deployments require LMCache 0.5.8rc1 or later, vLLM 0.11.1 or later (or SGLang 0.5.13 or later), and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.
How does KVTrace estimate cache performance without exposing private data?
KVTrace records only block hashes and token counts during runtime execution, completely omitting raw prompt text and token IDs, and supports an anonymization flag for extra security.
Which GPU instances support the distributed KV caching architecture?
The architecture requires GPU instances equipped with Elastic Fabric Adapter interfaces supporting Remote Direct Memory Access write operations, such as AWS p5 instances.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement