Use prompt caching with open-weight models on Amazon Bedrock
Amazon Bedrock added explicit prompt caching for open-weight models via the Chat Completions API, allowing prefix reuse with temporary 12-hour IAM tokens.
By Dillip Chowdary • Nov 16, 2026 • Source: AWS Machine Learning Blog

At a glance
- Product
- Amazon Bedrock prompt caching for open-weight models
- Feature mode
- Implicit and explicit prompt caching
- API
- Bedrock Chat Completions API
- SDK requirements
- Python openai 3.26.0+, aws-bedrock-token-generator 1.1.0+
- Agent package
- OpenCode 1.18.35
- Token validity
- Temporary API keys valid up to 12 hours
In this post
Amazon Web Services expanded prompt caching capabilities by adding explicit prompt caching support for open-weight models hosted on Amazon Bedrock. The capability allows generative AI applications to reuse computed prefix states across multiple inference calls through the Chat Completions API instead of reprocessing identical context. As detailed in AWS Machine Learning Blog's report, this optimization targets common multi-turn workloads that resend large system prompts, reference manuals, tool specifications, or conversation histories.
This report details the technical architecture, configuration procedures, and operational economics of deploying prompt caching with open-weight models on Amazon Bedrock. It provides engineering teams and AI developers with complete setup instructions for the OpenAI-compatible runtime endpoint, short-term authentication mechanisms, and cache usage telemetry. Readers will learn how to structure cache breakpoints to minimize inference latency and avoid unnecessary write costs during automated agent and conversational workloads.
01Prefix reuse across model requests
Generative AI applications frequently transmit identical reference materials, extensive tool definitions, and ongoing conversational context with every request. Amazon Bedrock eliminates redundant compute by caching the processed prefix states and serving matching subsequent requests directly from memory. User prompts and outputs remain private within customer environments and are never shared with model providers or used for training.
| Feature | Implicit caching | Explicit caching |
|---|---|---|
| Request changes | None required | Requires prompt_cache_options mode set to explicit and prompt_cache_breakpoint markers |
| What is cached | Eligible prefixes selected automatically on a best-effort basis | Content up to and including each specified cache breakpoint |
| Good fit | Stable prefix prompts including multi-turn conversations and agent loops | Fixed shared prefixes with changing user questions or layered prefixes |
Implicit caching operates automatically on a best-effort basis without modifying API payloads, making it suitable for multi-turn dialogues where earlier turns form the prefix for subsequent exchanges. Developers should position static content first and dynamic content last to maintain a stable prefix. If an early token sequence changes, the managed cache invalidates and rebuilds from that point.
02Cache write billing and breakpoint limits
Inference pricing on Amazon Bedrock treats cache reads at a substantially lower rate than standard input token processing, creating compounding savings across repeated requests. Depending on the model, cache writes can be billed at a higher rate than standard input. Because the service manages cache entries automatically, every successful cache hit resets the time to live duration for that prefix.
Workloads with changing user queries benefit from explicit caching because implicit caching can inadvertently write non-reusable dynamic questions to cache. Inserting explicit breakpoints limits cache writes strictly to the shared prefix, while subsequent content incurs standard input rates. This boundary prevents applications from paying elevated write rates on tokens that later requests will never reuse across the session.
03Runtime endpoint and authentication requirements
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Applications and autonomous agents interact with open-weight models through the Amazon Bedrock Runtime endpoint using standard OpenAI-compatible Chat Completions formats. Requests authenticate against the regional endpoint using temporary API keys generated dynamically from AWS Identity and Access Management credentials. This authentication workflow avoids creating or storing static, long-term API access tokens within application code or local environments.
Note: Model availability and prompt caching support vary by AWS Region, requiring users to verify regional support on individual model cards before deployment. Access also requires IAM invocation permissions for the chosen model along with compliance with both model-specific licenses and the AWS Responsible AI Policy.
The Python runtime environment requires version 3.12, the openai package at version 3.26.0 or later, and the aws-bedrock-token-generator library at version 1.1.0 or later. For coding agent workflows, the OpenCode agent installs at version 1.18.35 via the npm package manager. These packages enable local scripts to invoke Bedrock endpoints and inspect returned cache token usage.
04Virtual environment setup and client configuration
Setting up the environment involves creating a Python 3.12 virtual environment and installing the integration packages. The aws-bedrock-token-generator package provides the provide_token function, which turns active IAM credentials into a temporary Bedrock API key valid for up to 12 hours. This key authenticates the OpenAI client instance configured with the regional Bedrock runtime base URL.
- Environment setup Create a Python 3.12 virtual environment, activate it, and install openai version 3.26.0 or higher along with aws-bedrock-token-generator version 1.1.0 or higher using pip.
- Documentation asset creation Author manual.md containing approximately 4,000 tokens of Markdown documentation to satisfy model prefix length requirements, incorporating replacement schedules, error codes, and warranty details.
- Common client implementation Save common.py defining the target region, model identifier, system prompt, sample questions, OpenAI client instance, and a response reporting function to print cached token usage details.
The client configuration specifies a max_tokens value of 1024 so generated responses have sufficient headroom to complete. When open-weight models emit reasoning tokens during generation, those tokens count directly against the configured max_tokens limit. Establishing an adequate token limit prevents responses from terminating prematurely when complex queries produce extended reasoning traces alongside answers.
05AquaPure support manual and token telemetry
The reference implementation deploys a support assistant for a fictional AnyCompany AquaPure 500 water filter system. The system prompt pairs a three-sentence response limit with a 4,000-token product manual covering cartridge replacement schedules, operational error codes, and warranty terms. Specifically, the documentation details six-month sediment pre-filter and twelve-month carbon post-filter intervals alongside diagnostic procedures for error code E07.
Client code inspects cache efficiency by reading the prompt_tokens_details structure within the API usage payload. The cached_tokens field indicates tokens retrieved directly from memory, while the cache_write_tokens attribute records tokens written to cache during the request. Subtracting these values from total prompt tokens exposes the exact count of uncached tokens processed at standard input rates.
Frequently asked questions
What is the difference between implicit and explicit prompt caching on Amazon Bedrock?
Implicit caching automatically identifies and reuses eligible prompt prefixes on a best-effort basis without payload changes. Explicit caching requires setting the cache mode to explicit and designating cache breakpoints to mark exact reusable prefix boundaries.
How does prompt caching affect token billing on Amazon Bedrock?
Cached token reads are billed at a discounted rate compared to standard input tokens. Depending on the model, cache writes can be billed at a higher rate than standard input, making explicit caching useful for avoiding write charges on non-reusable dynamic queries.
How do applications authenticate with the Bedrock Chat Completions endpoint?
Applications use short-term API keys generated from active AWS IAM credentials using the provide_token function in the aws-bedrock-token-generator package. These temporary tokens remain valid for up to 12 hours without requiring long-term static keys.
What software dependencies are required to run prompt caching with the OpenAI SDK?
The implementation requires Python 3.12, openai package version 3.26.0 or later, and aws-bedrock-token-generator version 1.1.0 or later. For coding agent workflows, OpenCode version 1.18.35 can be installed via npm.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement