Grok 4.5 Prompt Caching & Context Cost Control
By Dillip Chowdary • July 21, 2026 • Grok 4.5 engineering series
Put stable tokens first
Cache hit rates collapse when every request mutates the system prompt. Keep policies, tool schemas, and style guides as a stable prefix.
Put volatile content last: user input, retrieved docs, and per-request IDs. That ordering matches how most provider caches fingerprint prefixes.
Measure cache hit ratio daily. A 5% drop often means someone “temporarily” edited the system prompt in code.
Context budgets that survive long sessions
Allocate explicit budgets: system 15%, tools 20%, retrieval 40%, dialogue 25% (tune per product).
Summarize older turns into a rolling memory block instead of replaying full transcripts. Keep the last N raw turns for fidelity.
For RAG, prefer fewer high-scoring chunks over dumping entire pages. Token bloat is usually retrieval, not the model.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
What to never put in the cached prefix
Timestamps, request IDs, and user PII invalidate or personalize cache keys and leak across tenants if mis-scoped.
Tenant-specific policies belong in a short uncached segment after the shared prefix, not inside the global system prompt.
Feature flags should swap named policy blocks, not rewrite prose ad hoc per deploy.
Observability for cost
Log estimated input/output tokens per route and per agent skill. Alert on p95 cost per successful user task.
A/B test shorter system prompts with evals. Cheaper is only a win if task success stays flat.
Document a kill switch: degrade to a cheaper model or disable non-critical agents when spend spikes.
Advertisement