2 million tokens is a superpower, but it costs a fortune. Learn strategies like
Why Bigger Context Still Costs Real Money
A two-million-token context window sounds like freedom: drop in whole codebases, long threads, and multi-document packs without aggressive truncation. That freedom has a price. Most APIs bill on tokens processed, so every message that re-sends a large history multiplies cost. Latency and rate limits often scale with input size as well. The trap is treating “max context” as the default strategy instead of a tool you use only when the extra tokens change the answer.
Think in terms of working set, not maximum capacity. You need the facts that drive the current decision—not every file, ticket, or chat that might be relevant someday. A smaller, carefully chosen prompt is usually faster, cheaper, and easier to debug than a dump that “might help.”
Shrink the Working Set Before You Expand the Window
Start by deciding what the model must know for this turn. Prefer summaries, interfaces, and error excerpts over full files. Keep stable facts—project rules, schema sketches, coding conventions—in a short, reusable preamble instead of re-pasting them from raw sources every time. When you do need raw material, pin only the sections that map to the task: the failing function, the relevant config block, the API contract under change.
- Replace long chat history with a concise decision log: goals, constraints, rejected options, and current state.
- Retrieve first, then load: search or filter so only ranked snippets enter the prompt.
- Prefer structured notes (bullet lists, tables of fields, step checklists) over narrative dumps.
- Drop redundant copies of the same code or docs that appear under different paths.
If the model keeps re-asking for context you already sent, the problem is usually organization, not window size. Clarify roles, mark what is authoritative, and separate “background” from “current task” so attention is not wasted on noise.
Design Multi-Step Flows Instead of One Giant Prompt
Large tasks rarely need everything in one call. Break work into stages: outline or plan with a small context, implement or analyze a single slice with focused context, then integrate. Each stage carries only what it needs and produces a short artifact for the next stage. That pattern cuts repeated full-history charges and makes failures local—you re-run one step instead of replaying a massive prompt.
Cache what you can on your side: store intermediate summaries, extracted APIs, and verified facts in your own notes or files. Feed those back in later turns rather than re-reading original sources. When a provider offers prompt caching or similar reuse for stable prefixes, structure prompts so the unchanging part is front-loaded and the variable task sits at the end. Even without special features, a stable system block plus a thin user message is cheaper than a growing transcript that re-sends everything.
Measure Cost by Outcome, Not by Tokens Available
Track rough token use per task type—bug fix, review, research synthesis—and notice when quality stops improving as input grows. If answers stay the same after you double the context, you are paying for unused capacity. Set soft budgets: a default small context for routine work, and an explicit “expand” path only when retrieval fails or the model lacks a critical dependency.
Two million tokens is a superpower when the problem truly spans a large corpus: cross-module refactors, multi-doc policy analysis, long-running investigations. For everyday coding and writing, disciplined selection, staged prompts, and local memory beat maxing the window. Spend the large budget on the few cases where breadth is the product, not the habit.