Adaptive limiters cut 429s by charging requests on estimated token cost, then reconciling actual usage. Build the pattern step by step. Read now.
Why Fixed Request Limits Break Down for AI APIs
Traditional rate limiting counts requests. Each call costs one unit, you allow N units per window, and anything over that gets a 429. This works when requests are roughly interchangeable, but AI API calls are not. One request might send a few words and get a short reply; another might send a long document and stream back thousands of tokens. Counting them as equal means you either set the limit low enough to survive the biggest request — wasting capacity most of the time — or set it high and get throttled the moment several large requests land together.
The real constraint is tokens, not requests. Providers meter and price by tokens, and that is where your ceiling actually lives. A limiter that ignores token cost is measuring the wrong thing, which is why request-based schemes still return 429s even when you are nowhere near the true limit.
Charging on Estimated Token Cost
An adaptive limiter charges each request against a token budget before it goes out. The trick is that you cannot know the exact cost in advance — output length is unknown until the model responds. So you estimate: count the input tokens you are about to send, add a projected output cost based on the request type or a max-tokens cap, and deduct that estimate from the current window's budget. If the estimate does not fit, you queue or reject locally instead of letting the provider reject you.
This front-loaded charge is deliberately conservative. Overestimating slightly means you occasionally hold back a request that would have fit, but you almost never blow through the ceiling. Since a local delay is cheap and a provider 429 forces a retry with backoff, biasing toward the safe side is usually the right trade.
Reconciling Actual Usage
After a response comes back, you know the real numbers — most APIs return actual input and output token counts. Now you reconcile: compare what you charged against what was truly consumed and refund or debit the difference to the budget. If you estimated 1,000 output tokens and the model used 300, the extra 700 goes back into the window immediately, freeing room for the next call.
Reconciliation is what makes the limiter adaptive rather than just pessimistic. Without it, conservative estimates would leave large amounts of budget stranded, and your effective throughput would sag well below the real limit. The estimate keeps you safe in the moment; the reconciliation recovers the slack so you stay efficient over time.
Building the Pattern Step by Step
You can assemble the whole thing from a few small pieces, each of which is straightforward on its own:
- A token-bucket budget sized to the provider's per-window token limit, refilling continuously rather than resetting in hard steps.
- An estimator that counts input tokens and projects output cost per request before sending.
- A reserve-then-send step that deducts the estimate and queues the call if the budget cannot cover it.
- A reconciliation step that reads actual usage from the response and adjusts the budget up or down.
Keep the estimator and the bucket separate so you can tune each independently — sharpen estimates without touching budget logic, or adjust headroom without rewriting the accounting. Start conservative, watch how often estimates overshoot, and tighten the projection as you gather real usage data. The result is a limiter that pushes close to the true ceiling while keeping 429s rare.