Token buckets limit LLM cost and latency per tenant while backpressure prevents retry storms; build a Redis-backed limiter for production. Read now.
Why Rate Limiting Has to Be Tenant-Aware
When many customers share one LLM-backed service, a single tenant can consume a disproportionate share of your capacity — running large batch jobs, retrying aggressively, or sending unusually long prompts. Because LLM calls carry real per-token cost and variable latency, one heavy tenant does not just slow itself down; it degrades response times and inflates spend for everyone. A global rate limit cannot express this, since it treats all traffic as one undifferentiated stream.
Tenant-aware limiting attaches quotas to the identity making the request rather than to the service as a whole. Each tenant gets its own budget, so noisy usage is contained to the account that created it. This keeps cost predictable per customer and makes latency something you can reason about and, if you sell tiers, something you can price.
Token Buckets for Cost and Latency
The token bucket is a good fit here. Each tenant has a bucket that refills at a steady rate up to a maximum capacity. A request is admitted only if the bucket has enough tokens; otherwise it is rejected or delayed. The refill rate sets the sustained throughput a tenant is allowed, while the bucket capacity sets how large a short burst can be before the limit bites.
For LLM apps, the unit in the bucket does not have to be "one request." Because cost and latency scale with the number of model tokens processed, you can charge the bucket by estimated or actual token count. That aligns the limiter with what you actually pay for and what actually drives latency, so a few very large prompts are treated as heavier than many tiny ones.
- Refill rate — the tenant's sustained allowance over time.
- Bucket capacity — how much burst you tolerate above that rate.
- Cost per request — flat per call, or weighted by token count for accuracy.
Backpressure Instead of Retry Storms
Rejecting an over-limit request is only half the design. If clients respond to rejection by immediately retrying, they generate a retry storm: the limiter keeps saying no, the client keeps hammering, and the extra traffic makes the overload worse. Backpressure is how you tell callers to slow down rather than simply bounce them.
Signal the limit clearly and give the caller something to act on. Return an explicit rejection with a hint of when capacity will be available, so a well-behaved client can wait instead of spinning. Pair this with client-side backoff and jitter so retries spread out rather than synchronize. The goal is to convert a wall of failed retries into an orderly queue that drains as buckets refill.
Building It Redis-Backed for Production
In production you almost always run several application instances, so the limiter state cannot live in each process's memory — every instance would enforce its own private quota and the real limit would be a multiple of what you intended. A shared store like Redis gives all instances one authoritative view of each tenant's bucket. Keying by tenant lets you store the current token count and last-refill timestamp, and update them atomically so concurrent requests cannot double-spend the same tokens.
Make the check-and-deduct step atomic, evaluate it on the hot path before you call the model, and let each bucket expire when a tenant goes idle so the store does not grow without bound. Emit metrics on admits, rejects, and remaining tokens per tenant, so you can spot who is hitting limits and tune each tenant's refill rate and capacity from real usage.