Three core algorithms shape API throttling: token bucket, leaky bucket, and fixed window. Compare tradeoffs, configs, and examples. Full breakdown.
What rate limiting actually controls
API rate limiting caps how often a client can call an endpoint so shared systems stay stable under burst traffic, abuse, or uneven demand. Three algorithms dominate production designs: token bucket, leaky bucket, and fixed window. Each answers the same question—may this request proceed right now?—with different rules for counting, refilling, and rejecting. Your choice affects burst tolerance, fairness, and how easy the limit is to explain to clients.
Pick an algorithm by matching traffic shape to policy intent. If short spikes are normal and safe, favor a design that allows bursts. If you need a smooth, predictable outflow, favor a design that smooths. If you need a limit that is simple to reason about and instrument, a fixed window is often enough.
Token bucket: bursts with a refill budget
A token bucket holds a maximum number of tokens. Each allowed request consumes one token. Tokens refill at a steady rate up to the bucket capacity. When the bucket is empty, requests are rejected or delayed until more tokens appear. Capacity sets the largest burst you will accept; the refill rate sets the sustained average you will allow over time.
This model fits APIs where clients occasionally send a cluster of requests (page loads, batch jobs, retries) but should not sustain that pace forever. Configure capacity from the largest legitimate burst you can serve without harming others, and set refill so average usage stays inside backend capacity. Document both numbers: clients need to know how large a burst they can send and how quickly allowance recovers.
Leaky bucket and fixed window: smooth vs simple
A leaky bucket processes requests at a constant outflow rate. Incoming work queues (or is dropped) when it arrives faster than the leak. The result is a smooth, nearly constant request rate into downstream systems. That stability is valuable when backends are sensitive to spikes, but legitimate bursts may wait or fail even when the long-term average is fine. Queue length becomes a key config: too short and you drop under brief load; too long and latency climbs.
A fixed window divides time into equal slices (for example one minute) and counts requests in the current slice. Once the count hits the limit, further requests wait for the next window. Implementation is straightforward and easy to report in headers or dashboards. The main tradeoff is edge clustering: a client can use almost a full quota at the end of one window and again at the start of the next, roughly doubling short-term load. Sliding or overlapping windows reduce that edge effect at the cost of more state and complexity.
Choosing configs and applying them in practice
- Token bucket — set capacity for allowed burst size; set refill for sustained rate; reject or queue when empty.
- Leaky bucket — set leak rate to match safe backend throughput; set queue depth for how much temporary backlog you accept.
- Fixed window — set window length and max count per key (user, API key, IP); watch for double-spend at window boundaries.
Apply limits per identity and per route when cost differs: a cheap read and an expensive write should not share one naive counter. Prefer clear responses when a request is limited—status, remaining quota, and when the client may retry—so callers can back off without thrashing. Start conservative on shared hot paths, measure real traffic shape, then raise capacity or shorten windows only where you see false positives. The right algorithm is the one whose failure mode you can live with: token bucket under bursts, leaky bucket under load smoothing, fixed window under operational simplicity.