Rate Limiting Algorithms: Token Bucket vs Sliding Window 

A weather app’s backend team once shipped a public API with no rate limiting at all, reasoning that their traffic was small enough not to matter. Three weeks later, a well-meaning developer building an automated dashboard wrote a polling loop with no delay, hammering the /forecast endpoint thousands of times a minute from a single laptop.

The API’s database connection pool exhausted within minutes, and every other client, including the paying customers the API was built for, started seeing timeouts. Nobody had done anything malicious; there simply wasn’t a mechanism to say “slow down.”

That gap is what rate limiting exists to close, and the algorithm you choose determines how gracefully your system handles the next runaway client. 

A Traffic Spike That Took Down an API 

Rate limiting sits at the boundary between a service and the world, deciding which requests get through immediately, which get delayed, and which get rejected outright. Without it, a single misbehaving client, a retry storm, or a scraper can consume resources meant for everyone else, and there’s no way to distinguish “busy legitimate traffic” from “a bug in someone’s client code.” 

The mechanism matters as much as the decision to rate-limit at all, because a poorly chosen algorithm can reject legitimate bursts of traffic while still letting sustained abuse through, or it can be so permissive that it fails to protect the backend during the exact moment it’s needed. Two algorithms dominate real-world systems, token bucket and sliding window, and knowing their mechanics is what lets you pick correctly for a given endpoint. 

Token Bucket Mechanics

The token bucket algorithm models each client (or API key, or IP address) as a bucket that holds a limited number of tokens. Every incoming request consumes one token; if the bucket is empty, the request is rejected or queued. Tokens refill at a steady rate, say, ten tokens per second, up to the bucket’s maximum capacity. 

This design naturally allows bursts: a client that has been idle accumulates tokens up to the bucket’s capacity, then can spend them all at once on a sudden flurry of requests, before falling back to the steady refill rate. That burst tolerance is often exactly what you want, a mobile app resuming from the background shouldn’t be throttled just because it needs to make several requests to refresh its state. 

A minimal implementation looks like this: 

import time 

class TokenBucket: 

def __init__(self, capacity: int, refill_rate: float): 

self.capacity = capacity 

self.tokens = capacity 

self.refill_rate = refill_rate # tokens per second 

self.last_refill = time.monotonic() 

def allow_request(self) -> bool: 

now = time.monotonic() 

elapsed = now - self.last_refill 

gained = elapsed * self.refill_rate 

self.tokens = min(self.capacity, self.tokens + gained) 

self.last_refill = now 

if self.tokens >= 1: 

self.tokens -= 1 

return True 

return False

Key properties worth calling out: 

• Burst capacity: controlled independently from the sustained rate, via the bucket size.
• Smooth refill: tokens trickle back in continuously rather than resetting all at once.
• Low memory footprint: only two numbers (token count and last refill time) need to be stored per client. 
• Predictable long-term rate: over any sufficiently long window, throughput converges to the refill rate. 

Sliding Window Counters and Logs 

Sliding window approaches take a different angle: instead of modeling a refillable resource, they count how many requests really occurred in a recent time window and compare that count against a limit.

The simplest version, the sliding window log, stores a timestamp for every request a client makes and, on each new request, discards timestamps older than the window and counts what remains. This is precise, it never over- or under-counts, but it’s memory-intensive at scale, since a busy client can generate a large number of stored timestamps. 

The more common production variant, the sliding window counter, approximates this by keeping counts in fixed buckets (for example, one bucket per second) and computing a weighted combination of the current and previous bucket based on how far into the current window you are: 

def sliding_window_allowed(current_bucket_count, previous_bucket_count, elapsed_fraction, limit): 

weighted_count = (previous_bucket_count * (1 - elapsed_fraction) 

+ current_bucket_count) 

return weighted_count < limit

This trades a small amount of precision for a large reduction in memory and computation, which is why it’s the version most rate-limiting libraries and API gateways really ship. 

Fixed Window vs Sliding Window 

It’s worth contrasting sliding window against the simpler fixed window approach, since fixed window is often the first thing teams build and the first thing they regret. Fixed window counters reset a counter to zero at the start of every window, every minute, say, and reject requests once the counter hits the limit. 

The failure mode is a boundary effect: a client can send the maximum allowed requests in the last second of one window, then immediately send the maximum allowed requests again in the first second of the next window, doubling the effective rate for a brief period around the boundary. For a limit of 100 requests per minute, that means up to 200 requests could land in a two-second span straddling the reset. 

• Fixed window: simplest to implement, cheapest to compute, but vulnerable to boundary bursts.
• Sliding window log: most accurate, but memory cost scales with request volume.
• Sliding window counter: a practical middle ground, smoothing the boundary problem without storing every timestamp. 
• Token bucket: naturally supports intentional bursts, decoupled from a fixed calendar boundary. 

Most API gateways, Kong, Envoy, and cloud-provider offerings like AWS API Gateway, default to some variant of token bucket or sliding window counter precisely because the fixed window’s boundary problem is well documented and easy to avoid. 

It’s worth noting that fixed window still shows up often in practice, and not always by mistake. Its simplicity makes it a reasonable choice for coarse, low-stakes limits, a nightly batch export capped at “50 per day,” for instance, where a brief doubling around midnight barely matters. The mistake is

applying fixed window to a limit that’s meant to protect a fragile backend resource in real time, where a short burst of double traffic is exactly the scenario the limiter was supposed to prevent. Choosing between these algorithms, in other words, is less about which one is objectively superior and more about matching the failure mode you can tolerate to the traffic pattern you really expect. 

Distributed Rate Limiting With Redis 

Rate limiting gets harder the moment your service runs on more than one instance, because a counter or bucket that lives in a single process’s memory only sees the traffic that process handled. A client could be limited to 100 requests per minute per server, but if there are ten servers behind a load balancer, the effective limit becomes 1,000. 

The standard fix is centralizing the counter in a shared, fast store, Redis is the overwhelming favorite, because its single-threaded command execution and atomic operations make race-free counting straightforward. A common pattern uses Redis’s INCR and EXPIRE together, or a Lua script to make the check-and-increment atomic: 

local current = redis.call("INCR", KEYS[1]) 

if tonumber(current) == 1 then 

redis.call("EXPIRE", KEYS[1], ARGV[1]) 

end 

if tonumber(current) > tonumber(ARGV[2]) then 

return 0 

end 

return 1

This gives every application instance a consistent view of how many requests a given client has made, at the cost of a network round trip to Redis on every request. For very high-throughput systems, teams sometimes shard the Redis layer itself, or accept a small amount of over-counting in exchange for local, in-memory approximations that sync periodically. 

Choosing Limits for Different Endpoints 

Not every endpoint deserves the same limit, and applying a single global rate limit across an entire API is usually a sign the limiting strategy hasn’t been thought through carefully enough. A read-heavy, cacheable endpoint can typically absorb far more traffic than a write endpoint that triggers a database transaction or an expensive computation. 

• Authentication endpoints: tight limits per IP address to slow down credential-stuffing attempts, often layered with account lockout logic. 
• Search or query endpoints: moderate limits, since these are often computationally expensive even though they’re reads. 
• Write endpoints: the strictest limits, since each request has downstream cost in database writes, cache invalidation, or triggered workflows.
• Bulk or export endpoints: separate, low limits with longer windows, since these are inherently resource-intensive and rarely need high-frequency access. 
• Internal service-to-service calls: often exempted or given much higher limits, since the traffic pattern and trust level differ from public clients. 

Tiered limits by client type, free tier, paid tier, internal service, are common as well, usually keyed by API key or authenticated account rather than IP address, since IP-based limiting breaks down for clients behind shared NAT gateways or corporate proxies. 

It also helps to separate limits by cost rather than by endpoint name alone, because two endpoints that look similar on the surface can have wildly different backend cost. A GET /users/:id lookup served from a cache might tolerate thousands of requests per minute per client without strain, while a GET /reports/annual-summary endpoint that triggers a heavy aggregation query across months of data might only tolerate a handful.

Some teams formalize this with a “cost” or “weight” assigned to each endpoint, so a single shared quota (say, 1,000 points per minute) can be spent faster on expensive calls and slower on cheap ones, which mirrors how GitHub’s GraphQL API and several other public APIs handle rate limiting today.

This approach is more work to build than a flat per-endpoint limit, but it avoids the awkward situation where a client has to track a dozen separate quotas for a dozen separate routes. 

Failure Modes of Rate Limiters 

A rate limiter is itself a piece of infrastructure, and it can fail in ways that are easy to overlook until they cause an incident. If the shared store backing a distributed rate limiter (commonly Redis) becomes unavailable, the application has to decide whether to fail open (allow all requests, risking overload) or fail closed (reject all requests, risking a full outage over what should have been a soft-limiting mechanism). 

Clock skew across servers can cause window-based algorithms to disagree about which window a request falls into, especially in systems that compute windows locally rather than relying on the shared store’s own clock. Poorly chosen limit keys, such as rate-limiting by a header a client can trivially spoof, provide no real protection at all. And limits set without headroom for legitimate retries can create a feedback loop: a client hits the limit, its retry logic fires immediately, and the retries themselves keep the client permanently rate-limited. 

There’s also a subtler failure mode worth watching for: the rate limiter itself becoming the slowest part of the request path. If every request has to make a round trip to a centralized Redis instance before it can proceed, and that Redis instance is under-provisioned or sitting in a different availability zone, the limiter adds latency to every single request, including the ones it ultimately allows through.

Teams that hit this usually respond by co-locating the rate-limiting store closer to the application tier, batching or pipelining the Redis calls, or accepting a slightly relaxed, eventually-consistent local cache of the limit state that only syncs with the central store periodically.

None of these are free, which is exactly why rate limiting deserves the same monitoring and capacity planning as any other piece of critical-path infrastructure, rather than being treated as a fire-and-forget middleware. 

Client-Side Backoff and Retry Behavior 

Rate limiting only works as intended if clients respond sensibly to being throttled, and a well-designed API makes that easy by returning clear signals. The HTTP 429 Too Many Requests` status code, paired with a Retry-After header, tells a well-behaved client exactly how long to wait before trying again. 

• Exponential backoff: doubling the wait time after each consecutive failure, preventing synchronized retry storms. 
• Jitter: adding randomness to the backoff interval so many clients don’t retry at exactly the same moment. 
• Respecting `Retry-After`: honoring the server’s explicit guidance rather than guessing at a delay.
• Circuit breaking on the client: giving up temporarily after repeated failures rather than retrying indefinitely. 

APIs that expose remaining-quota headers (X-RateLimit-Remaining, X-RateLimit-Reset) give clients enough information to self-throttle before hitting the hard limit at all, which reduces rejected requests and gives client developers a much better integration experience than discovering the limit through trial and error. 

Server-driven throttling, where a service starts adding artificial latency to responses as a client approaches its limit, rather than waiting for a hard cutoff, is another pattern worth considering for internal APIs where the consuming teams are known and reachable.

It smooths the transition from “fully allowed” to “fully blocked” and gives client-side monitoring a chance to notice degraded response times before requests start failing outright, which tends to produce fewer surprised pages during an on-call rotation than a limiter that behaves perfectly right up until the exact moment it doesn’t. 

Final Thoughts 

Rate limiting looks like a small, mechanical piece of infrastructure until the day it isn’t there and something breaks because of it. The choice between token bucket and sliding window isn’t about picking a “better” algorithm in the abstract, it’s about matching the algorithm’s behavior to how your traffic really looks, whether that’s bursty and forgiving or steady and strict.

Token bucket earns its popularity by tolerating the bursts real clients naturally produce; sliding window earns its place wherever precise, boundary-free counting matters more than burst tolerance.

Whichever you choose, the algorithm only does half the job, clear signals back to clients, sensible per-endpoint limits, and a defined failure mode for when the limiter itself has trouble are what turn rate limiting from a blunt instrument into infrastructure people barely notice, which is exactly the point.

Frequently Asked Questions 

Which algorithm should a new API default to? 

A token bucket with a reasonably generous burst capacity is a sensible default for most public APIs, since it tolerates normal usage patterns like a client catching up after a network hiccup while still enforcing a steady long-term rate. 

Can rate limiting be enforced entirely at the API gateway layer? 

Often, yes, for coarse-grained limits like requests per API key or per IP. Endpoint-specific or business-logic-aware limits (like “five password resets per account per hour”) usually need to live in the application layer, where the relevant context is available. 

How do you rate-limit unauthenticated traffic fairly?

IP-based limiting is the common fallback, accepting that clients behind shared NAT or corporate proxies will share a quota. Some APIs mitigate this by combining IP with other signals, like a device fingerprint or a lightweight token issued on first contact. 

Does rate limiting protect against denial-of-service attacks? 

It helps against unsophisticated or accidental overload, but a well-resourced, distributed denial-of-service attack usually needs to be handled upstream, by a CDN or DDoS mitigation service, before traffic ever reaches the rate limiter. 

Should rate limits be the same for all HTTP methods? 

No. GET requests are typically cheaper than POST, PUT, or DELETE, which often trigger writes or side effects, so many APIs apply separate, stricter limits to state-changing methods. 

What’s a reasonable way to communicate limits to API consumers? 

Document the limits clearly, return standard rate-limit headers on every response (not just when a client is throttled), and provide a 429 response with a Retry-After value so client libraries can implement backoff automatically instead of guessing.