The Runtime Theory

Token Bucket Rate Limiter: Refill, Consume, Allow

See the token bucket in action: how refill at a fixed rate, token consumption per request, and the burst allowance decide which requests pass and which get rejected.

The Runtime Theory Team08 stages

trace / request.md

BUCKET IS CREATEDREQUESTCARRIES ITS KEYTOKENS ACCRUEWITH TIMEONE TOKENPER REQUESTREQUEST PASSESEMPTY BUCKETREJECTSBURSTS RIDETHE RESERVEREFILLNEVER STOPS

readyThe limiter keeps a bucket per key — user, IP, or API key — with a capacity (the maximum burst) and a refill rate. Bucket state lives in the limiter's store, often Redis, for shared decision-making.

The token bucket is the rate limiter that behaves like a fluid. Capacity is the bucket size, the refill rate is the faucet, and each request is one token drained. This diagram walks one request through the decision: load the bucket, compute earned tokens from elapsed time, consume one, allow or reject. The lazy refill — computing accrued tokens from the timestamp instead of running a timer — is the implementation detail that makes it cheap: the limiter does no work between requests.

Two properties fall out of the mechanics. Burst tolerance: a quiet minute fills the bucket to capacity, so a sudden spike of up to capacity requests passes before the faucet rate takes over. And smoothness: unlike a fixed window that resets to zero, the bucket never rejects a request that the average rate can afford. The stages are ordered by what the limiter knows at each moment — and the atomic consume step is what keeps concurrent requests honest.