Skip to content

Glossary: LLMs in Production

Token bucket

A rate limit that refills continuously, so 1,000 a minute behaves like about 16 a second. A burst of 100 in one second can earn 429s even if the minute's total is fine.

Where the book teaches it

Related terms

Rate limitRate limiter