Skip to content

Glossary: LLMs in Production

Rate limiter

Also written: RateLimiter

A class that spaces out your own call starts and caps how many calls are in progress at once. It is set just under the provider's limits so you avoid 429 errors.

Where the book teaches it

Related terms

asyncio.Semaphoreasyncio.LockToken bucket