Skip to content

Glossary: LLMs in Production

Gate

A cap on how many requests are being worked on and how many may wait. When both are full, new requests are shed with a fast 503.

Where the book teaches it

Related terms

Load sheddingasyncio.Semaphore