Skip to content

Glossary: LLMs in Production

Load shedding

Refusing some requests quickly (with a 503 and Retry-After) so the rest stay fast, instead of letting a queue grow until everyone times out.

Where the book teaches it

Related terms

GateLoad test