Skip to content

Glossary: LLMs in Production

Load balancer

A component that spreads requests across servers. It can also retry a slow request, which is one cause of duplicate requests.

Where the book teaches it

Related terms

ReplicasIdempotency key