Skip to content

Glossary: LLMs in Production

API layer

The web service in front of the model (FastAPI here). It checks who is asking and what they sent, enforces limits, logs the request and streams the answer back.

Where the book teaches it

Related terms

Model gatewayFastAPIQueue / workers