Glossary: Serving LLMs at Scale

Streaming

Also written: stream=True

Sending an answer in pieces as it is written, instead of all at once. It is how clients measure TTFT and inter-token latency.