Glossary: Serving LLMs at Scale

Load test

Also written: loadtest.py

Sending realistic traffic to a server and recording what each request experienced.