Glossary: Serving LLMs at Scale

Shadow traffic

Copies of real questions sent to a new system while customers still see only the old system's answers, so the two can be compared before switching.