Glossary: Serving LLMs at Scale
Knee
The point in a concurrency sweep where latency shoots up because requests start queueing. Run below it.
Glossary: Serving LLMs at Scale
The point in a concurrency sweep where latency shoots up because requests start queueing. Run below it.