Glossary: Serving LLMs at Scale
Percentile
Also written: p50, p95, median
The value that a given share of results beat: p50 is the median, p95 the value 95% of requests beat. Averages hide the customers who suffer, so targets are set on p95.
Glossary: Serving LLMs at Scale
Also written: p50, p95, median
The value that a given share of results beat: p50 is the median, p95 the value 95% of requests beat. Averages hide the customers who suffer, so targets are set on p95.