Skip to content

Glossary: Serving LLMs at Scale

Counter

Also written: gauge

Two kinds of metric. A counter only goes up (tokens produced so far); a gauge goes up and down (requests running now).

Where the book teaches it

Related terms

PrometheusHistogram