Glossary: Serving LLMs at Scale
Counter
Also written: gauge
Two kinds of metric. A counter only goes up (tokens produced so far); a gauge goes up and down (requests running now).
Glossary: Serving LLMs at Scale
Also written: gauge
Two kinds of metric. A counter only goes up (tokens produced so far); a gauge goes up and down (requests running now).