Skip to content

Glossary: Serving LLMs at Scale

Cost per million tokens

Also written: cost per 1M tokens

A GPU's hourly price divided by the tokens it produces per hour, times a million. It is the best case, reached only at full load, and is what you compare with a hosted API's price.

Where the book teaches it

Related terms

Little's lawN+1