Glossary: Serving LLMs at Scale
Cost per million tokens
Also written: cost per 1M tokens
A GPU's hourly price divided by the tokens it produces per hour, times a million. It is the best case, reached only at full load, and is what you compare with a hosted API's price.