Glossary: Serving LLMs at Scale
CUDA graphs
Recorded GPU work that is replayed with little overhead. Engines use them automatically to speed up decode steps.
Glossary: Serving LLMs at Scale
Recorded GPU work that is replayed with little overhead. Engines use them automatically to speed up decode steps.