Glossary: Serving LLMs at Scale
Speculative decoding
A drafter guesses the next few tokens and the real model checks them all in one step, keeping those up to the first disagreement plus one of its own. Speed changes but output doesn't.
Glossary: Serving LLMs at Scale
A drafter guesses the next few tokens and the real model checks them all in one step, keeping those up to the first disagreement plus one of its own. Speed changes but output doesn't.