Glossary: Serving LLMs at Scale
Drafter
Also written: draft model
The fast guesser in speculative decoding that proposes the next few tokens for the real model to check. It can be a small model, n-gram lookup or extra prediction heads.
Glossary: Serving LLMs at Scale
Also written: draft model
The fast guesser in speculative decoding that proposes the next few tokens for the real model to check. It can be a small model, n-gram lookup or extra prediction heads.