Glossary: Serving LLMs at Scale
Acceptance rate
The share of a drafter's guessed tokens that the main model keeps during speculative decoding. A higher rate means a bigger speedup.
Glossary: Serving LLMs at Scale
The share of a drafter's guessed tokens that the main model keeps during speculative decoding. A higher rate means a bigger speedup.