Glossary: Serving LLMs at Scale

Inter-token latency (ITL)

Also written: ITL, inter-token latency

The gap between a request's later tokens, roughly one decode step (loadtest.py averages it per request). It is the speed the user feels while reading the stream.