Glossary: Serving LLMs at Scale

Sliding window

A limit counted over the last N seconds, such as at most 20 requests in the last 60 seconds for each user.