Glossary: Serving LLMs at Scale
Closed loop
Also written: open loop
Two ways to apply load in a test. A closed loop has N simulated users each send a request, wait and send again. An open loop sends requests at a fixed rate whether or not earlier ones finished, like real customers, and reveals queues.