Glossary: Serving LLMs at Scale
vllm bench serve
vLLM's built-in load-testing command, which works against any OpenAI-compatible server. --request-rate gives an open loop and --goodput takes SLOs in milliseconds.
Glossary: Serving LLMs at Scale
vLLM's built-in load-testing command, which works against any OpenAI-compatible server. --request-rate gives an open loop and --goodput takes SLOs in milliseconds.