Glossary: Serving LLMs at Scale
vLLM
Also written: vllm serve
The open-source GPU engine that introduced PagedAttention and is the most widely used way to serve open models on GPUs. It serves an OpenAI-compatible API on port 8000.
Glossary: Serving LLMs at Scale
Also written: vllm serve
The open-source GPU engine that introduced PagedAttention and is the most widely used way to serve open models on GPUs. It serves an OpenAI-compatible API on port 8000.