Glossary: Serving LLMs at Scale
Gateway API Inference Extension
Also written: production-stack, llm-d
Kubernetes and vLLM-project tools that add model-aware routing (queue length, KV cache use, prefix affinity) in production.
Glossary: Serving LLMs at Scale
Also written: production-stack, llm-d
Kubernetes and vLLM-project tools that add model-aware routing (queue length, KV cache use, prefix affinity) in production.