Glossary: Serving LLMs at Scale

llama.cpp

Also written: llama-server, llama-bench, llama-batched-bench, llama-perplexity

An open-source engine that runs models on CPUs, Macs and small deployments. llama-server is its multi-user server; the other llama-* programs benchmark and evaluate.