Glossary

113 terms from Serving LLMs at Scale, in plain words, each with the chapter that teaches it.

Search the glossary

A

Acceptance rate: The share of a drafter's guessed tokens that the main model keeps during speculative decoding. A higher rate means a bigger speedup.
Activations: The numbers that flow between a model's layers while it handles a request. They take working memory, and some quantization schemes shrink them too.
Affinity: Sending the same user or conversation to the same replica so that replica's prefix cache stays warm. The router falls back to the least busy replica if the home one is overloaded.
Alert rule: A Prometheus rule that fires when an expression stays true for a set `for` time, such as p95 TTFT above 2 seconds for 5 minutes. Keep the list short so people don't learn to ignore alerts.
Alertmanager: A Prometheus component that sends alerts to people, for example to Alex's phone.
Arithmetic intensity: The amount of calculation done per byte read from memory. It decides whether a step is limited by memory or by compute.
asyncio: Python's built-in way to wait on many network requests at once. An async function pauses at each await while others run, which is how loadtest.py sends many requests.
Autoscaling: Adding or removing instances automatically. Scale on what users feel, such as queue length or p95 TTFT, never on CPU use, because a busy GPU is the goal.
AWQ (activation-aware weight quantization): A 4-bit weight quantization method that protects the few weights that matter most, judged by the activations flowing through them. It needs a calibration set.

B

Batching: Processing several requests in the same model step. A decode step then reads the weights once and produces a token for every request in the batch.
BF16: A 16-bit number format that models are commonly trained in. It is the baseline against which FP8 is compared.
Block: A small fixed-size piece of KV cache memory (16 tokens by default in vLLM). A request's block table lists which blocks, anywhere in memory, belong to it.

C

Cache-ram: llama-server options that keep a prompt cache in RAM, which a slot can restore from. Idle slots can be saved into it.
Calibration set: A small sample of text used by methods like AWQ and GPTQ to guide quantization. FP8 'dynamic' schemes need none.
Checkpoint: A published set of model weight files. Model makers and groups such as Red Hat AI publish quantized checkpoints.
Chunked prefill: Splitting a long prompt's prefill into pieces mixed in between decode steps, so one huge document doesn't freeze everyone else's streaming.
Closed loop: Two ways to apply load in a test. A closed loop has N simulated users each send a request, wait and send again. An open loop sends requests at a fixed rate whether or not earlier ones finished, like real customers, and reveals queues.
Cloud Run: Google Cloud's service for running containers. It can attach an NVIDIA GPU to an instance and still scale to zero.
Cold start: The time from starting a model server until it can answer. It includes getting a machine, pulling the image, fetching the weights, loading them and warming up, so it is much longer than a web app's.
Compute-bound: Limited by how fast the processor can do arithmetic rather than by memory. Prefill is compute-bound because each weight read is reused for every prompt token.
Concurrency: The number of requests in progress at the same moment. It is lower than the number of open chats, because most customers are reading or typing.
Concurrency sweep: A load test that repeats a closed-loop run at rising concurrency (1, 2, 4, 8, 16 users) to see where throughput flattens and latency climbs.
Consistent hashing: A way of assigning users to replicas so that adding a replica moves only a few users, keeping most caches warm.
Container: A packaged, isolated copy of a program with its dependencies. Here the engine goes in an image and the weights come in through a volume.
Context size: The number of tokens of conversation (prompt plus answer) a server can hold. In llama-server, -c sets the total KV cache shared by the slots; in vLLM, --max-model-len caps each request.
Continuous batching: Re-forming the batch at every decode step so a finished request is replaced at once by a waiting one. Static batching instead runs a group until all of its requests finish, leaving slots idle.
Cost per million tokens: A GPU's hourly price divided by the tokens it produces per hour, times a million. It is the best case, reached only at full load, and is what you compare with a hosted API's price.
Counter: Two kinds of metric. A counter only goes up (tokens produced so far); a gauge goes up and down (requests running now).
CUDA graphs: Recorded GPU work that is replayed with little overhead. Engines use them automatically to speed up decode steps.
CUDA out of memory: An error from running out of GPU memory while serving. Lower --max-num-seqs, --max-num-batched-tokens or --gpu-memory-utilization.

D

Decode: The phase where the model writes the answer one token per step, each step reading the whole KV cache. It is memory-bound and sets inter-token latency.
Deployment: Kubernetes objects. A Deployment keeps N copies of a container running; each copy is a pod; a Service gives the pods one address and spreads requests over the ready ones.
Disaggregated serving: Running prefill and decode on separate GPUs. It is meant for large fleets.
Drafter: The fast guesser in speculative decoding that proposes the next few tokens for the real model to check. It can be a small model, n-gram lookup or extra prediction heads.
Drain: Stopping traffic to a replica before shutting it down, so in-progress requests aren't broken. The failure drill showed why this matters in production.

E

EAGLE / MTP: Extra small layers trained to predict several tokens ahead, used as a drafter. Some models, including Qwen3.5, ship with them.
Environment variable: A named setting read by a program from its surroundings. Writing NAME=value before a command sets it for that command only.

F

FastAPI: A Python web framework used to build router.py; uvicorn is the server that runs it.
Fixed state: A fixed-size memory each conversation needs in models like Qwen3.5, whose non-attention layers don't grow with length. It doesn't shrink when the KV cache is quantized.
FLOP: One arithmetic operation on decimal numbers; a teraFLOPS is a trillion per second. It measures a GPU's computing power.
FP8: An 8-bit floating-point number format. It is the default quantization choice on modern NVIDIA GPUs and nearly free in quality; INT8 is the 8-bit integer alternative for older GPUs, and NVFP4 is a 4-bit format for Blackwell.

G

Gateway API Inference Extension: Kubernetes and vLLM-project tools that add model-aware routing (queue length, KV cache use, prefix affinity) in production.
GGUF: The model file format used by llama.cpp.
Goodput: Requests per second that meet the SLO. A server answering 10 a second but making half wait 20 seconds has a goodput of 5.
GPTQ: A 4-bit weight quantization method that quantizes one layer at a time, adjusting the remaining weights to cancel each rounding error.
GQA (grouped-query attention): A design where several query heads share one key-value head, so the KV cache is smaller. Qwen3-8B has 32 attention heads but stores only 8 KV heads.
Grafana: A tool that usually draws dashboards from Prometheus data.
GuideLLM: A load-testing tool from the vLLM project that runs sweeps automatically and finds the highest request rate meeting your targets.

H

Hash: A fingerprint computed from data. vLLM hashes each full block of tokens with everything before it to find reusable cached blocks; the router hashes a user name to pick a home replica.
Health check: An endpoint that returns ready or loading, called by load balancers, routers and Kubernetes to decide where to send traffic. llama-server's needs no API key.
Histogram: A metric that counts values in buckets (under 0.1 s, under 0.25 s...). PromQL can estimate percentiles from the buckets.
Host addresses (127.0.0.1, 0.0.0.0): 127.0.0.1 means this machine only, which is where llama-server listens by default. 0.0.0.0 means all network interfaces, so anyone who can reach the port can use the server; add an API key when you use it.
HTTP status codes: Numbers servers return with a reply: 200 OK; 400 request too big (exceeds context); 401 missing or invalid API key; 429 Too Many Requests ('you, slow down', a user over their limit); 503 Service Unavailable ('we're full or loading, try again soon').
Hugging Face: The site where open models are downloaded. HF_TOKEN is an access token needed only for gated models.

I

Inter-token latency (ITL): The gap between a request's later tokens, roughly one decode step (loadtest.py averages it per request). It is the speed the user feels while reading the stream.

K

KEDA: A Kubernetes tool that scales deployments on a metric such as a Prometheus value.
Kernel: A small program that runs on the GPU. Engines choose fast attention kernels such as FlashAttention automatically.
Key vLLM flags: --gpu-memory-utilization is the share of GPU memory vLLM may use (default 0.92); --max-model-len is the longest prompt plus answer (lower it to free KV memory); --max-num-seqs is the most requests decoded at once (lower it to protect per-user speed).
Knee: The point in a concurrency sweep where latency shoots up because requests start queueing. Run below it.
Kubernetes: A system that runs containers across a cluster and keeps them running, driven by YAML descriptions. A GPU is requested like memory, as nvidia.com/gpu.
KV cache: The stored keys and values for every token already processed in a conversation. Weights are shared but each conversation needs its own KV cache, so memory usually limits how many users fit on a GPU.
KV space: The GPU memory left for the KV cache: GPU memory times the utilization cap, minus the weights and about 1 to 3 GB of overhead. Dividing it by the memory per conversation gives how many users fit.

L

Latency: How long one user waits: the time to first token, then the speed of later tokens. It trades against throughput.
Least outstanding requests: A routing rule that sends each request to the replica with the fewest requests in progress, rather than simply the next in line.
Little's law: Requests in flight equal arrivals per second times seconds per request. It turns daily traffic into real concurrency at the busiest hour.
llama.cpp: An open-source engine that runs models on CPUs, Macs and small deployments. llama-server is its multi-user server; the other llama-* programs benchmark and evaluate.
llm-compressor: The vLLM project's tool for quantizing your own models. The older AutoAWQ library is deprecated.
Load balancer: A component that picks a replica for each request. A general one spreads requests evenly; an LLM-aware router also uses load and cache affinity.
Load test: Sending realistic traffic to a server and recording what each request experienced.

M

Memory-bound: Limited by how fast the processor can read memory. Decode is memory-bound because each token reads every weight once for very little arithmetic.
MLX: Apple's machine-learning framework for Macs.

N

N+1: Enough capacity for peak load plus one spare, so the service survives a failure or an update.
N-gram speculation: A drafting method that copies the text that followed the same few tokens earlier in the prompt or answer. It helps only when answers repeat their input word for word.
NVLink: A fast link between GPUs in one machine, needed for tensor parallelism because GPUs exchange data at every layer.

O

OpenAI-compatible API: OpenAI's request format, which most engines copy, so the openai Python package works against llama-server or vLLM unchanged.

P

PagedAttention: vLLM's way of splitting the KV cache into small blocks handed out as a request grows, cutting waste from most of the memory to under 1% and letting blocks be shared.
Parallelism (data, pipeline, tensor): Ways to use several GPUs. Data parallelism runs whole copies of the model; pipeline parallelism gives each GPU a block of layers; tensor parallelism splits every layer across GPUs in one machine and needs a fast link like NVLink.
Percentile: The value that a given share of results beat: p50 is the median, p95 the value 95% of requests beat. Averages hide the customers who suffer, so targets are set on p95.
Perplexity: A measure of how surprised a model is by real text; lower is better. It is a quick way to compare two versions of the same model, such as 8-bit against 4-bit.
Preemption: vLLM pausing a request, freeing its KV cache and recomputing it later when memory runs out. The user sees a stall.
Prefill: The phase where the model reads the whole prompt in parallel and stores keys and values in the KV cache, ending with the first output token. It is compute-bound and mostly sets TTFT.
Prefix caching: Keeping the KV cache of a prompt's beginning and reusing it for any new request that starts with the same tokens, skipping their prefill. Put what never changes first and what changes last.
Probes: Kubernetes checks on a container. The startup probe allows time for loading, the readiness probe keeps traffic away until the server is ready, and the liveness probe restarts a pod that stops answering.
Prometheus: The standard open-source monitoring system. It scrapes (fetches) each server's /metrics page every few seconds, stores time series and evaluates alert rules.
PromQL: Prometheus's query language. For example, rate(x[5m]) turns a counter into a per-second rate.

Q

Quantization: Storing a model's numbers in fewer bits to save memory and speed up decoding. On a server it frees memory for more users.

R

RadixAttention: SGLang's way of keeping cached prefixes in a tree for reuse.
Replica: One complete copy of the model server. Several replicas give more throughput and survive one failing or being upgraded.
Retry-After: An HTTP header giving the seconds a client should wait before trying again. Well-behaved clients respect it.
Round-robin: A simple load-balancing rule that sends each request to the next replica in turn. It does poorly when request lengths vary a lot.
Router: A load balancer that understands LLM traffic, using least outstanding requests, affinity and health checks plus per-user limits. Kettle & Crane's router.py is a small one.

S

Safetensors: The standard weights file format on Hugging Face, used by vLLM and SGLang.
Scale to zero: Running no instances when idle so nothing is billed, at the cost of a cold start for the first customer afterwards.
Serving: Answering many people over a network at the same time, all day, optimizing both everyone's wait and the cost per answer.
SGLang: An open-source GPU serving engine from the LMSYS group, known for RadixAttention and fast structured output.
Shadow traffic: Copies of real questions sent to a new system while customers still see only the old system's answers, so the two can be compared before switching.
Sliding window: A limit counted over the last N seconds, such as at most 20 requests in the last 60 seconds for each user.
SLO (service-level objective): A latency target, such as p95 TTFT under 2 seconds. Goodput counts the requests that meet it.
Slot: A place in llama-server where one request is worked on, set with -np. All busy slots are decoded together, and extra requests wait in a queue.
Speculative decoding: A drafter guesses the next few tokens and the real model checks them all in one step, keeping those up to the first disagreement plus one of its own. Speed changes but output doesn't.
Streaming: Sending an answer in pieces as it is written, instead of all at once. It is how clients measure TTFT and inter-token latency.

T

TensorRT-LLM: NVIDIA's serving engine: fast, but with more setup.
TGI (Text Generation Inference): Hugging Face's engine for serving LLMs, now in maintenance mode, with its documentation recommending vLLM, SGLang, llama.cpp or MLX instead. It is not a good choice for starting new projects.
Thread: A separate line of work in the same program. A thread pool runs a function several times in parallel.
Throughput: The whole server's work: total tokens or requests per second across all users. Batching raises it a lot and each request's latency a little.
Time to first token (TTFT): How long a user waits until the first word of the answer appears. It is mostly prefill plus any queueing.
TLS: Encryption for web traffic, usually set up at the load balancer so keys and chats are encrypted in transit.
TPOT (time per output token): vLLM's name for the average time per output token, equivalent to inter-token latency.

U

Unified KV pool: One KV cache pool that any llama-server slot can fill up to the whole context, so long and short requests share memory flexibly. It is the default when -np isn't set.

V

vLLM: The open-source GPU engine that introduced PagedAttention and is the most widely used way to serve open models on GPUs. It serves an OpenAI-compatible API on port 8000.
vllm bench serve: vLLM's built-in load-testing command, which works against any OpenAI-compatible server. --request-rate gives an open loop and --goodput takes SLOs in milliseconds.
Volume: A folder mounted into a container, used to bring in model weights or keep downloaded ones between restarts.

W

W4A16 / W8A8: Quantization scheme names: W is bits for weights and A is bits for activations. W4A16 keeps 4-bit weights and 16-bit activations; W8A8 makes both 8-bit.