Glossary

98 terms from AI System Design, in plain words, each with the chapter that teaches it.

Search the glossary

A

ACL (access list): A list of which users or roles may read a document. In staff document Q&A each chunk carries its document's ACL, and search filters by it so shop staff never get contract text.
Agent: Code where the model chooses its own next step and tools in a loop. It is flexible but slower, costlier and harder to test than a workflow.
Allow-list: The list of exact model IDs each product may use, kept in the gateway so model upgrades happen in one place.
Anti-pattern: A design that looks reasonable but reliably causes trouble, such as building an agent first or leaving out an estimate.
Approach ladder: The six ways to build an AI feature, from simplest to most work: prompt only, stuffed context, RAG, workflow, agent, fine-tuning. Start at the bottom and climb only when a target forces you to.
Attach rate: The share of customers who buy an add-on. Ask Counter assumes 30% of Crane Counter cafés buy it, giving 360 tenants.
Availability: The share of time a system works. For a chain of required parts it is the product of the parts' availabilities.

B

Back-of-envelope estimate: A quick calculation, accurate to within about a factor of two, that tells you which design you need, for example 40 requests a day versus 40 a second.
Backfill: Re-running a batch job over all previously kept items, for example re-tagging two years of reviews after the taxonomy changes. It is limited by the provider's batch size and in-progress limits.
Batch API: A provider service that takes a file of requests and processes them within 24 hours at half price. It suits work nobody waits for, like Kettle & Crane's nightly tagging.
Break-even: The volume at which two options cost the same, such as a hosted API versus your own GPUs.
Burst factor: A multiplier for how much busier the busiest minutes are than the average minute within the busiest hour. It is used to turn daily traffic into peak requests per second, and 2 is a safe starting value.

C

Cached tokens: Input tokens read from the provider's prompt cache, at a tenth of the input price (a twentieth for Sonnet 5.5). Storing text in the cache the first time, a cache write, costs 1.25 times the input price.
Canary: Releasing a change to a small share of users first (5%, then 25%, then all) while watching error rate, cost and feedback.
CCPA (California Consumer Privacy Act): A California privacy law that gives residents rights over their personal data, including the right to have a business delete it on request. Systems that serve California customers must be able to find and delete one customer's records in every store.
CI (continuous integration): Automatic tests that run on every code change. Running a data-inventory check there means a new data flow cannot be added without being reviewed against the privacy rules.
Circuit breaker: A switch that stops calling a failing provider for a while so requests go straight to the backup.

D

Data inventory: A list of data flows and their fields kept in code, so rules can check automatically every time the design changes.
Data minimization: Sending and keeping only the data each step needs, for example giving the model a first name instead of a full address.
Data residency: Keeping data in a given country or region. It can rule out providers and models, so check it early; Anthropic's inference_geo setting keeps processing in the US at 1.1 times the price.
Data-flow diagram: A diagram of every place data is stored or processed and every arrow it travels along, labeled with which fields travel.
Deep dive: A closer look at one of the riskiest or costliest parts of a design, such as how retrieval filters by permission.
Denial of wallet: Abuse of a public endpoint to run up your model bill. Per-user and per-IP rate limits and daily spend caps defend against it.
Design doc: The eight-step design written on one page: targets, estimate, diagram, decisions and plans, with a named decider.
Design interview: A timed session where you design a system out loud, judged on how you reason and explain trade-offs more than on the final architecture.
Design review: A meeting where colleagues test a written design before it is built, when changes are cheap, ending in decisions by a named decider.

E

Embedding: A list of numbers that captures a text's meaning, stored for every chunk so a search can find similar text.
Error budget: The downtime an SLO allows each month, which you can spend on outages and risky releases. 99.9% allows 43.2 minutes.
Escalation: Sending a hard request to a bigger, costlier model (or a person) by rule. In the support assistant, 10% of conversations escalate to Sonnet 5.5, which makes the escalation rate the biggest cost lever.
Exact response cache: A cache that returns a whole stored answer for identical input. Safe only when the answer doesn't depend on the user or the time, like Priya's reviewed answers to common first questions.
Extract, then query: Turning documents into table rows once with a model, then answering questions with database queries. Jo's contract questions work this way.

F

Fair queue: A queue that takes turns between tenants so one heavy café can't slow the others.
Faithfulness: A quality measure meaning every claim in an answer is supported by the chunks it cites.
Fallback: Switching to another provider or model when the first fails, as the gateway does from Claude to gpt-6-luna.
Fine-tuning: Training a model further on your own examples. It pays only for a narrow task at high volume where a prompt can't get the style or format right.
float32, float16, int8, binary: Ways to store each number of a vector in 4 bytes, 2 bytes, 1 byte or 1 bit (binary keeps only the sign). Smaller formats save memory but lose some recall.
Fresh tokens: Input tokens paid at the full input price, as opposed to cached tokens.
Freshness: How quickly a change, such as a new document or return policy, must show up in answers.
Functional requirement: A statement of what the system does, written as a job for a named user, plus what it must never do.

G

Graceful degradation: Giving a worse answer instead of none. The support assistant has a five-step ladder whose last two steps need no model.

H

HNSW: The graph-based index most vector stores use. It stores about 2 × M links per vector (M is often 16), which adds to memory.
Hybrid search: Keyword search and vector search combined, which beats either alone on product names, ticket numbers and codes.

I

Idempotent: An action is idempotent when doing it twice has the same effect as once. An idempotency key lets the order service ignore a repeat so a retry can't order twice.

J

Judged sample: A share of live calls (2% for support) graded each day by a stronger model, with an alert if the score drops.

L

Latency: How long a user waits, measured as time to first token and total time, set with percentiles rather than averages.
Latency budget: A latency target split into limits per step. P95s of steps don't add up, so check the total by simulation or measurement.
Least privilege: Giving tools only the access they need: read-only by default, scoped to the signed-in customer in code, with writes behind confirmation.
Lethal trifecta: Simon Willison's term for a system with private data, untrusted content and a way to send data out, which can be tricked into leaking. Design at least one part out.
Little's law: Requests in progress equal arrivals per second times seconds each takes. At 1.8 calls a second and 4 seconds each, about 7 calls run at once.
Load balancer: A service that spreads requests across several copies of a server so none is a single point of failure.
Lognormal: A skewed distribution where most values are small with a long tail of large ones. It is the usual shape for latencies and for usage across tenants.

M

MCP (Model Context Protocol): A protocol for giving models tools. Kettle & Crane shares one MCP server for orders and stock, and Ask Counter gets a separate tenant-scoped one.
Model gateway: The one module or service that talks to model providers, adding retries, fallbacks, caching, budgets per product and cost tracking.
Multi-tenant: One system serving many customers whose data must stay apart, like Ask Counter over hundreds of cafés.

N

N+1: Enough servers for the peak plus one spare, so one can fail or be updated without an outage.
Noisy neighbor: One tenant whose heavy load slows the others, solved with quotas and a fair queue.
Non-functional requirement: How well a system must work: latency, cost, quality, availability and privacy, each with a number and a way to measure it.

O

Offline evaluation (test set): A fixed set of questions with known good answers, graded by code and an LLM judge before every release; a score drop beyond the margin blocks the release.
OpenTelemetry: A standard, vendor-neutral toolkit for collecting traces, metrics and logs from software. It is commonly used to build observability for model calls, such as recording tokens, cost and timings per request.

P

p95: The value 95% of requests beat, so 1 in 20 is slower. Targets use percentiles because averages hide the slow requests users remember.
pgvector: A PostgreSQL extension for vector search, suited to up to about 10 million chunks in an existing database.
Prefill: The time a model spends reading the prompt before it writes. It lowers real GPU capacity below the output-speed figure.
Prompt cache: A provider feature that reuses the processed start of a prompt, such as the help center, at a small fraction of the input price. Changing anything early in the prompt breaks it.
Prompt injection: Text in untrusted content, such as a customer message or ticket, that tries to steer the model. Treat such content as data, never instructions.
Prototype: The smallest real version of a system that exercises an assumption. It is used to check estimates, such as cost and latency, with real calls instead of guesses.

Q

QPS (queries per second): Requests per second, the usual unit for load. It is computed at the busiest moment, not the average.
Queue and workers: A list of jobs nobody waits for and the processes that take jobs from it at a steady pace, making retries easy.
Quota: A per-user or per-tenant limit on usage, such as 25 questions a day, that protects margins and neighbors. Without it, a few heavy users can cost more than they pay.

R

RAG (retrieval-augmented generation): Searching an index and putting only the best-matching chunks in the prompt, used when knowledge is too big, changes often or needs permissions.
Rate limit: A provider's cap per account on requests (RPM), input tokens (ITPM) or output tokens (OTPM) per minute. Exceeding one returns a 429 error, and the limit is shared by every product on the account.
Recall: The share of relevant chunks a search finds. Smaller vector formats trade some recall for memory.
Recall@k: The share of questions whose right chunk appears in the top k search results, measured on a labeled set.
Retention: How long data, such as prompts at a provider or chat logs, is kept. Storage is cheap, so privacy promises (30 days for redacted chats) set the limit, not cost.
Rollout: Shipping a feature safely in stages, with a fallback and a way to turn it off in minutes. It is step 8 of the design method.
Router: The part of the gateway that picks a model per request: by job, by cascade (try small, escalate when unsure) or by customer.
Row-level security: A database rule that hides rows from the wrong tenant, so a bug in a tool's query isn't enough to leak another café's data. Every Ask Counter tool also adds WHERE tenant_id = ?, with the ID taken from the sign-in token, never the prompt.

S

S3: Amazon's object storage, used here to price log retention at $0.023 per GB-month.
SaaS (software as a service): One hosted system serving many paying customers, like Crane Counter for independent cafés.
Semantic cache: Reusing a stored answer for a similar past question, found by embedding lookup. It can return a confidently wrong answer to a near-miss question like a $40 versus $60 order.
Shadow test: A new model answers real traffic silently alongside the live one, so scores and cost can be compared before switching.
SLO (service level objective): A target such as 99.9% availability. What it allows is the error budget.
Streaming: Sending text as it is generated so users see the first words after the TTFT instead of waiting for the whole answer.
Stuffed context: Putting whole documents in the prompt, as the support assistant does with the help center. With caching it beats RAG below about 27,000 tokens.
Subprocessor: A third party that processes your customers' data for you; model providers become subprocessors that must be listed in the contract.
System design: Choosing a system's parts, how they connect and how data flows, so it meets its goals at an affordable cost.

T

Taxonomy: A fixed list of labels, such as the topic, product, sentiment and urgency tags used in nightly enrichment.
Tenant: One customer in a multi-tenant system whose data lives in the shared system but must be invisible to others. Each Crane Counter café is a tenant.
Throughput: How much work is done per second or per night; it matters most for batch jobs.
Token counting: A free Anthropic API call (messages.count_tokens) that counts a request's input tokens without running the model; it showed the 4-characters-per-token rule was 26% low.
Trace: The linked record of every call made for one request or conversation, tied together by a request ID.
Trade-off sentence: The four-part way to record a decision: the choice, the reason, what it costs, and when you would revisit it.
Trust boundary: A line where data passes to someone with different rules, such as from your systems to a model provider. Privacy problems usually sit on arrows that cross one.
TTFT (time to first token): How long until the first text appears in a streamed answer; the support assistant's target is p95 under 2 seconds.

U

Upsert: Insert a row, or update it if it already exists, so re-running a nightly job never duplicates tags.
Utilization: The share of capacity actually used. A rented GPU costs the same idle or busy, so low utilization makes self-hosting expensive.

V

Vector store: A system that holds the embedding index for search. Pick the simplest one that fits: in-memory for small indexes, pgvector for millions, a dedicated cluster beyond that.

W

Workflow: Code that calls the model at fixed steps. Preferred over an agent whenever the steps can be written down, as in the order workflow.

Z

ZDR (zero data retention): A provider agreement to keep no prompts or outputs, available by separate agreement for Anthropic.