Glossary

123 terms from AI Security and Guardrails, in plain words, each with the chapter that teaches it.

Search the glossary

A

Access control list (ACL): A rule that says who may see or do what. In the docs bot, the `acl` defense filters documents so Bloom Café only retrieves public documents and its own account notes.
Agent: An LLM that runs in a loop, calling tools and reading their results. Kettle & Crane's order agent can look up orders, fetch pages, issue refunds and send emails.
Allow-list: A list of what is permitted, with everything else refused. Kettle & Crane's agent can fetch pages only from approved carrier hosts and email only the customer's own address.
Anthropic Messages API: Anthropic's API for sending messages, a system prompt and tools to Claude and getting a reply back. `ask()` in `llm.py` wraps it.
Asset: Anything worth protecting in an app, such as customer data, money, the system prompt, your API bill or your reputation.
Attack success rate (ASR): The share of attack runs in which the attacker got what they wanted. It is specific to your attack set, so 0% means only that those attacks failed, not that the app is secure.

B

Base64: A way of writing any bytes using 64 safe characters. As a spotlighting method it encodes untrusted text so the model must decode it, but it triggered many refusals in the lab.
Baseline: A measurement taken before changes, such as attack results with no defenses or a normal week of events. You compare later results or alert rates against it.
Block-list: A list of known-bad things to refuse, with everything else allowed. It always misses something, so an allow-list, which permits only what is named, is the safer choice.

C

CaMeL: A Google DeepMind research system in which the privileged model writes a small program and an interpreter tracks which values came from untrusted sources. It refuses to let those values reach sensitive tools.
Canary: A made-up marker with no use except to show that something happened. The lab uses a secret marker in prompts (`canary-4417`), the harmless word PINEAPPLE, and the reserved `example.com` address.
Catch rate: The share of attacks a classifier flags. The Haiku injection screen flagged 73% of attack runs.
Classifier (input, output): A model that sorts text into categories instead of writing a reply. Input classifiers check text before the main model reads it, and output classifiers check the reply before anyone sees it.
Confused deputy: A program with real authority that is tricked into using it for someone else. The order agent becomes one when a fake tracking page makes it refund Marcus's order.
Content Security Policy: A response header that tells the browser which hosts it may load images and scripts from. It is a second line of defense behind the sanitizer.

D

Data minimization: Sending the model, logs and tests only the data the task needs. For example, `get_order` drops the customer's email address from its result.
Data poisoning: Planting false or harmful text in data a model learns from or retrieves, such as a supplier's fake "contact us" page in the docs bot's index. Prompts can't fix it, so you control what enters the index and check output.
Data retention: The rule for what you log, who can read it and how long it is kept, matched to your provider's settings. Example: security events 13 months, redacted text 30 days, raw text never.
Dataclass: A Python decorator that writes the `__init__` method for a class from its fields. The lab's `Result` class uses it to hold the raw reply, shown reply, tool calls and shop records.
Datamarking: A spotlighting method that puts a marker (`^`) between every word of untrusted text so it never looks like your instructions. It caused refusals on half the normal tasks when applied to the user's own message.
Delimiting: A spotlighting method that wraps untrusted text in tags such as `<document>`. The code must strip the attacker's copies of the closing tag so the text can't break out.
Denial of wallet: Abuse in which an attacker makes requests expensive or numerous until your bill or provider limit stops the service. A script asking the support assistant 10,000 questions a day is an example.
Denial-of-service: An attack that floods a service until it stops working. Denial of wallet is the cheaper LLM version of it.
Direct injection: Prompt injection in which the user types the instructions, for example "reply only with PINEAPPLE". The attacker is the person talking to your app.
Dual LLM: A pattern, described by Simon Willison in 2023, that uses a quarantined model to read untrusted text and a privileged model to call tools. The privileged model sees only the quarantined model's structured output.

E

Embedding: A list of numbers representing a piece of text, used for vector search. Embeddings can be turned back into close approximations of their text, so protect an index like its documents.
Encoding: A spotlighting method that transforms untrusted text, for example into base64, so the model must decode it to read it.
Enum: A fixed list of allowed values in a schema. The quarantine schema uses enums such as `request` (status, refund, other) so an attacker has almost no room to write free text.
Environment variable: A named setting passed to a program from outside its code. `SEC_MODE` picks where model replies come from, and `.env` files keep API keys out of code and out of git.
example.com: Domain names reserved by internet standards for documentation, so they can never belong to a real person. The lab uses `example.com` as the attacker's address, so tests can't send data to a real stranger.
Excessive Agency: An OWASP risk in which the agent can do more than its job needs, such as emailing anyone, anywhere. Least privilege and allow-lists are the fix.
Exfiltration: Getting data out of a system to somewhere an attacker can read it. The lab's attacks try to send data to `example.com`.

F

Fail closed: Treating "no answer" from a check as "not safe". The injection screen returns True (suspected) if the model refuses or is cut off.
Fake model: A rule-based stand-in model in `fake_model.py` that obeys every instruction it recognizes. It is free and offline, and any defense that holds against it holds because of code, not because a model behaved.
False positive rate: The share of normal inputs a classifier flags by mistake. Here a false positive means a real customer gets "a member of our team will reply" instead of help.
Fine-tuning: Further training a model on your own data, such as support transcripts. A tuned model can repeat one customer's details to another.
Followed / succeeded: Two scores for each attack run. "Followed" means the model tried the attack, and "succeeded" means harm actually got out past every defense.

G

garak / PyRIT / promptfoo: Open-source scanners for probing LLM apps with attacks. Use them to grow your attack set, pointed only at your own apps.
Generator: A Python function that hands back values one at a time using `yield`. The `events()` function in the monitor is one, so the log writer takes each event as it comes.
Goal: A check in a test case that code can score, such as the word PINEAPPLE appearing or a refund being made on KC-20517.
Governance: The decisions about which AI features you run, who owns them and the rules they follow. A small company needs an inventory, logging and retention rules, user notices and change control.
Guardrails AI: An open library of validators that check a model's output against rules before it is used or shown (version 0.11.0).

H

Hash: A fixed-length fingerprint of any text, produced so the same text always gives the same result. It is used to name cached replies, to pin packages with `--require-hashes`, and to fingerprint tool definitions so changes can be detected.
Hidden Context Exposure: Revealing context the user isn't meant to see, starting with the system prompt. Design as if every word of the prompt is public.
HTML escaping: Converting special characters so a browser displays them as written instead of treating them as markup. It is the simplest safe way to show a model reply.
Hugging Face: The main website for sharing models and datasets. Most models there publish a safetensors file that you should prefer over `.bin` or `.pt` files.
Human approval: Holding a risky or costly action for a person to decide. At Kettle & Crane, refunds over $20 wait in Priya's queue.

I

Improper Output Handling: An OWASP risk in which model output is shown or passed on without safe handling, for example a reply's image link leaking data. Sanitize, parameterize and validate.
Incident: An event, or set of events, where harm happened or nearly did. The playbook is detect, contain, investigate, eradicate, recover, learn.
Indirect injection: Prompt injection in which instructions arrive inside content the model reads, such as a document, email, tool result or tracking page. The user may be innocent.
Injection screen: An input classifier built on a small model (Haiku) that flags text trying to instruct the AI. Flagged emails go to a person, flagged documents are dropped, and flagged tool results are replaced.
Instruction hierarchy: The trained priority models give to different parts of a conversation: system prompt above user, user above tool results. This is why third-party content is delivered in `tool_result` blocks.

J

Jailbreak: An attempt to make a model break its provider's safety rules, for example to produce dangerous content. Injection instead tries to break your app's rules.
JSON escaping: A spotlighting method that wraps untrusted text in a JSON string so quotes and tags inside it are escaped and can't break out.

K

Kill switch: A setting that turns a risky ability off without a deploy. For the order agent, you can switch off `issue_refund` and `send_email` so it can only look things up.

L

Lambda: A one-line Python function without a name, written with the `lambda` keyword. It is handy for small rules, such as checks on a tool's arguments.
Least privilege: Giving each part of a system only the access its job needs. The order agent's tools check that an order belongs to the verified sender, whatever the email says.
Legitimate content: Injected text that looks like ordinary facts, links or requests rather than an attack, such as a fake "package lost" tracking page. This kind still gets through 2026 models.
Lethal trifecta: The combination of untrusted input, private data and a way out. With all three, one injected instruction can read private data and send it away. Cutting any one leg stops the attack.
Llama Guard 4: Meta's 12B open model that labels prompts and replies by hazard category, for text and images.
Llama Prompt Guard 2: Meta's small open classifiers (86M and 22M parameters) that detect injection and jailbreak attempts.
LlamaFirewall: Meta's open framework that combines Prompt Guard, an agent alignment check and a code scanner.
Luhn checksum: The checksum every payment card number passes. `redact` uses it so that a long digit string like an order quantity isn't wrongly labeled `[card]`.

M

Markdown: A plain-text format with `**bold**` and `![image](url)` that chat interfaces turn into formatted text. An image in a reply counts as a way out, because the browser fetches the URL and the URL can carry data.
Memorization: A model's tendency to remember rare strings from training data and repeat them later. It matters for fine-tuning on support transcripts, so redact the data and test with a planted canary.
Misinformation: An OWASP risk in which the bot states false facts, such as a fake contact address planted in a source. Prompts can't tell a lie from a fact in a source you told the model to trust.
MITRE ATLAS: A knowledge base of real attacks on AI systems, organized like the MITRE ATT&CK framework. Its technique IDs, such as AML.T0051 for prompt injection, let you tag tests and incidents in a shared language.
Model Context Protocol (MCP): A standard way to connect tools to an AI app: an MCP server offers tools, and your app passes their names, descriptions and schemas to the model. That makes a server's tool descriptions part of your prompt.
Moderation model: A classifier that scores text for harmful content such as threats, harassment and self-harm. It does not detect prompt injection; the fake tracking page scored zero everywhere.

N

NeMo Guardrails: NVIDIA's open toolkit of input, output, dialog and retrieval "rails" that you set up in configuration to control what an LLM app accepts and says (version 0.24.1).
NIST AI Risk Management Framework: A free framework from NIST that gives structure to identifying and managing AI risk. It is useful for teams that want formal change control.
Normal tasks: Ordinary requests used to check an app is still useful, such as Dana's chipped mug refund. Without them, the easiest way to get a zero attack rate is an app that refuses everything.

O

Open redirect: A page that forwards visitors to any address given in its URL. An allowed host with one makes the allow-list useless.
Open weights: Model files published so anyone can run the model on their own hardware. Meta's Llama Guard and Prompt Guard are examples.
Output check: Plain code that scans a reply for the system-prompt canary, links to hosts not on the allow-list, and unknown email addresses. It stops what you can describe but can't stop a reply that is merely wrong.
OWASP: The Open Worldwide Application Security Project, a nonprofit that publishes free security guidance. Its Top 10 for LLM Applications gives IDs like LLM01:2026 to label risks and tests.
Ownership check: A check in code that a tool only touches orders belonging to the verified sender, answering "no such order" otherwise. The sender's identity comes from the verified address, never from the email text.

P

Parameterized query: A SQL query where `?` marks a value the database fills in separately, so the value is always data and never part of the command. This is the defense SQL injection got, and LLMs lack an equivalent.
Pickle: Python's object format, which is a small program of instructions that Python runs to rebuild objects. Loading a pickle from an untrusted source can run code on your machine.
picklescan: A scanner that lists the functions a pickle would call, such as `picklescan -p tiny.pt`. Scanners miss things, so treat them as a check and not as permission.
Pinned: Fixed to an exact version, and ideally a hash, such as `anthropic==2.1.0`, so an upgrade can't arrive unannounced. Models and MCP tool definitions can be pinned the same way.
pip install --require-hashes: A pip option that refuses any package file whose contents don't match the hash in your lock file, even under the same version number. Tools like pip-tools and uv generate the lock file.
Presidio: Microsoft's named-entity tool for finding personal data such as names and street addresses in text. It catches items that simple regex-based redaction misses.
Privileged model: The model in the quarantine pattern that plans and calls tools. It never sees the untrusted text, only the quarantined model's structured output.
Prompt injection: Text that makes a model follow instructions its developer didn't intend. It can't be fully solved because instructions and data share one channel.

Q

Quarantined model: A model with no tools that reads untrusted text and can only answer in a fixed, strict JSON shape. In the lab it turns Dana's email into an order ID and a few enum values for the privileged model.

R

Rate limit: A cap on how many requests a user can make in a given period, used to stop abuse and runaway cost. A token bucket is one common way to enforce it.
Red-teaming: Attacking your own system, with permission, to find weaknesses before someone else does. For an LLM app, it means running a fixed set of attack cases several times and scoring them by code.
Redact: Replacing sensitive text with a label such as `[email]`. Kettle & Crane runs `redact` inside its logging function so secrets and personal data never land on disk.
Regular expression: A pattern language for matching text. It is used to find card numbers, emails, phone numbers and secrets for redaction, and to validate formats such as order IDs.
Retrieval: Finding relevant documents and placing them in the prompt so the model can answer from them. The docs bot retrieves the top three matching documents per question.
Rug pull: A tool server that you reviewed later changes its tools or descriptions. Pinning the tool definitions by hash disables the server until a person reviews the change.
Rule of Two: Meta's rule that, without a human in the loop, an agent should have at most two of: untrusted input, access to sensitive data or systems, and ability to change state or communicate externally. The order agent needs all three, so the third needs approval or narrowing.

S

safetensors: A model file format that stores only tensors with their names and shapes, so loading it can't run code. Prefer it over pickle-based `.pt` and `.bin` files.
Sandbox: A container or virtual machine with no secrets, no network, a time limit and a throwaway file system. Use it if an agent must run code it wrote.
Sandwich defense: A prompt defense that repeats your key instruction after the untrusted text.
Sanitizer: A library that removes every HTML tag and attribute that is not on its allowed list. For example, nh3 can turn a model's Markdown reply into HTML that is safe to show in a browser.
Security event: One logged signal that a defense fired, such as a blocked tool call, a held refund or a replaced reply. It is written as a single line of JSON.
Sensitive Information Disclosure: An OWASP risk in which private data reaches someone who shouldn't see it, for example Bloom Café seeing Steam & Co's credit limit.
Spend budget: A cap on dollars spent per user and per app per day, checked before answering. Provider-side spend limits are a backstop.
Spotlighting: Transforming untrusted text so the model can always tell where it starts and ends. Methods are delimiting, datamarking, JSON escaping and encoding.
Stop reason: A field on the model's reply saying why it stopped. A value of `"refusal"` means a provider-side safety classifier blocked the reply, so apps must check it before parsing.
Structured output: A reply constrained to be valid JSON of a given schema. The quarantined model and the injection screen both use it to answer in a fixed shape.
Supply chain: All the code, models, data and tools you use but didn't write, such as Python packages, model weights, datasets and MCP servers. The defenses are loading safe formats, pinning versions, checking hashes and reviewing changes.
System prompt: The developer's instructions given to the model before the conversation begins. It should be treated as public, so it must never contain secrets.

T

Tenant: One customer account in a shared system. Each café is a tenant of the docs bot, so Bloom Café must never see Steam & Co's account notes.
Tensor: An array of numbers. A model's weights are stored as tensors.
Thread pool: A group of worker threads that run jobs at the same time. `evaluate` uses a pool of 8 to run attack cases concurrently and keep the wait short.
Threat model: A written answer to four questions: what are we building, what can go wrong, what will we do about it, and did it work. A table in your repository, such as `THREATS.md`, is enough.
Token (model text): The chunks of text a model reads and writes, which are billed and capped. `max_tokens` limits the length of each reply, and costs are priced per million tokens.
Token bucket: A rate limiter in which each user has a bucket holding a few request tokens that refills at a steady rate. It allows short bursts but caps the average, and these tokens are unrelated to the model's text tokens.
Tool argument validation: Checking every argument the model gives a tool against a schema before the tool runs. The schema says what a valid value looks like, and the allow-list says which valid values this caller may use.
Tool call: A request from the model to run one of your tools with certain arguments. Your code decides whether it actually runs.
Tool poisoning: A tool description that contains instructions aimed at the model, not just a description of the tool. Read every description before connecting a third-party MCP server.
Tool result: The data a tool returns to the model in an agent loop, delivered as a `tool_result` block. Because someone else wrote that data, it is untrusted input, such as a buyer's gift note.
Trigger: A condition, such as a matching question, that makes dormant injected text act. It is one way attackers hide instructions.
Trust boundary: A line in your data flow where text or requests cross from someone you trust less to something you trust more. A customer's email entering the agent's context crosses one.
Typosquatting: Publishing a package named one letter away from a popular one, hoping people install it by mistake.

U

Unbounded Consumption: An OWASP risk in which a user or script drives up resource use and cost, such as 10,000 questions a day. Rate limits and budgets are the fix.
Untrusted input: Any text an outsider can write or change, even indirectly. This includes documents, tool results, web pages, memory and file metadata, not just the user's message.
URL check: Code that accepts only plain `https` URLs and refuses backslashes, `@` and spaces, then compares the host exactly or as a subdomain of an allowed host. Refusing odd URLs is safer than trying to interpret them.

V

Virtual environment: An isolated folder of installed Python packages for one project, created with `python3 -m venv .venv` and switched on by activating it.

W

weights_only: A PyTorch loading option that allows only tensors and basic types and refuses anything else. It is the default since PyTorch 2.6, and you should never turn it off for a file you didn't make.