Glossary

96 terms from Transformers Basics, in plain words, each with the chapter that teaches it.

Search the glossary

A

AdamW: An optimizer that updates a model's weights during training. It is Adam with weight decay applied separately from the gradient step, which usually generalizes better.
Attention: A step where each token takes a weighted average of other tokens' value vectors, with weights set by how well its query matches their keys. It lets a token such as 'mug' pull in information from 'blue'.
Attention head: One of several smaller attention computations that run side by side in multi-head attention, each with its own queries, keys and values. Different heads can learn different jobs, such as looking at the previous character.
Attention weights: The softmax of the scaled query-key scores: for each token, positive numbers that add up to 1 and say how much it takes from each other token's value.

B

Base model: A pretrained model that only continues text and does not follow instructions. Asked to write a blurb for a mug, it may just continue with more requests.
Batch: A group of training examples processed together in a single training step. For a language model it is typically a stack of random text windows, such as 32 windows of 64 tokens.
BERT: An encoder-only transformer from 2018 trained by masked language modeling. It suits classification, tagging and search more than text generation.
Bigram model: A language model that predicts the next token using only the previous token. It can be built as a table counting which character follows which in a text, such as Kettle & Crane's blurbs and messages.
BPE (byte pair encoding): A tokenizer that starts from single characters (or bytes) and repeatedly merges the most frequent neighboring pair into a new token. Common words end up as one token and rare words are built from pieces.
Byte-level BPE: BPE that starts from the 256 possible byte values instead of characters, so any text in any language can be encoded without unknown characters. GPT-2 and Llama 3 use it.

C

Causal mask: A True/False table that stops each token from attending to later tokens, by setting those scores to minus infinity before softmax. Without it, the model would copy the answer it is supposed to predict.
Chat template: The format that turns a conversation into one text, using special tokens that mark where each speaker's turn starts and ends. Using the wrong template is a common reason a fine-tuned chat model performs badly.
Checkpoint: A saved file holding a model's weights, along with anything needed to use them, such as the list of characters the tokenizer knows. It lets training be resumed or the model be reloaded later for generation or fine-tuning.
Chinchilla: A 2022 DeepMind model and paper showing that, for a fixed compute budget, model size and training tokens should grow together, at about 20 tokens per parameter.
Compute-optimal: Training a model with the best balance of size and data for a fixed compute budget, about 20 tokens per parameter by Chinchilla's rule. Labs often train smaller models past this point because they are cheaper to serve.
Context length (context window): The most tokens a model can look at when predicting the next one. Your tiny GPT has a context of 64 characters and GPT-2 has 1,024 tokens.
Cosine schedule: A learning-rate schedule that starts high and decreases smoothly toward 0 along a cosine curve over the training steps. It allows fast early learning and fine adjustments near the end, and is often paired with AdamW.
Cross-attention: An extra attention layer in an encoder-decoder's decoder where queries come from the decoder but keys and values come from the encoder's output. It lets each output token ask which parts of the input it needs.
Cross-entropy loss: The average of minus the log of the probability the model gave to the true next token. Lower is better; blind guessing among 72 characters scores ln 72 = 4.277.

D

d_model (C): The size of the vectors that flow through the model, also called its width. In code it is often written C; GPT-3's is 12,288.
Decoder-only model: A transformer made of causal-attention blocks that predicts the next token, like GPT and Llama. Most general-purpose models today are decoder-only and handle other tasks through prompts.
DPO (direct preference optimization): A way to tune a model on human comparisons of answers directly, without a separate reward model or reinforcement learning. Llama 3's post-training uses SFT followed by DPO.

E

Embedding: A learned vector of numbers standing in for each token id, looked up from a table with one row per token. Tokens that behave alike end up with similar vectors.
Encoder-decoder: A transformer where an encoder reads the input with full attention and a decoder writes the output with causal attention, linked by cross-attention. T5 is an example, good for translation and summarization.
Encoder-only model: A transformer whose blocks use full attention, so every token sees text on both sides. It suits understanding tasks such as classifying and tagging, like BERT.

F

Fine-tuning: Continuing to train a pretrained model on a smaller, targeted dataset, usually with a lower learning rate. Fine-tuning on Jo's blurbs teaches the model her style.
FlashAttention: A fast GPU implementation of attention that never stores the full T × T table of weights, saving large amounts of memory on long sequences.
FLOPs (floating-point operations): Individual arithmetic operations such as multiplications, used as a measure of compute. Training costs about 6 × parameters × tokens FLOPs.

G

GELU: A smooth activation function used in GPT models. Like ReLU it passes large positive values and blocks large negative ones, but it bends smoothly around zero.
GPT: A family of decoder-only transformer language models from OpenAI that generate text by predicting the next token. GPT-2's architecture, scaled down to a tiny size, is a common model for learning how these systems work.
Gradient clipping: Scaling gradients down whenever their combined size exceeds a set limit, such as 1.0. It guards against occasional huge updates that can destabilize training.
Greedy decoding: Always choosing the single most likely next token. It gives the same output every time and often loops.
Grouped-query attention (GQA): A variant where groups of query heads share one set of keys and values. It uses much less memory when generating text.

H

Hallucination: Fluent, confident output that isn't true, such as the invented maker 'Stoneware Studio'. Models produce likely text, not true text.
Hidden state: The vector a recurrent network carries from step to step, summarizing everything it has read so far.
Hugging Face: A company whose Hub website hosts over a million pretrained models and whose free Python libraries, such as `transformers`, load and run them. `from_pretrained` downloads a model's settings, weights and tokenizer by name.

I

Inputs and targets: The training pair cut from text: a window of tokens is the input, and the same window shifted one place to the right gives the target, so every position's answer is the next token.
Instruction tuning (SFT): Fine-tuning on thousands of examples of instructions paired with good responses. Afterward the model answers instead of just continuing the text.

K

Knowledge cutoff: The date after which a model's training data contains nothing, so it knows nothing about later events unless they are in the prompt.
KV cache: Stored keys and values of earlier tokens, reused while generating so each step computes only the new token. It makes steps cheaper but uses memory that grows with the context.

L

Language model: A model that takes the text so far and gives a probability for every possible next token. Writing means sampling a token, appending it and repeating.
Language-model head: The last linear layer of a language model, turning each vector into one score per vocabulary token.
Layer normalization: Rescaling each token's vector to mean 0 and standard deviation 1, then applying a learned scale and shift. It keeps values steady as they pass through many layers.
Leak test: A check that changes one token and confirms that only that position and later ones change in the output. If earlier positions move, the causal mask is broken.
Llama: A family of open models from Meta, and an example of a modern large decoder-only transformer using RoPE, RMSNorm, SwiGLU and grouped-query attention (GQA).
Logits: The raw scores a model outputs, one per vocabulary token, before softmax turns them into probabilities.

M

Masked language modeling: Training by hiding some tokens and having the model fill them in, as BERT does. Labels of −100 mark unhidden positions so the loss ignores them.
Mixture of experts: A design where each block has many MLPs and a small router sends each token to only a few. The model holds many parameters but uses only some per token.
MLP (multilayer perceptron): A small feed-forward network. In a transformer block it widens each token's vector four times, applies an activation, and narrows it back, processing each token on its own.
Multi-head attention: Several smaller attention heads running side by side, each with its own queries, keys and values, whose outputs are joined and mixed. It costs about the same as one wide head.

O

One-hot vector: A vector of zeros with a single 1 at one position. Multiplying a one-hot vector by the embedding table picks out one row, which is what an embedding lookup does.
Output projection: The linear layer that mixes the joined outputs of all attention heads, so what one head found can combine with another's.

P

Padding: Filler tokens added to make sequences in a batch the same length. A padding mask stops real tokens from attending to them.
Parameters: The learned numbers inside a model, such as embedding tables and linear-layer weight matrices. They can be counted with `sum(p.numel() for p in model.parameters())`; a tiny GPT has about 163,000 and GPT-2 small about 124 million.
Perplexity: e raised to the loss, read as 'as unsure as choosing among N equally likely tokens'. The bigram model's loss of 2.49 is about 12.
Pipeline: A Hugging Face function that wraps tokenizer, model and generation in one object and applies the model's chat template for you.
Position embedding: A vector added to each token's vector to say where it is, because attention alone can't see word order. It can be learned (GPT-2) or fixed from sine and cosine waves.
Pre-norm: Placing layer norm at the input of each part, x + f(LayerNorm(x)), instead of after the addition as in the 2017 paper. It trains more stably in deep stacks and is now standard.
Pre-splitting: Splitting text at spaces and punctuation before BPE merging, so no token crosses a word boundary. Our simple tokenizer lacks it, which is why 'one ships' got split oddly.
Preference tuning: Tuning a model with human comparisons of pairs of answers so it prefers the better ones, done with RLHF or DPO after instruction tuning.
Pretraining: Training on trillions of tokens of general text to predict the next token. The result is a base model.
Prompt: The text a model continues. For a decoder-only model, an input like 'Translate to Spanish: …' is just a context to continue from.

Q

Quantization: Storing a model's weights in fewer bits, such as 4 or 8, to cut memory use.
Query, key, value: The three vectors attention gives each token: a query for what it is looking for, a key for what it offers for matching, and a value for the information it passes on if chosen.

R

register_buffer: A PyTorch way to store a tensor, such as the causal mask, that belongs to the module and moves with it and is saved in the state_dict, but isn't trained.
Regular expression: A text pattern, used in Python through the `re` module, for checking or finding matching text. For example, one can test whether a product blurb ends with a price and a period.
Residual connection: Adding a layer's input to its output, x + f(x), so each layer only learns a change to the vector and gradients flow straight back through the additions.
Reward model: A model trained on human comparisons of answers to score how good an answer is. In RLHF, reinforcement learning uses its scores to push the language model toward better answers.
RLHF (reinforcement learning from human feedback): Human comparisons of answers train a reward model, and reinforcement learning then nudges the language model toward high-scoring answers. It is part of what turns a base model into an assistant.
RMSNorm: A simpler layer norm that rescales by the root mean square without subtracting the mean, used in Llama-style models.
RNN (recurrent network): A network that reads one token at a time while carrying a hidden state. It squeezes the whole past through one vector and can't compute positions in parallel.
RoPE (rotary position embedding): Rotating each query and key by an angle proportional to its position inside attention, so scores depend only on how far apart tokens are. Llama-style models use it.

S

Sampling: Choosing the next token at random according to its probability, rather than always taking the top one.
Scaled dot-product attention: Attention where scores are the query–key dot products divided by √d_k, so large vectors don't make softmax put nearly all weight on one token. The formula is softmax(QKᵀ/√d_k)V.
Scaling laws: The finding that a language model's loss falls predictably as a power law in parameters, data and compute. Labs use it to predict a big model's quality from small experiments.
Self-attention: Attention where the queries, keys and values all come from the same sequence, so the sequence attends to itself.
SentencePiece: A tokenizer library and method used by T5, with a 32,000-token vocabulary.
Sinusoidal positions: Fixed position vectors built from sine and cosine waves of different speeds, as in the original transformer. They need no learning and can be computed for any position.
Smoothing: Adding a small count, such as 1, to every possible pair so that none has probability zero. This prevents an unseen pair from making the loss infinite.
Special tokens: Reserved tokens that mark structure rather than text, such as GPT-2's end-of-text token, BERT's [CLS], [SEP], [MASK] and [PAD], and chat models' turn markers.
state_dict: A PyTorch dictionary of a model's weights and buffers. `state_dict()` saves them and `load_state_dict` restores them into a model of the same sizes.
SwiGLU: A gated MLP used in Llama-style models, where a learned gate multiplies two branches. It gives better quality for the same compute than a plain GELU MLP.

T

T5: A 2019 encoder-decoder model that treats every task as text in, text out, pretrained by filling in hidden spans of text.
Temperature: A setting that divides the logits before softmax: below 1 sharpens the distribution toward the top token, above 1 flattens it. Temperature 0 means greedy decoding.
Token: A piece of text the model handles as one unit, such as a character, a word or part of a word, each with an integer id.
Tokenizer: The tool that splits text into tokens and turns them into ids and back. A model must always be used with the tokenizer it was trained with.
Top-k sampling: Sampling only from the k likeliest tokens, after setting all others' logits to minus infinity.
Top-p sampling: Sampling from the smallest set of top tokens whose probabilities add up to at least p. It keeps few tokens when the model is sure and many when it is unsure.
Transformer: The network design inside ChatGPT-style models, introduced in 2017's 'Attention Is All You Need'. It stacks blocks of attention and an MLP, with residual connections and layer norm.
Transformer block: The repeating unit of a transformer: attention (tokens exchange information) then an MLP (each token alone), each preceded by layer norm and added back through a residual connection. The block keeps the vector shape, so blocks stack.

U

UTF-8: The standard way to store text as bytes (numbers 0 to 255): English letters take one byte and other characters two to four. Byte-level BPE builds on it.

V

Validation loss: The loss measured on held-out text, here the last 10% of the file, that the model never trains on. It shows whether the model generalizes or just memorizes.
Vocabulary: All the tokens a tokenizer knows. Ours has 72 characters at first; GPT-2 has 50,257 tokens.

W

Weight tying: Using one matrix as both the token embedding table and the language-model head, so the vector for 'e' going in is also used to score 'e' coming out. It saves parameters.
weights_only: A `torch.load` option that accepts only tensors and plain Python values, so loading a checkpoint can't run malicious code.
WordPiece: A subword tokenization method, a close cousin of BPE, used by BERT with about 30,000 tokens.