Glossary

123 terms from NLP with Transformers, in plain words, each with the chapter that teaches it.

Search the glossary

#

-100 label: A label value that PyTorch's cross-entropy loss ignores. In NER it goes on special tokens and on every word piece after the first, so only each word's first piece is trained.

A

accelerate: A Hugging Face library that runs training on whatever hardware you have. `Trainer` needs it, and the `transformers[torch]` extra installs it.
Aggregation strategy: A pipeline setting that decides how word pieces are merged into whole entities. `"first"` labels each word by its first piece, so you get 'UPS' instead of fragments like 'UP' and '##S'.
Apache 2.0 / MIT / cc-by-nc-4.0 licenses: Licenses listed in a model card's metadata. Apache 2.0 and MIT allow commercial use, but `cc-by-nc-4.0` is non-commercial, so Kettle & Crane couldn't use it to route email.
Apache Arrow: A column-based table format that the `datasets` library stores on disk. It lets a Dataset be much bigger than your memory.
argmax: A function that returns the position of the largest value. It turns logits or similarity scores into the index of the winning label or document.
Attention mask: A list of 1s and 0s given to the model with a batch. A 1 marks a real token and a 0 marks padding, so attention ignores the `[PAD]` positions.
Auto classes: Classes that read a model's configuration file and pick the right architecture. The same code then works for BERT, DistilBERT, RoBERTa and others. `AutoModel` loads a model with no task head.

B

Bag of words: A text representation that counts which words appear and ignores their order. It is the standard baseline for text, as in TF-IDF.
Baseline: The simplest model that could work, such as always guessing the most common label or TF-IDF with logistic regression. A transformer is only worth its cost if it beats it.
Batching: Running several inputs through a model in one call. Per-call overhead is shared and matrix math runs faster, so the cost per message drops.
Beam search: A way of generating text that keeps the few best partial outputs at each step instead of only one. With `num_beams=4`, T5 tracks four candidate summaries.
BERT: A pretrained encoder transformer trained by hiding words in text and guessing them. It can fill in masked words directly with a `fill-mask` pipeline, and it uses a WordPiece tokenizer with a 30,522-token vocabulary. Smaller models such as DistilBERT are derived from it.
Bigram: A pair of neighboring words used as a feature, such as 'do you'. Setting `ngram_range=(1, 2)` in TF-IDF adds these pairs to single words.
BIO tags: A word-labeling scheme for entities. `B-` marks the first word of an entity, `I-` a word inside it, and `O` a word outside any entity, as in B-PRODUCT I-PRODUCT for 'Night Owl'.
Bootstrap (paired): A method that resamples the test messages with replacement many times and watches how a score difference moves. If the interval for DistilBERT minus the baseline stays above 0, the gain is probably real.
BPE (byte pair encoding): A subword method that repeatedly merges frequent pairs of symbols. WordPiece is described as close to the BPE built in the earlier Transformers Basics book.

C

Cache: The folder where downloaded models are kept and shared by all your projects. Set `HF_HOME` to move it.
Callback: An object `Trainer` calls at fixed moments during training. `EarlyStoppingCallback` is one that stops training when the chosen metric stops improving.
Chat model: A generative language model that follows instructions given as a list of messages, each with a role such as `system` or `user`. A small example is SmolLM2-360M-Instruct, which can write summaries and answer questions from a passage.
Chat template: The text format, with special tokens, that a chat model was trained to read conversations in. The pipeline applies it for you when you pass `system` and `user` messages.
Checkpoint: A saved copy of a model's weights, either at some point during training or the pretrained starting point such as `distilbert/distilbert-base-uncased`. Training checkpoints also hold optimizer state, so the `runs/` folder gets large.
Class weights: Multipliers on the loss that make rare labels count more. Wholesale messages get a weight of 2.35 because they are only about 10% of Priya's training data.
Classification head: The small layer or layers on top of a pretrained model that map its vectors to one logit per label. Fine-tuning adds a new, randomly initialized one for Priya's four labels.
ClassLabel: A `datasets` column type that stores labels as integers and remembers their names. Stratified splitting needs it.
Confidence threshold: The score below which Priya's assistant hands a message to a person instead of acting on its label. Raising it sends more messages to Priya, but confident mistakes still get through.
Configuration: The description of a model's architecture: vocabulary size, width, layers and heads. A model built from a configuration has random weights, unlike one loaded with `from_pretrained`.
Confusion matrix: A table where rows are true labels and columns are predicted labels, showing which classes get mixed up.
CoNLL-2003: A news-text dataset with four entity types: people, places, organizations and miscellaneous. The general NER model `dslim/distilbert-NER` was fine-tuned on it.
Cosine similarity: A measure of how closely two vectors point the same way: 1 for the same direction, 0 for unrelated. Semantic search ranks FAQ entries by it.
Counterfactual test: A bias check that changes only a detail that shouldn't matter, such as a customer's name or a country, and confirms the model's output barely moves.
CUDA: NVIDIA's platform for running computations on its GPUs. A torch version ending in `+cu130` is built for CUDA 13.0, while `+cpu` runs on the CPU only.

D

Data collator: Code that builds each training batch from a list of rows. It pads them to the longest in the batch and renames `label` to `labels`.
Data leakage: When the same or nearly the same messages appear in both training and validation data, so a model scores well by memory. Nine of 240 validation messages were duplicates of training ones.
Dataset, DatasetDict: A Dataset is a table stored on disk in Arrow format. A DatasetDict is a dictionary of named Datasets, usually the train, validation and test splits.
DistilBERT: A smaller, faster version of BERT with 6 layers, made by distillation (training a small model to imitate a larger one). It is a common choice for fine-tuning a text classifier, such as a message router for Priya's inbox.
Distillation: Training a small 'student' model to imitate a large 'teacher' model's outputs, not just the labels. DistilBERT was made from BERT this way.
Dropout: A training technique that randomly switches off some units. It is on only during training, which is why `model.eval()` turns it off for predicting.
DummyClassifier: A scikit-learn model that ignores the text and always predicts the most common label. It is the do-nothing baseline.
Dynamic quantization: Quantization where weights are stored as 8-bit integers and each layer's inputs are also converted to 8 bits on the fly. Matrix multiplications then run in integer arithmetic.

E

Early stopping: Ending training when the chosen metric hasn't improved for a set number of evaluations, then reloading the best epoch. It needs `load_best_model_at_end=True`.
Encoding: What a tokenizer returns: a dictionary of `input_ids`, `attention_mask` and sometimes `token_type_ids`, with one entry per token.
Environment variable: A named setting that programs read from the shell, such as `HF_HOME`. On macOS and Linux you set one with `export NAME=value`.
Epoch: One full pass over the training data. DistilBERT was fine-tuned for 3 epochs, while the small from-scratch model reached its best score at epoch 14.
evaluate: A Hugging Face library of ready-made metrics such as accuracy and ROUGE.
Extractive question answering: Answering by finding a span of a passage: the model scores each token as a possible start and end. It always returns some span, even when the passage has no answer.
Extras: Optional groups of dependencies you request in brackets when installing. `transformers[torch]` adds `accelerate`, which training needs.

F

F1 score, precision and recall: Precision is the share of predictions that are right, and recall is the share of true items that were found. F1 combines the two into one number from 0 to 1.
Fast tokenizer: A tokenizer written in Rust, which is the default from `AutoTokenizer`. It can report offsets and word IDs.
fill-mask: A pipeline task where the model guesses the likeliest words for a hidden word marked `[MASK]`. It works out of the box with BERT because of how BERT was pretrained.
Fine-tuning: Training a pretrained model a little more on your own task with a small learning rate. The model keeps its general language knowledge but learns Priya's four labels.

G

Gated model: A Hub model you can download only after accepting its terms on the website and logging in with an access token.
generate: The method that runs a model's decoding loop to write new text. `max_new_tokens` caps its length and `do_sample=False` makes output repeatable.
GPU: A graphics processor that trains and runs neural network models much faster than a CPU. It isn't required for small models, but services such as Google Colab offer one, often on a free tier.

H

hf auth login: A command from `huggingface_hub` that logs you in with an access token from your Hub settings. You need it for gated models and for pushing your own models.
HF_HOME and HF_HUB_OFFLINE: Two environment variables. `HF_HOME` moves the cache and login token to another folder, and `HF_HUB_OFFLINE=1` stops all Hub contact so only cached files are used.
Hub: The website (huggingface.co) where people share trained models and datasets. Each repository has an ID like `owner/name`.
huggingface_hub: The library that handles downloads, uploads and caching of Hub files, and provides the `hf` command.
hypothesis_template: A sentence pattern such as 'This message is {}.' that zero-shot classification fills with each candidate label to make an NLI hypothesis.

I

id2label / label2id: Dictionaries stored in a model's config that map label numbers to names and back. With them, predictions say 'wholesale' instead of `LABEL_2`.
inference_mode / no_grad: PyTorch contexts that skip gradient tracking when you only predict. `inference_mode` is the stricter, faster version.

J

joblib: A Python library used here to save and reload the scikit-learn baseline as `baseline.joblib`.
JSONL: A file with one JSON object per line. The NER training data (`ner_train.jsonl`) uses it, and `load_dataset("json")` reads it.

L

Label noise: Labels that are simply wrong. About 3% of Priya's labels are, so some validation errors are the labels' fault, not the model's, and reading them is worth the time.
Latency: The time a model takes to answer one request, measured here in milliseconds per message.
Learning rate: The size of each weight-update step. Fine-tuning uses small values like 5e-5 so the pretrained knowledge is adjusted rather than overwritten.
Load report: The table `from_pretrained` prints listing weights that were newly initialized (`MISSING`) or not used (`UNEXPECTED`). For a new classification head this is normal.
Logistic regression: A simple classifier that learns one weight per feature and label, then turns the weighted sum into label probabilities. With TF-IDF word features it makes a fast, easy-to-inspect baseline for text classification.
Logits: The raw scores a model outputs, one per label for each text, before softmax turns them into probabilities.
LoRA (low-rank adaptation): A fine-tuning method that trains small added matrices while the model's own weights stay frozen. It is covered in a later book.

M

Macro F1: The F1 score averaged over labels with equal weight, so neglected rare labels can't hide. It is reported next to accuracy with imbalanced classes.
map: A `datasets` method that runs a function over every row and adds the columns it returns. With `batched=True` it passes 1,000 rows at a time, which is much faster for tokenizers.
Mean pooling: Averaging the per-token vectors, ignoring padding, into one sentence vector.
MiniLM: A small sentence-embedding model (87 MB, 384-number vectors) used for FAQ search. A multilingual version handles Spanish questions against the English FAQ.
Model card: A model's README on the Hub, describing its uses, training data, evaluation, limitations and license. Read it before using a model.
Multilingual model: A model trained on many languages, so text in different languages lands in the same space. English-only models fail on Spanish messages; multilingual ones can match a Spanish question to an English answer.
MultipleNegativesRankingLoss: A loss for embedding models that pulls each question toward its answer and away from the other answers in the same batch, so no wrong answers need labeling.

N

Named entity recognition (NER): Labeling the words that name things, such as products, cities and carriers, so a complaint can be filed with its details.
Natural language inference (NLI): Deciding whether a premise implies a hypothesis. Zero-shot classification reuses this skill by turning each label into a hypothesis.
Normalizer, pre-tokenizer, post-processor: Three tokenizer parts. The normalizer cleans text, the pre-tokenizer splits it into words, and the post-processor adds special tokens such as `[CLS]` and `[SEP]`.

O

Offsets: Each token's (start, end) character positions in the original text, so `text[start:end]` recovers the characters.
ONNX Runtime: An engine for serving models efficiently. It is the usual next step for serving at scale, through the `optimum-onnx` package or `backend="onnx"` in sentence-transformers.
Overfitting: When a model learns its training data too specifically and does worse on new data. Rising validation loss is the warning sign.

P

Padding, truncation: Padding adds `[PAD]` tokens to short texts so a batch has equal lengths. Truncation cuts texts that are longer than the model's limit, such as 512 tokens for BERT.
Parameters (weights): The numbers a model learns during training. DistilBERT has about 67 million, and the small from-scratch model has 682,884.
Pipeline: A `transformers` helper that bundles a tokenizer, a model and post-processing, so a model runs in about three lines. You give it a task name and a model ID.
Pretraining: Training a model on huge amounts of general text before it sees your specific task. For BERT, this means guessing hidden words. The model can then be fine-tuned on a smaller labeled dataset, usually doing better than one trained from scratch.

Q

Quantization: Storing weights as 8-bit integers instead of 32-bit floats. It made DistilBERT 15% to 40% faster here without changing an answer.

R

Random seed: A number that fixes a random process so it repeats exactly, such as a data split or the starting weights. It keeps score changes tied to your changes, not to luck.
Recall@k: The share of search queries whose right answer appears in the top k results.
Revision: A branch, tag or commit hash of a Hub repository. Pin one for production, since repositories can change.
RoBERTa / XLM-RoBERTa: BERT-family encoder models. RoBERTa uses `<s>` and `</s>` as special tokens, and XLM-RoBERTa is multilingual.
ROUGE: A word-overlap score between a summary and a human reference. It counts words, not meaning, so it can rate a summary with a flipped meaning almost perfectly.

S

Safetensors: A weight file format that can't run code when loaded. Prefer it to older `.bin` files, which can.
scikit-learn: A Python machine-learning library with classifiers, text feature tools and evaluation metrics. It is often used to build a TF-IDF baseline and to produce accuracy, F1 and classification reports.
Semantic search: Searching by meaning rather than shared words, by embedding the query and the documents and ranking by cosine similarity. It finds the right FAQ for 'Can you send a bag to Honolulu?'.
Sentence embedding: One vector for a whole text, made so that similar meanings point in similar directions.
sentence-transformers: A library that turns sentences into vectors for search. Its trainer is `Trainer` adapted for embedding models.
Seq2seq: An encoder-decoder model that reads one text and writes another, such as T5 for summaries. You load it with `AutoModelForSeq2SeqLM` and call `generate`.
Softmax: A function that turns logits into probabilities that add up to 1. It always shows confidence in a choice among the available labels, even wrong ones.
Special tokens: Reserved tokens with jobs: `[CLS]` starts an input, `[SEP]` ends or separates texts, `[PAD]` fills short texts, `[MASK]` hides a word and `[UNK]` stands for unknown text.
SQuAD: The Stanford Question Answering Dataset. v1.1 questions always have an answer in the passage, while v2.0 adds unanswerable ones so models can learn to say 'no answer'.
SST-2: A dataset of movie-review sentences labeled positive or negative. Its fine-tuned model can only say POSITIVE or NEGATIVE, so it can't separate Priya's four labels.
Stratified split: A train/validation split that keeps each label's share the same in both parts. It needs a ClassLabel column, and a `seed` makes it repeatable.
Subword: A piece of a word that a tokenizer uses as a token. Common words stay whole, and rarer ones split into pieces, like `es ##press ##o`.

T

T5: A seq2seq (encoder-decoder) transformer that chooses its task from a text prefix such as `summarize: `. The small `google-t5/t5-small` version can summarize text using the model's `generate` method.
Tensor: PyTorch's array type for numbers. `return_tensors="pt"` makes a tokenizer return PyTorch tensors instead of lists.
Test set: Data used once, at the end, to estimate real performance. Each time you change a model because of it, it stops being an honest estimate.
TF-IDF: A word-weighting method: high when a word is frequent in this message but rare across messages. It makes each message a vector with one number per vocabulary word.
Throughput: How many messages a model handles per second.
Token classification: One label per token instead of one per text, as in NER.
Token type IDs: A list marking which of two paired texts each token belongs to: 0 for the first, 1 for the second.
Tokenizer: The component that turns text into token IDs a model understands. Always use the one that came with the model. The `tokenizers` library also lets you train your own.
torchao: A library whose `quantize_` function applies int8 quantization to a model. It replaces PyTorch's deprecated `torch.ao.quantization`.
Trainer, TrainingArguments: `TrainingArguments` holds the training settings, and `Trainer` runs the loop: batches, loss, optimizer steps, evaluation and checkpoints. `compute_metrics` turns logits into scores after each evaluation.
trust_remote_code: A loading option that, when `True`, runs the repository's own Python code on your machine. Leave it `False`, the default.

V

Validation set: Data for making choices such as how many epochs or what threshold to use, kept apart from the training data and the final test set.
Vector database: A system for storing and searching millions of embeddings. It is mentioned as the tool for large-scale search.
Vocabulary: The fixed list of tokens a tokenizer can produce, each with a numeric ID. BERT's uncased tokenizer has 30,522 tokens, while a tokenizer trained from scratch on a small dataset may have only a few thousand.

W

Weight decay: A small penalty on large weights, used here with AdamW, the default optimizer in `Trainer`.
Word IDs: A list giving each token's word number in the original text, or `None` for special tokens. It lets you align word-level labels with subword tokens.
WordPiece: BERT's subword method, where common words are single tokens and rarer ones split into pieces. A `##` prefix marks a piece that continues the previous one.

Z

Zero-shot classification: Classifying with labels you choose at run time, by turning each into an NLI hypothesis, with no training. It is a useful start but weak on rare labels like wholesale (recall 0.200).