Glossary

119 terms from Deep Learning Basics, in plain words, each with the chapter that teaches it.

Search the glossary

#

__getitem__: The two methods a custom Dataset must define: `__len__` returns the number of examples and `__getitem__` returns example `i`.

A

Accelerator: A chip, such as a graphics card, that speeds up the math of training. `torch.accelerator.is_available()` tells you whether PyTorch can see one; if not, it uses the CPU.
Accuracy: The share of predictions that are right. For the FashionMNIST classifier, it is the mean of `logits.argmax(dim=1) == labels`.
Activation function: A function applied to each number a layer outputs, bending it so stacked layers can learn curves. Without activations, many layers collapse into one linear layer.
Adam: An optimizer that keeps running averages of each parameter's gradient and its size, and scales each parameter's step separately. Its default learning rate of 0.001 makes it a common first choice.
AdamW: A version of the Adam optimizer that applies weight decay directly to the weights, shrinking them slightly at each step, instead of adding an L2 penalty to the gradient. It is used to regularize neural networks and reduce overfitting.
argmax: Returns the position of the largest value. Applied to a row of logits, it gives the predicted class.
Autograd: PyTorch's system for computing gradients automatically. Mark a tensor with `requires_grad=True`, call `.backward()` on a result, and read `.grad`.

B

backward: The call that makes autograd compute the gradient of the loss with respect to every parameter. Gradients are added into `.grad` until you clear them.
Batch: A small group of examples (typically 32 to 256) used for one training step. Its gradient is a noisy but cheap estimate of the full gradient.
Batch normalization: A layer that standardizes each feature across the batch to mean 0 and standard deviation 1, then applies a learned scale and shift. In evaluation mode it uses running averages saved during training.
BCE (binary cross-entropy): The yes/no loss: −log of the probability given to the true answer, so confident wrong answers cost a lot. `BCEWithLogitsLoss` is the version you should use because it takes logits.
BCEWithLogitsLoss: PyTorch's loss for yes/no predictions. It takes raw logits and float 0.0/1.0 targets and applies the sigmoid inside, which avoids the silent-learning-stop problem of a sigmoid followed by `BCELoss`.
bfloat16: A 16-bit number format with float32's range, supported on newer GPUs and CPUs. It needs no gradient scaler for mixed precision.
Bias: The number added to a layer's weighted sum. Along with weights, it is one of a model's learned parameters.
Broadcasting: Stretching a smaller tensor to match a bigger one's shape. Shapes are compared from the right and a size of 1 is stretched, as when a (2, 1) discount column multiplies prices of shape (3,).

C

Channel: One layer of an image or feature map: 1 for grayscale, 3 for color (red, green, blue). Image batches are shaped (N, C, H, W).
Checkpoint: A saved file holding everything needed to resume training: the model's weights, the optimizer's state and the epoch (and optionally the scheduler).
CNN (convolutional neural network): A network built from convolution, activation and pooling blocks followed by linear layers, designed for images. It beat an MLP of similar size on FashionMNIST.
Confidence: The highest probability a classifier gives an image, from softmax. Low confidence is a useful warning sign, though some wrong answers are still over 90% confident.
Confusion matrix: A table counting each true class (rows) against each predicted class (columns). It shows which classes the model mixes up, such as Shirt and T-shirt/top.
Convolution: Sliding a small grid of weights (the kernel) across an image, multiplying and summing at each position to build a feature map. The same weights are used everywhere, so a pattern learned in one place is recognized in all places.
copy.deepcopy: Makes a fully independent snapshot of an object. It is needed when storing `model.state_dict()`, which otherwise returns live tensors that later training steps change.
CPU: The computer's main processor, and the default device on which PyTorch tensors live and computations run. It works for any model, though an accelerator such as a GPU can train large models faster.
Cross-entropy: The loss for multi-class problems: −log of the probability given to the true class. `nn.CrossEntropyLoss` applies softmax internally and takes int64 class numbers as targets.

D

Data augmentation: Adding altered copies of training examples, such as flipped or shifted images, to fight overfitting.
Data leakage: Information from validation or test rows sneaking into training, for example by fitting a scaler on all the data. Fit preprocessing on training rows only.
DataLoader: Takes a Dataset and hands out batches, reshuffling every epoch when `shuffle=True`. Options: `batch_size`; `drop_last=True` skips a smaller final batch (avoids batch-norm errors on a 1-example batch); `num_workers` loads in background processes (needs the `if __name__ == "__main__":` guard above 0); `pin_memory=True` speeds copies to a GPU.
Dataset: An object that knows how many examples there are (`__len__`) and how to fetch example `i` (`__getitem__`). You can write your own, like `SubscriberData`.
Dead network: A network whose ReLUs output 0 for every input, so no gradient flows and every prediction is the same class. Lower the learning rate and check for unscaled inputs.
Decorator: A line starting with `@` above a function that wraps it. `@torch.no_grad()` runs the whole function without recording gradients.
Deep learning: Training neural networks with several layers that learn their own features from raw inputs such as pixels, sound or words.
detach: Returns a tensor with the same numbers but cut off from autograd's record. Useful when storing predictions for later.
Device: Where a tensor lives and is computed: `cpu` (main processor), `cuda` (NVIDIA GPU) or `mps` (Apple-silicon GPU, M1 and later). Everything in one calculation must be on the same device; move things with `.to(device)`.
Dropout: Randomly sets a share `p` of a layer's outputs to zero at each training step, scaling the rest up, so the network can't rely on a few units to memorize rows. It does nothing in evaluation mode.
dtype: The type of number a tensor stores, such as float32 or int64. Mismatched dtypes cause errors.

E

Early stopping: Keeping the weights from the epoch with the lowest validation loss and stopping once it hasn't improved for `patience` epochs (the number of epochs to wait).
Epoch: One pass through all the training examples. With 480 examples and batches of 32, one epoch is 15 steps.
Error analysis: Studying which mistakes a model makes and why, such as the most common confusions and its most confident wrong predictions, rather than only how often it errs.

F

FashionMNIST: A practice dataset of 70,000 small 28 × 28 grayscale clothing images in 10 classes (60,000 for training, 10,000 for testing). Zalando made it as a harder replacement for MNIST, a set of handwritten digits.
Feature map: The output grid produced by one convolution kernel sliding over an input. A layer with 16 kernels makes 16 feature maps, i.e. 16 channels.
Feature scaling: Rescaling inputs (and targets) to small, similar-sized numbers, such as mean 0 and standard deviation 1, because gradient descent works best on them. New inputs must be scaled exactly like the training data.
flatten: Turns each example's multi-dimensional feature maps into one long row. `nn.Flatten()` turns (32, 7, 7) into 1,568 numbers before the linear layers.
float32, int64: float32 is the 32-bit decimal type used for data and weights; int64 is the 64-bit integer type PyTorch wants for class labels.
forward: The method of an `nn.Module` that says how data flows through its layers. You call the model itself, not `forward` directly.
functional (F): `torch.nn.functional`, imported as `F`, holds layers as plain functions, such as `F.conv2d`.

G

Google Colab: A service that runs Python notebooks in the browser on Google's machines, with a free GPU when available. Save checkpoints to Google Drive because sessions end.
GPU: A graphics processing unit does thousands of multiplications at once, so training on it can be many times faster. Tensors and models must be moved to it with `.to(device)`.
Gradient: The list of slopes of the loss with respect to each parameter. Training steps each weight a little downhill along it.
Gradient boosting: A strong, fast model for tabular data that builds many small decision trees in sequence, each correcting the errors of the previous ones. It needs little tuning and, along with logistic regression, is usually a better first choice than a neural network for tables.
Gradient clipping: Scaling gradients down whenever their combined size (the norm) is too large, to stop exploding gradients in deeper networks. `nn.utils.clip_grad_norm_` does it, is called between `backward` and `step`, and returns the norm before clipping.
Gradient scaler: Multiplies the loss before `backward` in float16 training so tiny gradients don't round to zero, then unscales them before the optimizer step.

H

Hidden layer: A layer between the input and the output, such as the 8-unit layer in a small `nn.Sequential` network.

I

item: Turns a single-number tensor into a plain Python number, handy for printing or storing a loss.

K

Kernel (filter): The small grid of weights a convolution slides over the image. Kernels are learned like any other weights.

L

Label noise: Wrong or doubtful labels in the data itself. Some of the clothing classifier's most confident mistakes may be label noise.
lambda: A one-line Python function without a name, written as `lambda arguments: expression`. It is handy for making small throwaway functions, such as a recipe that builds an optimizer from a model's parameters.
Layer: One step of a network, such as `nn.Linear`: a weighted sum of its inputs followed by a bend, with weights learned from data.
Layer normalization: Standardizes each example across its own features. It doesn't depend on the batch, so it behaves the same in training and evaluation, and transformers use it.
Learning rate: The size of each optimizer step. Too small barely moves; too large overshoots and the loss explodes to `nan`. Try values a factor of 10 apart.
Learning-rate schedule: A plan that starts with a big learning rate and shrinks it during training. `StepLR` multiplies it by `gamma` every few epochs; `CosineAnnealingLR` glides it toward zero along a half cosine.
load_state_dict: Copies saved weights into a model with the same architecture, as in `restored.load_state_dict(...)`.
Logistic regression: A classifier for yes/no questions that computes a weighted sum of the inputs and turns it into a probability. It is equivalent to a one-layer neural network, such as `nn.Linear(22, 1)` trained with `BCEWithLogitsLoss`.
Logit: A model's raw output score before sigmoid or softmax. Large positive means very likely yes, large negative very likely no, and 0 means 50/50.
Loss function: Turns the gap between predictions and true answers into one number; lower is better. Use MSELoss for numbers, BCEWithLogitsLoss for yes/no and CrossEntropyLoss for classes.

M

MAE (mean absolute error): The average size of the errors, in the original units, such as 8.5 cups for the cold brew network.
Mask: Indexing a tensor with a True/False tensor, such as `orders[orders > 0]`, which keeps only the True positions.
Max pooling: Keeps the largest value in each small block (2 × 2 for `nn.MaxPool2d(2)`), halving height and width. It saves work and makes the network less sensitive to small shifts.
Mixed precision: Training that does most of the math in 16-bit numbers, which can be about twice as fast on modern GPUs and halves memory use. It mainly pays off for big models, and with float16 it uses a gradient scaler to keep tiny gradients from rounding to zero.
MLP (multilayer perceptron): A network of `nn.Linear` layers with activations between them. It treats an image as 784 unrelated columns, unlike a CNN.
model.train() / model.eval(): `model.train()` makes dropout active and batch norm use batch statistics; `model.eval()` switches them to prediction behavior. Call train before training batches, and eval together with `torch.no_grad()` before measuring or predicting.
Momentum: Each SGD step also carries part (e.g. 90%) of the previous step, like a ball rolling downhill, speeding through long gentle slopes.
MSELoss: The loss for predicting a number: the average of the squared errors. It equals `((pred - target) ** 2).mean()`.

N

nan: "Not a number": appears when a calculation goes out of range, then spreads through every result it touches. Causes include a too-high learning rate, gaps in data and a hand-made log of a probability.
Neural network: A stack of layers, each a weighted sum of its inputs followed by a nonlinear bend, with the weights learned from data.
nn.Linear: A fully connected layer: each output is a weighted sum of all inputs plus a bias, computed on a batch as `x @ W.T + b`.
nn.Module: The base class of every PyTorch model. You write `__init__` to create layers (calling `super().__init__()` first) and `forward` to say how data flows through them.
nn.Sequential: A container that chains parts in order, feeding each one's output into the next.
Normalization layer: A layer that rescales the numbers flowing between layers so each gets inputs of a steady size, which helps deep networks train faster and more stably.

O

One-hot encoding: Turning a category into a set of 0/1 columns, one per category, so a network can take it as numbers.
Optimizer: An object that updates a model's parameters from their gradients, at a given learning rate. Each step is `zero_grad`, forward and loss, `backward`, `step`.
Overfitting: Learning the training data's noise: training loss keeps falling while validation loss turns upward. Priya's network began overfitting after epoch 6.

P

Padding: A border of zeros added around an image so a convolution keeps the input's size. With a 3 × 3 kernel, `padding=1` keeps a 28 × 28 image 28 × 28.
Parameters: A model's learned numbers: its weights and biases. Count them with `sum(p.numel() for p in model.parameters())`, since a model with far more parameters than examples can memorize them.
Pickle: Python's format for saving objects. Loading a pickle can run any code hidden inside it, which is why you only load files you trust and use `weights_only=True`.
PIL (Pillow): Python's image library, installed with torchvision. It opens, resizes, converts and inverts pictures such as Priya's photos.
PyTorch: A widely used deep learning library that provides tensors, automatic differentiation (autograd), layers, loss functions, optimizers and data loading. It is used in both research and industry to build and train neural networks.

R

random_split: Divides a Dataset at random into parts of given sizes; a seeded generator makes it repeatable. Each part is a `Subset`, a view of the original through a list of indices.
randperm: Gives the numbers 0 to n − 1 in random order, used to shuffle and slice data into batches or splits.
Regularization: Anything that helps a model generalize to new data, usually by stopping it fitting training noise, such as dropout, weight decay, early stopping, a smaller network or data augmentation.
ReLU: The activation max(0, z), the default between layers. Its slope is 1 for positive inputs, so gradients pass through many layers without shrinking.
requires_grad: A tensor flag telling autograd to record operations on it so gradients can be computed. `nn.Linear` sets it on its weights automatically.
reshape: `reshape` changes a tensor's grid while keeping its numbers in order (`-1` means work out this size). `unsqueeze(d)` adds a size-1 dimension and `squeeze()` removes them, for example adding a channel axis to images.
ROC AUC: The chance that the model ranks a random canceler above a random stayer: 0.5 is guessing and 1 is perfect. It only depends on ranking, so logits work directly.

S

Sanity checks: Three quick checks before a long run: the starting loss should be about log(k) for k classes, the model should overfit one batch to near-zero loss, and a printed batch should have sensible shapes, dtypes and ranges.
Seed: A number that fixes PyTorch's random number generator so "random" values and shuffles repeat each run; a separate `torch.Generator` keeps a split fixed regardless of other random use. Pair it with `torch.set_num_threads(1)`, since splitting work across cores can add numbers in a different order and change the last digits.
SGD: Stochastic gradient descent: each step moves the parameters by `lr * grad` computed from one random batch. "Stochastic" refers to using random batches.
Shape: The size of each dimension of a tensor. Image batches are (N, C, H, W) and the first dimension is the batch.
Sigmoid: Squeezes any number into 0 to 1, so it turns a logit into a probability. It also appears as an activation function.
Softmax: Turns k logits into k probabilities that add up to 1, by exponentiating each and dividing by the total. Don't put it in the model when using `CrossEntropyLoss`, which applies it itself.
state_dict: A dictionary mapping each parameter's name to its tensor, for a model or an optimizer. It's what you save with `torch.save` and restore with `load_state_dict`.
Stride: How far a kernel moves at each step as it slides across the image; the default is 1.

T

tanh: An activation that squeezes values into −1 to 1.
Tensor: PyTorch's array: a grid of numbers with any number of dimensions, with a shape, a dtype and a device. It can also record operations for autograd.
TensorDataset: The ready-made Dataset for data already in tensors, pairing row i of the inputs with row i of the targets.
Test set: Examples kept apart from training and validation and scored once at the end, to give an honest estimate of performance on new data.
torch.compile: A PyTorch function that turns a model into optimized code the first time it runs, which can speed up large models. The compile step takes time and can make small models slower, so measure before relying on it.
torch.no_grad: A block (or decorator) inside which results don't record gradients. Use it when only predicting, to save memory and time.
torch.save / torch.load: `torch.save` writes tensors or dictionaries of tensors to a file (usually `.pt`) and `torch.load` reads them back. Load with `weights_only=True` for safety, and add `map_location="cpu"` to load a GPU-saved model on a CPU.
torchvision: PyTorch's companion library for images: it downloads standard image datasets and converts pictures into numbers.
Transform: A function applied to each example as it's loaded. In `torchvision.transforms.v2`, `ToImage` makes an image tensor, `ToDtype(torch.float32, scale=True)` scales 0 to 255 into 0 to 1, and `Compose` chains them.
Transformers: The network design behind ChatGPT, used for text, where meaning depends on word order and context. They use layer normalization.

U

Underfitting: When both training and validation loss stay high, because of too much regularization or a model that is too small.

V

Validation set: Examples held back from training, used to tune choices such as the number of epochs. The test set is kept separate and used once at the end.
Vanishing gradients: Gradients that shrink toward zero so weights barely change and learning stalls, for example when huge inputs push a softmax to exactly 0 or 1. ReLU's slope of 1 helps avoid it.
Virtual environment: An isolated Python setup into which `pip install torch torchvision` installs. Activate it before installing so the packages go in the right place.

W

Weight decay: Shrinks every weight a little toward zero at each step, so only weights the data keeps pushing on stay large. It is the network version of the penalty in ML Basics' logistic regression.
weights_only=True: A `torch.load` setting that accepts only tensors and plain values like numbers, strings, lists and dicts, refusing anything else. It's the default since PyTorch 2.6, but write it anyway.

Z

zero_grad: Clears the old gradients before each backward pass. Forgetting it makes gradients pile up and the loss rise.