Glossary

109 terms from Calculus for ML, in plain words, each with the chapter that teaches it.

Search the glossary

A

Accuracy: The fraction of predictions a model gets right. Sam's café neuron got 18 of 20 right, so its accuracy is 0.9.
Activation function: A function such as tanh or ReLU that is applied to each hidden unit's score. It bends the output, which lets a network fit curves a straight line can't.
Adam: A popular optimizer that combines momentum with a separate step size for each parameter. In PyTorch you use it by swapping `SGD` for `torch.optim.Adam`.
Analytic gradient: A gradient computed from a formula you derived, rather than estimated by nudging inputs. A gradient check compares it with a numerical gradient.
Antiderivative: A function whose derivative is the function you started with. For example, x³/3 is an antiderivative of x², which is how the area under x² from 0 to 3 comes out as exactly 9.
Automatic differentiation (autodiff): A way to get exact gradients without writing formulas. The library records every operation as your code runs, then runs the chain rule backward through that record.
Average rate of change: The change in output divided by the change in input between two points, Δy/Δx. For Sam's demand line, going from $14 to $16 gives -18 bags per dollar.

B

Backpropagation: The chain rule applied backward through a network's layers, one local derivative at a time, to get the gradient for every parameter. Alex's cold brew network is trained this way.
Bias: A parameter added to a weighted sum that shifts every prediction up or down. In the demand line `w·x + b`, b is the predicted sales at $15.
Broadcasting: NumPy's rule for combining arrays of different shapes. For example, adding `b1` of shape (H,) to an (n, H) array adds it to every row.

C

Central difference: A numerical derivative that steps a little both ways and takes the slope across the middle: (f(x+h) - f(x-h)) / (2h). It is much more accurate than stepping only to one side.
Chain rule: When one function feeds into another, their derivatives multiply: dy/dx = dy/du · du/dx. It is how a price change turns into a change in pounds of coffee, and how a loss turns into a gradient for a weight.
Computation graph: A calculation drawn as boxes for operations and arrows for the values passing between them. You run it forward to get values and backward to get slopes.
Constant multiple rule: The derivative of c·f(x) is c·f'(x). Stretching a curve 7 times taller makes its slopes 7 times steeper.
Converge: To settle on values that further steps no longer change. Gradient descent on Sam's price data converged at an MSE of 14.985.
Cross-entropy: A loss for yes-or-no predictions, -[y·log(p) + (1-y)·log(1-p)]. A confident wrong answer is punished heavily, and a confident right one costs almost nothing.
CUDA: NVIDIA's software for running computations on its graphics cards. The `+cu130` in a PyTorch version string names a CUDA build.

D

Derivative: The slope of a function at one exact point, written f'(x) or dy/dx. It is the value the slope over a shrinking interval h settles toward.
Diverge: To grow out of control. With a learning rate that's too big, each gradient step overshoots and lands higher, so the numbers explode.
Dot product: Multiply matching elements of two vectors, then add them up. A neuron's weighted sum is a dot product of its inputs and weights.

E

Epoch: One full pass through all the training data. In the mini-batch example, one epoch is two batches of 5 weeks.
Exponential function: exp(x), also written eˣ, raises e ≈ 2.718 to the power x. It is always positive, grows faster and faster, and is its own derivative.

F

Feature: One input column of the data. Each café has two features: weeks since its last order and bags ordered.
Float: How computers store decimal numbers, keeping only about 16 significant digits. That is why `0.1 + 0.2` gives `0.30000000000000004`.
Forward pass: Computing every value in a calculation from the inputs to the output, and keeping them for later. The backward pass needs these stored values.
Function: A rule that turns an input number into an output number. For example, demand(p) = 400 - 18p gives bags sold at price p.
Fundamental theorem of calculus: The link between slopes and areas: the area under a curve from a to b is an antiderivative at b minus at a. It means the integral of a rate is a total.

G

Global minimum: The lowest point of a function overall. Gradient descent can miss it and get stuck in a shallower valley.
Gradient: The vector of all partial derivatives of a function. It points in the direction of steepest uphill, and its length tells you how steep that is.
Gradient ascent: Stepping with the gradient instead of against it, to climb to a maximum. It could be used to find the peak of the profit curve.
Gradient check: Comparing your hand-derived gradients with numerical ones to catch backpropagation bugs. Such bugs otherwise fail silently, because the model still trains, just badly.
Gradient descent: Repeatedly moving the parameters a small step against the gradient to lower the loss: params ← params - η·∇L. It needs only a gradient, so it can train almost any model.
Graph: A picture of a function that plots each input across the page and its output upward. It is also just a table of inputs and outputs.

H

Hidden layer / hidden unit: A hidden layer is a layer of neurons between the input and the output, and a hidden unit is one neuron in it. The cold brew network has 3 hidden units.

I

Inner / outer function: In f(g(x)), g is the inner function, computed first, and f is the outer function, applied to g's result. The chain rule multiplies their slopes.
Integral: The sum of a function's values across a range, which on a graph is the area between the curve and the horizontal axis. It is written ∫ₐᵇ f(x) dx.
Intercept: The value of y where a line crosses the vertical axis (x = 0), the b in y = m·x + b. Sam's demand line has intercept 400.

L

Label: The known right answer for an example, such as 1 if a café reordered and 0 if not. The loss compares the model's prediction with it.
lambda: A Python keyword for a one-line function with no name. `lambda x: x ** 3` takes x and returns x ** 3.
Layer: A group of neurons that all take the same inputs. A neural network stacks layers, each feeding the next.
Learning rate: The small positive number that sets the step size in gradient descent. Too small and training crawls, too big and it overshoots and diverges.
Length (norm): The size of a vector: the square root of the sum of its squared elements. The gradient [4, 6] has length about 7.21.
Limit: The value a quantity settles toward as something shrinks. The slopes of x² from 1 to 1+h tend to the limit 2 as h tends to 0.
Linear: Changing by equal amounts for equal steps in the input, so the graph is a straight line. Sam's demand function loses exactly 18 bags per $1.
Linear approximation: Treating a curve as its tangent line near a point: f(x+Δx) ≈ f(x) + f'(x)·Δx. It predicts what a small nudge will do, and gradient descent is built on it.
Local derivative: The slope of just one operation (one box) in a computation graph. The backward pass multiplies one local derivative per box.
Local minimum: A point lower than everything nearby but not the lowest overall. Gradient descent can get stuck there because it only sees the slope where it stands.
Logarithm (natural): log(x) undoes exp, and only works for inputs above 0. Its derivative is 1/x.
Logistic regression: A single neuron with a sigmoid output, used for yes-or-no predictions such as which cafés will reorder.
Loss: A number measuring how wrong a model is. With the data fixed, it is a function of the parameters, and training lowers it.

M

Matrix-vector product: Multiplying a matrix by a vector, giving one dot product per row. `X @ w` with shapes (20, 2) @ (2,) gives one score per café.
Mean absolute error: The average of the absolute (always positive) errors. It cares less about big misses than MSE, but its slope jumps at zero error, so it is awkward to minimize.
Mean squared error (MSE): The average of the squared errors over a dataset. Squaring makes every miss positive, punishes big misses most, and gives a smooth bowl with a clean gradient.
Mini-batch / SGD: Taking each gradient step using a small random batch of the data, such as 5 of Sam's 10 weeks. SGD stands for stochastic gradient descent, where 'stochastic' means random. It is cheaper per step and its randomness helps shake training off saddle points.
Minimum / maximum: The lowest or highest point of a function. At either one the slope is zero, and finding a minimum is what training a model does.
Model: A formula with adjustable numbers inside it (its parameters) that makes predictions. Training adjusts the parameters so the predictions are less wrong.
Momentum: Keeping part of the previous step's direction during gradient descent, like a ball rolling downhill. It helps coast across plateaus.

N

Neural network: Layers of neurons, each layer feeding the next. Alex's cold brew network has a hidden layer of 3 tanh units and one output neuron.
Neuron: A weighted sum of inputs plus a bias, passed through an activation function. With a sigmoid activation, a neuron can turn input features, such as a café's weeks since last order and order size, into a probability like the chance of reordering.
Normal distribution: The familiar bell-shaped probability density. Alex uses one, with mean 3 days and standard deviation 0.8, to model UPS delivery times.
np.argmax: A NumPy function that gives the position of the largest value in an array. It finds the grid price with the highest profit.
np.linspace: Makes `num` evenly spaced numbers from `start` to `stop`, both ends included.
np.loadtxt: Reads a text file of numbers into a 2D NumPy array. `delimiter=","` splits on commas and `skiprows=1` skips the header line.
np.polyfit: A NumPy function that fits a polynomial to data directly using least squares. `np.polyfit(x, y, 1)` fits a line, and gradient descent found the same line.
np.trapezoid: A NumPy function that estimates an integral by joining points with straight lines and adding up the trapezoids. `np.trapz` is the old name, removed in NumPy 2.0.
Numerical derivative: A derivative estimated from function values using a small h rather than a formula. It is approximate, but it works on anything and is used to check every hand-derived formula.
NumPy: The Python library for arrays of numbers. It does arithmetic on every element at once, with no loop.

O

Optimizer: An object that applies an update rule to a list of parameters, such as `torch.optim.SGD` doing plain gradient descent. Each training step calls `zero_grad`, then `backward`, then `step`.
Output layer: The last layer of a network, which produces the prediction. The cold brew network's output layer is one neuron with no activation.
Overfitting: Memorizing the individual training examples instead of learning the general pattern. The tiny café network got all 20 right and became overconfident.
Overflow: When a calculation produces a number too large for a float, such as `np.exp(-z)` for z below about -710. It shows up as a RuntimeWarning, then `inf` or `nan`.

P

Parabola: The U-shaped graph of x². Equal steps in x give unequal changes in the output.
Parameter: An adjustable number inside a model, such as a weight or bias. Training finds parameter values that make predictions less wrong.
Partial derivative: The slope with respect to one input while all the others are held fixed, written ∂f/∂x. You treat the frozen inputs as constants.
Plateau: A flat stretch where the loss barely changes for many steps before falling again. It is common near saddle points.
Polynomial: A sum of powers of x times numbers, such as x² + 5x + 3. The power, sum and constant multiple rules differentiate any polynomial.
Power rule: The derivative of xⁿ is n·xⁿ⁻¹: bring the power down in front, then lower it by 1. It also works for negative and fractional powers.
Probability: A number from 0 to 1 saying how likely something is. A model's probability p is its confidence that the answer is yes.
Probability density: A curve whose total area is 1, where the area between two values is the probability of landing between them. About 89% of Alex's packages arrive within 4 days.
Product rule: The derivative of a product: d/dx (f·g) = f'·g + f·g'. In machine learning you can usually expand the product or use the chain rule instead.
PyTorch: A widely used deep learning library that has automatic differentiation built in. You mark parameters with `requires_grad=True` and call `.backward()` to get gradients.

R

Relative error: |a - n| / (|a| + |n|), a comparison of two numbers that is fair for both big and small gradients. Below 1e-6 means your gradient is correct, and above 1e-3 means a bug.
ReLU: An activation function, max(0, z). Its slope is 1 where z > 0 and 0 elsewhere.
requires_grad: A PyTorch flag telling it to record operations on a tensor so gradients can be computed. After `backward()`, each gradient is in `.grad`, and new gradients are added to old ones unless you clear them.
Reverse mode: Running the chain rule backward from the output, which is how autodiff libraries get all gradients for about the cost of one extra pass.

S

Saddle point: A point where the gradient is zero but which is neither a minimum nor a maximum, like the middle of a horse's saddle. Training slows down near one.
Scientific notation: Writing numbers as powers of 10. In Python, `1e-5` means 0.00001 and `1e3` means 1000.
Score: A neuron's weighted sum plus bias, before any activation is applied. The sigmoid then turns the score into a probability.
Secant line: A line through two points on a curve. As the second point closes in on the first, the secant becomes the tangent line.
Second derivative: The slope of the slope, which measures how sharply a curve bends. It sets the learning-rate limit: the rate must stay below 2 / (second derivative).
Seed: A number that initializes a random number generator so its random draws repeat exactly on every run. Fixing a seed makes things like shuffles and random starting weights reproducible.
Shape: An array's size along each axis, such as (20, 2) for 20 cafés and 2 features. `.reshape(-1, 1)` turns a flat array into a column, with -1 meaning 'work out this size yourself'.
Sigmoid: σ(z) = 1/(1 + e⁻ᶻ), which squashes any score into the range 0 to 1, so it can turn a score into a probability. Its slope is σ(1 - σ).
Slope: How far up a line goes for each step to the right. It is positive if the line rises, negative if it falls, and 0 if it is flat.
Squared error: The loss (prediction - actual)² for one example. Squaring makes every miss positive and punishes big misses most.
Standard deviation: A measure of how spread out values are around their mean. It is used to standardize inputs.
Standardize: Subtract the mean and divide by the standard deviation so inputs are on a similar scale centered on 0. This reshapes the loss into a rounder bowl that gradient descent crosses easily. New data must use the training mean and std.
Step (iteration): One update of the parameters in gradient descent.
Sum rule: The derivative of a sum is the sum of the derivatives: d/dx (f + g) = f' + g'. You differentiate a sum one piece at a time.
Summation (Σ): The capital Greek sigma means 'add up over every example'. In MSE, Σ adds the squared errors of all n examples.

T

Tangent line: The straight line that just touches a curve at one point. Its slope is the derivative there.
Tanh: An S-shaped activation function like the sigmoid but running from -1 to 1. Its slope is 1 - tanh(z)².
Tensor: PyTorch's version of a NumPy array, which can record operations for automatic differentiation.
Threshold: The cut-off, such as 0.5, that turns a probability into a yes or a no. Lowering it to 0.3 predicts more cafés will reorder.
Training: Finding parameter values that make a model's predictions less wrong, by repeatedly nudging each parameter the way that lowers the loss.
Transpose: A matrix with its rows and columns swapped. Multiplying by `X.T` adds up each weight's input times the error across all examples.

V

Vector: An ordered list of numbers, stored as a 1D NumPy array. The gradient is a vector.

W

Weight: A parameter that multiplies an input, setting how much that input pushes toward the output. In Alex's staffing model, w multiplies website visits.

#

Δ (delta): The Greek capital letter meaning 'the change in'. Δp = 16 - 14 = 2 is the change in price, always computed as new minus old.