Glossary

111 terms from Probability for ML, in plain words, each with the chapter that teaches it.

Search the glossary

A

Accuracy: The share of predictions where the model's top class was the true class. Priya's classifier got 41 of 48 right, so its accuracy was 0.854. It ignores how confident the model was.
Addition rule: To get P(A or B), add P(A) and P(B), then subtract P(A and B), because the cases where both happen were counted twice. For mug (0.3) and express (0.2) it gives 0.44.
argmax / argmin (np.argmax, np.argmin): NumPy functions that return the position (index) of the largest or smallest value in an array, not the value itself. They are used to pick the best candidate on a grid of parameter values or a model's predicted class from its scores.
Array: NumPy's version of a list of numbers that you can do math on all at once. Comparing or adding two arrays works element by element.

B

Base rate: How common something is before any evidence. At Kettle & Crane only 0.5% of orders are fraud. Ignoring that low base rate makes a fraud flag look far more trustworthy than it is.
Bayes' rule: A formula that turns P(evidence | cause) into P(cause | evidence): P(H | E) = P(E | H) × P(H) ÷ P(E). It tells Sam how likely a flagged order is to be fraud.
Bernoulli: A random variable for a single yes/no trial that is 1 with probability p and 0 otherwise. Adding n independent Bernoullis gives a binomial.
Bin: One of the ranges that values or predictions are grouped into. In a reliability table, predictions are grouped into bins such as 0.8 to 1.0.
Binomial: The distribution of the number of yeses in n independent trials that all share the same probability p. The number of mug orders in Kettle & Crane's 20 daily orders is Binomial(20, 0.3).
Bits, nats: Units for measuring entropy. Using base-2 logs gives bits, and using natural logs gives nats, which ML libraries use.
Boolean array: An array of True and False values, usually made by comparing an array with something, like `rng.random(10) < 0.3`. NumPy counts True as 1, so `.sum()` counts them and `.mean()` gives their share.
Boolean indexing: Putting a boolean array inside square brackets to keep only the elements where it is True. For example, `code[new]` gives the code column for new customers only, which is a way to compute conditional probabilities in code.
Bootstrap: Making new datasets by picking from your real data at random with replacement, then repeating your calculation on each. The spread of the results shows how much they depend on which examples you happened to have.

C

Calibrated: A model is calibrated when its predicted probabilities match how often things happen: among cases where it says 0.7, about 70% come true. This is separate from accuracy.
Central limit theorem (CLT): Averages of many independent values are close to a normal (bell) curve, whatever shape the individual values have. That's why mean ± 1.96 SE works as a 95% range for an average.
Classifier: A model that puts things into categories, called classes, such as spam or ok, or complaint, question or praise.
Complement: The event "A doesn't happen", with probability 1 − P(A). It makes "at least one" questions easy: P(at least one mug in 5 orders) = 1 − 0.7⁵.
Conditional distribution, marginal distribution: A marginal distribution describes one variable on its own, found by adding over the other variable in a joint table. A conditional distribution is a row (or column) of the joint table divided by its total.
Conditional probability: The probability of A once you know B happened, P(A | B) = P(A and B) ÷ P(B). Among Sam's new customers 48.6% used a discount code, compared with 28.4% overall.
Confidence: For a classifier, the probability it gives to its predicted class. Calibrated means that among predictions made with 70% confidence, about 70% are right.
Confusion matrix: A table of counts for each true class against each predicted class. In Priya's project, the complaint row [14, 0, 2] means 14 complaints were predicted as complaints and 2 as praise.
Continuous: A random variable that can take any value in a range, such as a bag's weight, instead of separate listable values. Its probabilities are described by a density.
Correlation: Covariance divided by both standard deviations, so it has no units and always lies between −1 and 1. It measures straight-line co-movement.
Covariance: A measure of whether two variables move together: positive if they tend to be high or low at the same time, negative if one rises as the other falls. Zero covariance does not prove independence.
Covariance matrix: A square table holding every variance (on the diagonal) and every covariance between pairs of variables. For a weighted total with weights w, the variance is `w @ C @ w`.
Cross-entropy (log loss): The average of −log(probability the model gave to the true class). It punishes confident mistakes heavily and is the same as the negative log-likelihood, so training on it is maximum likelihood.
Cumulative distribution function (cdf): The probability that a random variable is at most x, P(X ≤ x). In scipy.stats it's the `.cdf(x)` method.

D

Dict comprehension: A one-line way to build a Python dict, such as `{c: 0 for c in classes}`. It's the dict version of a list comprehension.
Discrete: A random variable that takes separate values you could list, such as the number of bags in an order.
Distribution: A description of how a random variable's probability is spread over its values. Named families like binomial and normal come with ready-made code in scipy.stats.
Dot product: Multiplying matching numbers in two lists and adding up the results, written `np.dot(a, b)` in NumPy. It computes weighted sums such as expected values and the law of total probability.

E

ECE (expected calibration error): One number that summarizes a reliability table: the gap between predicted and observed rate in each bin, weighted by the share of predictions in that bin. Lower is better.
Entropy: The average surprise of a distribution, −Σ p log p. It measures how unpredictable the distribution is: a fair coin has 1 bit, a certain outcome has 0. It's also the lowest cross-entropy any model can reach.
Event: A set of outcomes you care about, such as "the order includes a speckled blue mug". Its probability is written P(event).
Expected value (mean): The long-run average of a random variable: multiply each value by its probability and add. For bags per order it is 1.75, even though no order has 1.75 bags.
Exponential: The distribution of waiting times between events that happen at a steady rate λ, with mean 1/λ. The gap between Kettle & Crane's online orders is exponential.

F

Fancy indexing: Using an array of positions as an index, so `late_rate[which]` looks up every parcel's own carrier rate at once.

G

Gambler's fallacy: The mistaken belief that chance evens out in the short run, so that after five orders with no mug the next is more likely to have one. The law of large numbers only dilutes early luck.
Gradient descent: An optimization method that uses calculus to find which way is downhill on a loss such as the negative log-likelihood, then adjusts parameters step by step to lower it. It is how models with too many parameters for a grid search are trained.
Grid: Evenly spaced candidate values to try, such as the 99 values of p made by `np.linspace(0.01, 0.99, 99)`. You compute the likelihood at each and pick the best.

H

Held-out data: Data kept aside and not used for fitting a model, so it can give an honest check of how the model performs. For example, a temperature can be fitted on one half of the data and calibration checked on the other half.
Hypothesis, evidence: In Bayes' rule, the hypothesis H is what you want to know (the order is fraud) and the evidence E is what you observed (the order was flagged). P(E) is how often the evidence happens at all.

I

Independent: Two events are independent when knowing one happened doesn't change the other's probability, P(A | B) = P(A). Only then can you multiply P(A) × P(B) for "and".
Indicator: A random variable that is 1 when an event happens and 0 otherwise. Its expected value equals the event's probability, which is why the mean of a boolean array estimates a probability.

J

Joint distribution: The probability of each combination of values of two or more variables, such as bags and mugs in one order. Its row and column totals give the marginal distributions.

K

KL divergence: Cross-entropy minus entropy: a measure of how far a predicted distribution is from the true one. Training can shrink it but never the entropy itself.

L

Label, training data: A label is an example's known correct class. Training data is the set of labeled examples a model learns from, such as Priya's eight labeled messages.
Laplace smoothing: Adding 1 to every word count so that no word ever gets probability 0 in a class. Without it, one unseen word would zero out the whole product.
Law of large numbers: The average of many independent draws gets closer and closer to the expected value. It is why simulation works, but it does not mean luck evens out in the short run.
Law of total probability: The overall probability of something is a sum over groups of each group's probability times the group's share. Alex's overall late rate of 0.034 weights each carrier's late rate by its share of parcels.
Leave-one-out: A way to test honestly with little data: for each example, train on all the others and predict the one held back, then repeat for every example (48 times for Priya's 48 messages).
Library: Code you install separately and import, such as NumPy and SciPy.
Likelihood: The probability of the observed data as a function of a parameter value, such as how probable 3 mugs in 10 orders is for each candidate p. It scores how well a value explains the data and is not the probability that the value is right.
List comprehension: A one-line Python way to build a list: `[f(x) for x in items]` runs the expression once per item and collects the results.
Log-likelihood: The sum of the logs of the probabilities of the data. It avoids underflow and has its maximum at the same parameter value as the likelihood.
Log-odds: The score whose sigmoid gives a probability; it is the same thing as the logit. A model that doubles each case's log-odds becomes overconfident.
Logarithm (log): A function that turns products into sums: log(a × b) = log a + log b. Taking natural logs lets many small probabilities be added instead of multiplied, which avoids underflow.
Logistic regression: A basic two-class model that gives each case a score by weighting and adding its features, then turns the score into a probability with the sigmoid.
Logit (score): A model's raw score for a class before softmax: any number, positive or negative, where bigger means more likely.

M

Margin of error: How far a measured proportion or average is likely to be from the truth, based on its standard error. For a proportion the SE is √(p(1 − p) ÷ n), and about 1.96 SEs gives a 95% range.
Matrix multiplication (@): The Python `@` operator, which multiplies matrices or vectors. With a vector on each side, `w @ C @ w` gives a single number, the variance of a weighted total.
Maximum likelihood estimate (MLE): The parameter value that makes the observed data most probable. For a probability it is the observed share, and for a Poisson rate it is the mean count.
Mean squared error: The average squared miss between predictions and actual numbers. Minimizing it is the same as maximizing the likelihood of a normal model around the predictions.
Memoryless: A property of the exponential: the time you have already waited doesn't change the remaining wait. A shop that has waited 30 minutes for an order isn't "owed" one.
Multiplication rule: P(A and B) = P(B) × P(A | B) always holds. It shrinks to P(A) × P(B) only for independent events.
Multivariate normal: The bell curve for several variables together, set by a vector of means and a covariance matrix. Jo uses it to model espresso and decaf demand moving together.
Mutually exclusive: Events that can't happen together, like a parcel shipped by UPS and by FedEx. Their overlap is 0, so you can simply add their probabilities, and they are never independent.

N

n choose k (binomial coefficient): C(n, k) counts the ways to pick which k of n trials are the yeses. It appears in the binomial formula and is computed by `math.comb(n, k)`.
Naive Bayes: A classifier that scores each class as its prior times the product of P(word | class), treating words as independent clues within a class. It trains by counting words, but tends to be overconfident.
Natural frequencies: Counts of imagined cases, like 100,000 orders with 500 frauds, used instead of probabilities to make Bayes' rule easy to see and check.
Negative log-likelihood (NLL): The log-likelihood with its sign flipped, so it is positive and smaller is better. It is the loss that models minimize when they train.
Normal: The symmetric bell curve, set by its mean and standard deviation. The filling machine's bag weights are normal with mean 12.2 oz and SD 0.15 oz.
np.where: `np.where(condition, x, y)` builds an array that takes each element from x where the condition is True and from y where it is False. It can be used to give different groups different probabilities.
NumPy: A Python library that stores numbers in arrays, does fast math on them, and generates random numbers. It is conventionally imported as `np`.

O

Outcome: One thing that can actually happen in a chancy situation. All possible outcomes make up the sample space.
Outer product (np.outer): `np.outer(a, b)` builds a table of every element of a times every element of b. Applied to two marginals, it gives the joint table you would see if the variables were independent.
Overconfident: A model is overconfident when its probabilities are more extreme (closer to 0 and 1) than how often it is right. Naive Bayes tends to be overconfident; an underconfident model is the opposite.
Overfitting: Fitting the quirks of the data you have, so results look better on it than they will on new data. Fitting the temperature and checking it on the same 48 messages is an example.

P

Parameter: A number that picks one member of a distribution family. Binomial(20, 0.3) and Binomial(50, 0.1) are different members of the binomial family. In ML, training adjusts a model's parameters.
Percentile: The value with a given share of the data below it, such as the 95th percentile. In scipy.stats it's `.ppf(q)`, and for data it's `np.percentile`.
pip: Python's package installer, used to install NumPy and SciPy into your virtual environment.
PMF (probability mass function): A table giving each possible value of a discrete random variable with its probability. The probabilities add up to 1.
Poisson: The distribution of the count of events in a period when they arrive independently at a steady average rate λ. Its mean and variance are both λ, as with Kettle & Crane's orders per hour.
Prior, posterior: The prior is your belief before seeing the evidence and the posterior is your belief after. Today's posterior becomes tomorrow's prior when new evidence arrives.
Probability: A number from 0 to 1 saying how often something happens in the long run. 0 means never, 1 means always, and 0.3 means 30% of the time.
Probability density function (PDF): The curve describing a continuous distribution, whose area over a range is the probability of landing there. The curve's height, the density, is not a probability and can exceed 1.

R

Random number generator: An object that produces a stream of numbers that look random, starting from a number called the seed. The same seed always produces the same stream, so results can be repeated exactly.
Random variable: A number whose value depends on chance, usually named with a capital letter like X, such as the number of bags in an online order.
Reliability table: A table comparing predicted probability with the observed frequency, bin by bin. Drawn as a chart it is a reliability diagram. Always print the counts beside the rates.

S

Sample space: The set of all possible outcomes of a chancy situation, such as the 36 ways two dice can land.
SciPy (scipy.stats): A Python library whose `scipy.stats` module holds ready-made probability distributions and whose `scipy.special` holds functions like softmax and the sigmoid.
Seed: The starting number given to a random number generator. The same seed always gives the same stream of numbers, making simulations repeatable.
Set: A Python collection that keeps each item only once. It is handy for building a vocabulary of distinct words, adding words from each message with `.update`.
Shape: An array's size in each direction, such as (100000, 20) for 100,000 rows and 20 columns.
Sigmoid: The function 1 ÷ (1 + e^(−s)) that turns a score into a probability between 0 and 1. It is softmax for two classes and is computed by `scipy.special.expit`.
Simulation: Estimating a probability by running many random trials in code and counting how often the event happens. It can be used to check a formula or library result against a direct count.
Skewness: How lopsided a distribution is: 0 for symmetric, positive for a long right tail. Exponential gaps are strongly skewed, but averages of many of them become nearly symmetric.
Softmax: A function that turns a list of scores into probabilities: exponentiate each score, then divide by the total. Subtracting the largest score first avoids overflow without changing the result.
Standard deviation, variance: Measures of spread. Variance is the expected squared distance from the mean, and the SD is its square root, in the original units. Variances of independent variables add but SDs don't.
Standard error (SE): The standard deviation of an average, σ ÷ √n; for a proportion it is √(p(1 − p) ÷ n). It tells you how far a measured accuracy or average might be from the truth.
Standard normal: The normal distribution with mean 0 and SD 1, the default for `stats.norm`. A z value counts how many SDs a point is from the mean.
Surprise: −log p: how unexpected an outcome of probability p is. Likely outcomes surprise little and rare ones a lot; entropy is the average surprise.
Survival function (sf): P(X > x), which is 1 − cdf. Note `.sf(7)` gives P(X ≥ 8) for whole-number variables, because it means "more than 7".

T

Temperature: A number that scores are divided by before softmax or sigmoid. T above 1 softens probabilities and T below 1 sharpens them, without changing the ranking of classes.
Temperature scaling: Fixing an overconfident model by dividing its scores by a temperature T found by maximum likelihood on held-out data. It improves calibration without changing accuracy.
Test set: Labeled examples kept separate from training and used to measure how well a model works. Because accuracy on it is an average, it has a standard error, and small test sets can't separate similar models.
Tuple: A fixed group of values in Python, such as `(label, words)`, that can be unpacked into separate names.

U

Underflow: When a number is too small for the computer to store and becomes 0.0. Multiplying thousands of small probabilities underflows, so their logs are summed instead; overflow is the opposite problem of numbers too big to store, such as e¹⁰⁰⁰.
Uniform: A distribution where every value in a range is equally likely, like UPS arriving at a random time between 2 and 4 p.m. In scipy.stats, `scale` is the width, not the end.

V

Virtual environment: A private set of installed packages for one project, so its libraries don't interfere with other projects. You create it with `python3 -m venv .venv` and activate it before using pip.
Vocabulary: All the distinct words a text model saw in training. Words outside it are skipped when scoring.