Glossary

102 terms from Statistical Inference and A/B Testing, in plain words, each with the chapter that teaches it.

Search the glossary

A

A/A test: A test that shows the same version to both groups. Any difference is pure chance, so it checks that the setup (splitting, counting) works.
A/B test: A randomized experiment comparing two versions: A, usually the current one (the control), and B, the change (the treatment or variant). Sam's coupon test was one.
Absolute difference: The gap between two rates in percentage points, such as 4.10% − 3.60% = 0.5 points for the coupon.
Agresti-Coull interval: A confidence interval for a proportion that stays sensible with small counts by adding about two yeses and two noes to the data before calculating. The simpler 'plus four' version (add 2 yeses and 2 noes, then use the usual formula) turns Alex's 2 broken mugs in 40 into an interval of 0.6% to 17.6% instead of a impossible negative lower bound.
Alpha (α): The threshold set in advance for calling a result significant, usually 0.05. It is also the share left outside a confidence interval, and the Type I error rate you accept.
Alternative hypothesis: What you conclude if you reject the null hypothesis: that there is a difference, such as the courier's true on-time rate not being 90%.
Analysis ToolPak: A free add-in for Excel on Windows and Mac that prints ready-made statistical tests such as t-tests. It isn't needed because T.TEST gives the same p-values.

B

Baseline: The control's current rate (or, for means, its standard deviation), taken from past data. It is one of the four inputs to a sample size calculation.
Bayesian method: An approach that combines the data with prior assumptions to give a probability that B is better than A. It is not a p-value and depends on those assumptions.
BINOM.INV: A spreadsheet function that, given RAND() as its last argument, picks a random count of yeses from n tries with probability p. It lets one cell simulate a whole group of visitors.
Binomial distribution: The distribution of the number of yeses among a fixed number of independent tries that each have the same chance, such as orders from 3,000 visitors.
Bonferroni correction: A cautious fix for multiple comparisons: multiply each p-value by the number of comparisons k, or equivalently use alpha ÷ k. Jo's 12 segments turned p = 0.03 into 0.36.
Bootstrap: A way to estimate noise by treating your sample as a stand-in for the population and drawing many new samples from it with replacement. Jo used it on her 20 bag weights.

C

Central limit theorem: The result that means and proportions from reasonably large samples are roughly normally distributed, whatever the shape of the individual data.
Change aversion: A new version doing worse at first because regular customers are used to the old one and can't find things. It fades as they adjust.
Chi-square statistic: The sum of (observed − expected)² ÷ expected over every cell of a table of counts. A larger value means the counts are further from what chance would give.
Chi-square test: A test of whether the rows and columns of a table of counts are related, or whether counts differ from expected by more than chance. Sam used it on three newsletter subject lines.
CHISQ.DIST.RT: A spreadsheet function giving the right-tail p-value for a chi-square statistic with given degrees of freedom.
CHISQ.TEST: A spreadsheet function that takes observed and expected counts and returns the p-value of a chi-square test. It is also used for the goodness-of-fit check on a planned split.
Cohen's d: A unit-free effect size: the difference in means divided by the pooled standard deviation. It is used mainly for planning, and the 0.2/0.5/0.8 labels are rough.
Confidence interval: A range of believable values for a parameter, built from a sample. '95% confident' means the method captures the truth in about 95% of repeated samples, not that 95% of individual values fall inside.
Confidence level: The share of samples for which the interval-building method would capture the true value, such as 90%, 95% or 99%. A higher confidence level gives a wider interval, and 95% (alpha 0.05) is the usual convention.
CONFIDENCE.NORM: A spreadsheet function that returns the margin of error for a mean using the normal critical value 1.96 instead of a t value. It gives a slightly narrower margin that is too optimistic for small samples, so `CONFIDENCE.T` is preferred for means.
CONFIDENCE.T: A spreadsheet function that gives the margin of a t-interval in one step from alpha, the standard deviation and n.
Control: Version A of an A/B test, usually the current one, against which the change is compared.
Conversion rate: The share of visitors (or other units) who take the target action, such as placing an order. For the coupon test it is orders divided by visitors.
Critical value: The number of standard errors the margin of an interval spans: 1.96 for a large-sample 95% interval, or a t value for small samples.

D

Degrees of freedom: A number that sets the exact shape of a t or chi-square distribution. For one mean it is n − 1, and for a chi-square table (rows − 1) × (columns − 1).

E

Effect size: The size of a difference in meaningful units (dollars, minutes, points) or standardized. It tells you whether a real difference matters, which a p-value can't.
Estimate: Your best single guess of a parameter from the sample, such as the 0.5-point gap in Sam's coupon test.
Expected count: The count a table cell would have if rows and columns were unrelated, calculated as row total × column total ÷ grand total.

G

Guardrail metric: A metric the change must not make worse, even if the primary metric improves, with a written tolerance. Average order value is one for the checkout test.

H

Haybittle-Peto rule: A cautious rule for stopping early: stop only if p is below 0.001, and use 0.05 at the planned end.
Hypothesis: A statement that names the change, the metric and the reason, such as 'a one-page checkout will raise completion because shoppers abandon at the address step'. It is different from the statistical null hypothesis.
Hypothesis test: A check of whether chance alone could explain a result: state the null, measure the distance in standard errors, find the p-value, and decide.

I

Independent: Not affecting each other: one visitor's decision doesn't change another's. Binomial counts and t-intervals assume it.
INDEX: A spreadsheet function giving the k-th value in a range. With RANDBETWEEN it picks a random bag in the bootstrap.
Interference: When one unit's version affects another unit's outcome, such as cafés comparing notes on a new price list. It breaks the assumption behind randomized comparisons.

L

Lift (relative difference): The gap between two rates divided by the control's rate. The coupon's 0.5 points on a 3.6% base is a 13.9% lift.

M

Margin of error: The 'plus or minus' part of a confidence interval, equal to the critical value times the standard error.
Minimum detectable effect (MDE): The smallest true effect a test is sized to detect with the chosen power. Sam chose 5 points for the checkout test.
Minimum worthwhile effect: The smallest effect that would change your decision, usually from costs. For Priya's cards it was 2.5 points.
Multiple comparisons: Running many tests, metrics or segments at once, which raises the chance that at least one is a false positive.

N

Name box: The small box left of the formula bar in a spreadsheet, where you can type a range like A2:A3001 to select it.
Noise: Variation caused by chance, such as part of the 0.5-point gap in Sam's coupon test.
Non-inferiority: A decision rule where you don't need to prove the change is better, only that its interval rules out a loss bigger than an agreed margin.
NORM.S.DIST: A spreadsheet function giving the share of the standard normal curve below z when its second argument is TRUE. It turns a z-score into a p-value.
NORM.S.INV: A spreadsheet function that takes a share and returns the z-score with that share of the standard normal curve below it, e.g. 0.975 gives 1.96.
Normal distribution: The symmetric bell-shaped distribution that sample means and proportions roughly follow, by the central limit theorem.
Novelty effect: A new version doing better at first just because it is new and catches attention, an effect that fades.
Null hypothesis: The boring 'no effect' explanation a test assumes, such as the courier's true on-time rate really being 90%.

O

Observed count: The actual count in each cell of a table, which a chi-square test compares with the expected count.
One-sided, two-sided test: A two-sided test counts surprises in both directions; a one-sided test counts one direction and gives half the p-value. A/B tests should be two-sided unless one direction was fixed in advance.

P

p-value: If the null hypothesis were true, the chance of a result at least as far from the null as the one you saw. It is not the chance the null is true, and it says nothing about size.
Paired t-test: A t-test on the differences within matched pairs, like each taster's score for the new roast minus the current one. It removes differences between people and can be far more sensitive.
Parameter: A number that describes the whole population, such as a true conversion rate. You never see it, only estimate it.
Peeking: Checking a running test and stopping early because it looks significant. Daily peeking turned a 5% false positive rate into 27%.
Percentage points: The unit for the absolute gap between two percentages: going from 3.6% to 4.1% is +0.5 points, not +0.5%.
Pooled rate: Both groups' successes divided by both groups' sizes. It is the shared rate if the null is true, used for the z-test's standard error.
Pooled SD: A combined spread of two groups, √((s₁² + s₂²) ÷ 2) for equal sizes, used as the divisor in Cohen's d.
Population: The whole group you care about, such as every new visitor Kettle & Crane will get in coming months.
Power: The probability that a test detects an effect (p below alpha) when it's really there at a stated size; 80% is the usual target. It equals 1 minus the Type II error rate.
Practical significance: Whether a real difference is big enough to matter for a decision, judged against costs and the minimum worthwhile effect. A result can be statistically significant without it.
Primary metric: The one metric, defined exactly and named in advance, that decides the test. For the checkout test it is completion.
Proportion: A share expressed as a fraction, such as the share of visitors who convert. Its standard error is √(p(1 − p) ÷ n).

R

RAND: A spreadsheet function giving a random number between 0 and 1, recalculated every time the sheet updates. `=RAND()<0.0385` simulates one visitor who buys.
RANDBETWEEN: A spreadsheet function giving a random whole number between two limits, used with INDEX to pick a random bag in the bootstrap.
Randomization: Letting chance decide which version each unit gets. It makes groups alike except for the change, so a difference is either the change or chance.
Randomized experiment: A comparison where chance alone decides who gets which version, like Sam's coupon test. It is what makes the comparison fair.
Randomized unit: The thing that is randomly assigned to a version, such as a visitor, shopper or café. Each unit must see the same version every time, and metrics should be measured per unit.

S

Sample: The part of the population you have data on, such as September's 6,000 visitors.
Sample ratio mismatch (SRM): A split between versions further from the plan than chance allows, e.g. 2,350 vs 2,100 in a planned 50/50 test. It means something other than chance decided who was counted, so results can't be trusted.
Sample size: How many people (or other units) each group needs for a test to reach its chosen power. It is worked out before the test from alpha, power, the baseline and the MDE, and halving the MDE quadruples it.
Sampling distribution: How a statistic, such as a conversion rate, varies across many repeated samples. It is the picture of noise that inference is built on.
Secondary metric: A metric used for understanding a result, not for deciding it.
Segments: Slices of results by device, state, new versus returning and so on. Looking at many unplanned segments is a multiple comparisons trap.
Sequential testing: Methods that adjust the threshold for repeated looks, so you can stop early without inflating false positives.
Signal: A real effect underneath the noise.
Simpson's paradox: A comparison that reverses when data are split into groups. In a test, changing the traffic split midway can make B win each week yet lose when pooled.
Standard error (SE): The standard deviation of a sampling distribution: the typical size of chance noise in a statistic. Four times the data halves it.
Statistic: A number calculated from a sample, such as a measured 3.60% conversion rate.
Statistical inference: Methods for reasoning from sample statistics to population parameters, with an honest statement of uncertainty.
Statistically significant: A result with a p-value below alpha, so chance is an unlikely explanation. It says nothing about whether the effect is big enough to matter.
SUMIFS: A spreadsheet function that totals a range for rows meeting one or more conditions, used to total checkouts and orders by version, week or device.

T

t distribution: A bell curve like the normal but with fatter tails, used when the standard deviation is estimated from a small sample. Its shape depends on degrees of freedom.
t-test: A hypothesis test for means whose statistic is the distance from the null in standard errors, with the p-value taken from the t distribution.
T.DIST.2T: A spreadsheet function giving the two-sided p-value for a positive t statistic and degrees of freedom.
T.INV.2T: A spreadsheet function returning the critical t value for a given alpha and degrees of freedom, e.g. 2.093 for alpha 0.05 and df 19.
T.TEST: A spreadsheet function that returns the p-value of a t-test from two ranges, the number of tails (2) and the type: 1 paired, 2 equal spreads, 3 unequal spreads (Welch).
Test plan: A document written and shared before a test starts, covering the decision, hypothesis, primary metric, guardrails, sample size, checks and decision rule.
Test statistic: The distance of the data from the null, measured in standard errors, such as the z of −2.50 for the courier.
Tolerance: How much worse a guardrail metric may get before it fails, such as average order value falling by no more than $2.
Treatment: Version B of an A/B test, the change being tested, also called the variant.
Two-proportion z-test: The standard test for a difference between two rates: pool the rates, compute the SE under the null, divide the gap by it to get z, then find the p-value.
Type I error: Rejecting a null that is actually true, i.e. calling a do-nothing change significant. Alpha is the rate you accept.
Type II error: Failing to reject the null when a real effect exists, usually because the sample was too small.

U

Underpowered: Too small to detect the effects that matter. Sam's coupon test had only a 17% chance of detecting a real half-point lift.

V

VAR.S: A spreadsheet function for the sample variance, which is the standard deviation squared.

W

Welch's t-test: The two-sample t-test that doesn't assume equal spreads in the groups. It is the safe default for two separate groups.
Winner's curse: Significant results from small, underpowered tests overstate the true effect, because only lucky overestimates cross the line. It is a form of regression to the mean.
With replacement: Sampling where each picked item goes back and can be picked again, so a bootstrap resample may contain one bag several times and another not at all.

Z

z-score: How many standard errors (or standard deviations) a value is from a reference value. Sam's coupon gap had z of about 1.0.