Glossary

93 terms from ML Beyond the Basics, in plain words, each with the chapter that teaches it.

Search the glossary

A

Adversarial validation: A drift check where you train a classifier to tell old rows (label 0) from new rows (label 1), such as Kettle & Crane's history against September. A cross-validated ROC AUC near 0.5 means the two look alike, and a higher AUC means they differ.
ALE plot (accumulated local effects): A plot of a feature's effect that compares only nearby, realistic values instead of inventing impossible rows. It is an alternative to partial dependence when features are correlated.
Average of ranks: A way to combine models by turning each model's scores into ranks and averaging the ranks, not the probabilities. It gives an ordering but not probabilities, and it suits models whose scores are on different scales.
Average precision (AP): A single number that summarizes the precision-recall curve and focuses on the rare class. Random ordering scores about the cancel rate (roughly 0.09 here), so that is its floor.

B

Base value: The model's average output, which is the starting point for SHAP. The base value plus a row's SHAP values equals that row's prediction in log-odds.
BaseEstimator / TransformerMixin: scikit-learn classes you inherit from when writing a custom step. BaseEstimator gives `get_params`/`set_params` so `clone` can copy the step, and TransformerMixin gives `fit_transform`.
Baseline: The simple reference model every improvement must beat. Here it is the ML Basics logistic regression, scored honestly by time.
Beeswarm plot: A SHAP chart with one dot per subscriber per feature, placed by SHAP value and colored by the feature's value. It shows both which features matter and in which direction they push.
Break-even threshold: The risk above which an action pays for itself: a $10 call that saves a $36 subscriber pays when risk is above 10 ÷ 36 ≈ 0.278. Priya's call capacity of 150 a month can be tighter than this line.
Brier score: The mean squared difference between a predicted probability and the actual 0/1 outcome, where lower is better. It rewards well-calibrated probabilities.

C

Calibrated: A model is calibrated when its probabilities mean what they say: of the subscribers it scores at 20%, about 20% really cancel. This matters whenever the number itself is used, for planning or for money decisions.
CalibratedClassifierCV: A scikit-learn class that learns a mapping from a model's scores to calibrated probabilities (sigmoid, isotonic or temperature). Pair it with `FrozenEstimator` to calibrate a model that is already trained.
Callback: A function a library calls at set points during training, such as after each tree. LightGBM's `early_stopping` and `log_evaluation` are callbacks.
Categorical (pd.Categorical): A pandas column type with a fixed list of allowed values. It makes every file you load encode the column the same way, and lets LightGBM treat it as a category.
Class weight: A multiplier that makes mistakes on one class count more during training. `class_weight="balanced"` makes the rare cancelers count as much in total as the common stayers, but it inflates the predicted probabilities; LightGBM and XGBoost also offer `scale_pos_weight`, a multiplier for the positive class.
clone: A scikit-learn function that makes a fresh, unfitted copy of a model or step with the same settings. Cross-validation uses it so no fitted state carries from one fold to the next.
Concept drift: A change in the link between inputs and outcome. For example, after a price rise, the same skip pattern may signal more cancel risk than before. It shows only once new labels arrive.
Cross fitting: Encoding each training row using values learned only from the other folds, so no row sees its own label. scikit-learn's `TargetEncoder` does this inside `fit_transform`.
Custom transformer: Your own pipeline step with `fit` and `transform` methods that follows scikit-learn's conventions. It lets learned steps be fitted on training rows only, fold by fold.

D

Data drift: A change in the distribution of the inputs over time, such as more late deliveries or a new carrier for rural orders. You can see it the day new data arrives, without labels.
Development data: All data except the test months (September to June here). You use it for training, validation and every modeling choice, so the test stays untouched.
dtype and parse_dates (pd.read_csv): Arguments to `pd.read_csv`: `dtype` forces a column type (such as text, so zip prefix `096` keeps its zero), and `parse_dates` converts columns into real dates.

E

Early stopping: Stopping training once a held-out score hasn't improved for a set number of rounds, keeping the best round. It chooses the number of trees for you and needs a held-out set separate from the one you judge with.
Ensemble: A combination of several models' predictions. It helps most when the models differ in kind and make different mistakes.
Error analysis: Studying where and how a model is wrong, for example by segment, and by looking at the cancelers it missed. It shows what to improve.
Expanding-window folds: Time-based cross-validation folds that train on all earlier periods and validate on the next one, then move forward. `month_folds` does this on whole months, whereas scikit-learn's `TimeSeriesSplit` can cut a month in half.

F

Feature: An input column that a model learns from. Useful features are often built from business knowledge, such as dates turned into durations, ratios of a count to what was expected, and summaries of a customer's own recent history.
FrozenEstimator: A scikit-learn wrapper that marks an already-trained model as fixed, so `CalibratedClassifierCV` uses it as is and learns only the calibration. It replaces the removed `cv="prefit"` option.
FunctionTransformer: A scikit-learn class that wraps an ordinary function as a pipeline step. It is used for stateless rules, like filling 'nothing happened' gaps. `feature_names_out="one-to-one"` tells scikit-learn the columns are unchanged.

G

Gradient boosting: A method that builds trees one after another, each fitting the errors that remain and scaled down by the learning rate. LightGBM, XGBoost and `HistGradientBoostingClassifier` are implementations.
GroupKFold: A cross-validation splitter that never puts the same group (here, the same customer) in two folds. It keeps each customer's months together.
GroupMedianImputer: A custom stateful scikit-learn step that fills a missing value with the median of its group, such as `avg_rating` by `roast`. It learns the medians in `fit` from training rows only, then applies them in `transform`.

H

Held-out set: Rows kept out of fitting so they can judge or steer a model. Use one set for decisions such as early stopping and a different one for the final verdict, or the verdict is optimistic.
High-cardinality: Describes a categorical column with many distinct values, like `zip_code` with 2,598 values. One-hot encoding would add thousands of nearly empty columns.
HistGradientBoostingClassifier: scikit-learn's histogram-based gradient boosting classifier, modeled on LightGBM. It handles missing values natively and needs no scaling, so it works as a quick yardstick for testing new features.
Hyperparameter: A setting you choose before training, such as the number of trees or leaf size, rather than something the model learns. Optuna searches for good ones.

I

ICE curve: A curve per subscriber showing how that person's prediction changes as one feature varies. It reveals whether an effect is the same for everyone.
if __name__ == "__main__":: A Python guard that runs the code under it only when the file is run directly, not when another script imports from it.
Imbalanced: One class is much rarer than the other, as with the roughly 9% of subscribers who cancel. Accuracy becomes misleading: 'nobody cancels' scores 90.7%.
Isotonic calibration: Calibration with a flexible step function that only goes up, so it can fix any shape of distortion. It needs thousands of positives and its flat steps create tied scores, which hurt ranking here.

J

joblib: A Python library commonly used to save a fitted model to a file and load it later. A saved model is best stored with its column list and the scikit-learn version, since it may not work under a different version.

K

kc.py: A shared Python module of Kettle & Crane helper functions, such as `load`, `split_by_time`, `month_folds`, `add_features` and `basics_model`. It collects the loading, splitting and feature code that the analyses reuse.

L

Leakage from the future: A feature that uses information from after the snapshot date, such as counting a customer's total rows in the file. It makes validation scores look great but fails on real new data, and a time split can't catch it.
learning_rate: How much each new tree counts in boosting. Smaller values need more trees.
Lift: A segment's AP divided by its cancel rate, meaning how many times better the model ranks people there than random ordering would.
LightGBM: A fast, widely used gradient boosting library that takes pandas DataFrames directly and treats `category` columns as categories. `LGBMClassifier` is its scikit-learn style interface.
LightGBM tree parameters: The main LightGBM controls: `num_leaves` (leaves per tree, i.e. complexity), `min_child_samples` (fewest rows in a leaf), `subsample` with `subsample_freq` (random share of rows per tree; does nothing unless `subsample_freq` is at least 1), `colsample_bytree` (random share of columns per tree) and `reg_lambda`/`reg_alpha` (L2/L1 penalties on leaf values).
Log loss: A probability score that heavily punishes confident mistakes. LightGBM minimizes it, which is why it is usually well calibrated, and early stopping watches it by default.
Log-odds: log(p / (1 − p)), the scale on which logistic regression and boosted trees add up their contributions. SHAP values for LightGBM are in log-odds, and 1 / (1 + e^−x) turns them back into probabilities.

M

Meta-model: In stacking, the small model that learns how much to trust each base model, using their predictions as its features.
Model card: A short document describing a model's purpose, testing, results and limits, plus how to monitor it. Kettle & Crane's covers calibration, weak segments and a retraining trigger.
Module: A `.py` file you can import from other scripts, such as `kc.py` or `steps.py`.
Monotone constraint: A rule that a prediction may only rise (or only fall) as a feature rises. LightGBM takes one number per column: 1 for never decreases, −1 for never increases, 0 for none.

N

n_estimators: The maximum number of trees in a boosted model. Set it generously and let early stopping choose, then fix it when retraining on everything.
NaT: pandas' marker for a missing date, short for 'not a time'. It shows up where `last_ticket_date` is missing, for people who never had a support ticket.
Nested validation: Putting the whole tuning search inside an outer loop. For each outer month you tune using only earlier months, then score the winner on that unseen month, exposing how optimistic the best tuning score is.

O

One-hot encoding: Turning a categorical column into one 0/1 column per value. It works well for a few values (like `plan`) but adds thousands of nearly empty columns for high-cardinality columns.
Optuna: A free library that searches for good hyperparameters adaptively, learning from earlier tries. You write an objective function and a study runs many trials.
Out-of-fold predictions: Predictions for rows made by models that didn't train on those rows. Stacking needs them, because in-sample predictions would teach the meta-model to trust whichever model memorized the training data.
Overfitting: When a model learns details of its training rows that don't carry over to new data, so training scores keep improving while validation scores stall or fall. Early stopping, simpler trees and penalties limit it.
Oversampling, undersampling: Resampling methods: oversampling repeats rare-class rows, and undersampling drops common-class rows. Do either on training rows only, inside each fold.

P

Paired bootstrap: Resample whole customers with replacement many times, score both models on each resample, and look at the spread of their difference. Pairing removes shared luck, and an interval that includes zero means the models are effectively tied.
Partial dependence: The average prediction as one feature is set to each value on a grid for every row. It shows, for example, how risk changes with `skip_share`, but it can invent impossible rows when features are correlated.
Pipeline: A scikit-learn object that chains steps and fits them as one model. Anything learned from other rows (medians, scaling, encodings) belongs inside it so each fold learns from training rows only.
Pipeline drift: A change in the data caused by how it was collected or built, not by the world. For example, `days_since_ticket` looked drifted because the file starts in September 2025.
Platt scaling (sigmoid): Calibration that fits a small logistic regression mapping a model's scores to probabilities. With only two numbers to learn it needs little data and keeps the ranking unchanged.
Population stability index (PSI): A measure of how far one column's distribution has moved between old and new data. Below 0.1 is stable, 0.1 to 0.25 is worth a look, and above 0.25 is a real shift.
Precision at k / recall at k: Precision and recall for only the first k names on the list, such as Priya's 150 calls a month. They judge a model at the capacity you really have.
Precision-recall curve: A plot of precision against recall as the threshold moves down a list sorted by risk. Its left end, the top of the list, is where Priya acts.

R

Reason code: A short label saying why a score is high, built from a person's largest feature contributions. Priya can use them to know what to say on a call.
Recalibration: Refitting the calibration step (for example sigmoid) on recent labeled months when predicted totals drift from actual ones. It is cheap and needs no retraining of the model.
Reliability diagram: A chart of average predicted probability against the observed rate, by bin of prediction. A well-calibrated model follows the diagonal.
Resampling: Changing the training rows instead of their weights to handle imbalance, by oversampling or undersampling. Resample only training rows, and never validation or test rows.

S

Sampler, trial, study: Optuna's vocabulary: the sampler picks which settings to try next, a trial is one tried set of settings, and a study is the whole search. The objective is the function that takes a trial and returns a score.
Segment: A group the business already thinks in, such as a tenure band, plan, channel or area. Checking performance by segment reveals weaknesses that one overall number hides.
SHAP values: Each feature's contribution to one prediction, from game theory's fair credit sharing. They add up (with the base value) to the model's output in log-odds, and `TreeExplainer` computes them quickly for tree models.
Shift, rolling and expanding (pandas history tools): pandas tools for features from a customer's own past. `shift(n)` takes the value n rows earlier, `rolling(3).sum()` totals this row and the two before, and `expanding().mean()` averages all rows so far. All look only backward, so they don't leak.
Silent cancelers: Subscribers who cancel without showing any warning sign in the data, looking almost like loyal customers. No tuning finds them; only new data, such as payment failures, might.
Smoothing: Pulling a small group's average toward the overall average, more strongly for small groups. It stops a zip code with one canceler from getting a rate of 1.0.
SMOTE: An oversampling method that invents new rare-class rows between existing ones. It comes from the separate imbalanced-learn library, which offers resampling tools for imbalanced data.
Snapshot: One customer as they looked at the end of one month. The `canceled` label says whether they canceled in the following month.
Sparse matrix: A matrix that stores only its nonzero values, which saves memory when most entries are 0, as in one-hot encoded columns. It can be converted to an ordinary dense array with `.toarray()`.
Stacking: Combining models by training a meta-model on out-of-fold predictions of the base models. With time data you build those predictions yourself with time folds.
Stateful, stateless step: A stateless step applies a fixed rule and learns nothing, like filling gaps with 400. A stateful step learns something in `fit` (such as medians) and uses it in `transform`.
Stress test: Feeding a model deliberately changed inputs to check it reacts sensibly, such as giving everyone one more late delivery and checking that risk never falls.

T

Target encoding: Replacing a category with the average outcome for it, such as each zip code's cancel rate. It must be smoothed and cross-fitted, or each row's own label leaks into its encoding.
Temperature scaling: A calibration method in scikit-learn 1.9 that divides the model's log-odds by one learned number. It mainly helps models with more than two classes.
Ties go to the simpler model: A decision rule: a more complex model replaces a simpler one only if the paired bootstrap interval for the gain is entirely above zero. Otherwise you keep the simpler model.
Time-based split: Training on earlier periods and validating and testing on later ones. It avoids a random split's problems of repeated people, future leaking into the past, and a changing world.
Timedelta: A length of time, the result of subtracting two dates. `.dt.days` turns it into a number of days.
TPE (Tree-structured Parzen Estimator): Optuna's default sampler, which looks at which settings scored well so far and tries more like them while still exploring. Optuna 5.0 uses the multivariate version by default.

V

Value at risk: Risk multiplied by monthly spend, used to rank subscribers by dollars at stake instead of by probability alone. It puts more valuable, long-standing subscribers on the call list.

X

XGBoost: Another popular gradient boosting library with many of the same parameter names as LightGBM. Which library wins varies by dataset, with usually small differences.