Glossary

128 terms from ML in Production, in plain words, each with the chapter that teaches it.

Search the glossary

#

.dockerignore: A file listing folders, such as data and tests, that Docker should keep out of the build context so they don't end up in the image.

A

A/B test: An experiment where two groups get different versions, such as two models, for a few weeks and you compare business results. It answers questions like whether model B's calls save more subscribers per dollar.
Alert: An automatic message raised when a monitored number, such as PSI or AUC, crosses a set limit. At Kettle & Crane an alert fails the weekly job so Sam gets emailed.
Alias (model): A name such as `champion` or `challenger` that points to exactly one model version in the registry. Moving the alias is how you deploy or roll back.
API version: A path prefix like `/v1` that promises the endpoint's inputs and outputs won't change in a way that breaks callers. Breaking changes go into a new prefix such as `/v2`.
Artifact: A file produced by an MLflow run, most importantly the saved model itself.
AUC (ROC AUC): A score from 0 to 1 for how well a model ranks real positives (such as subscribers who cancel) above negatives; 0.5 is no better than guessing and 1 is a perfect ranking. In the Kettle & Crane project, the logistic regression churn model scores a test AUC of 0.777.

B

Background task: Work FastAPI runs after the response has been sent, such as writing the prediction log. The caller doesn't wait for it.
Bash script: A text file of shell commands run in order. The first line `#!/usr/bin/env bash` tells the system to run it with bash, and `chmod +x run_weeks.sh` makes it executable so `./run_weeks.sh` plays the twelve weeks.
Batch job: A script that runs on a schedule and processes a whole file at once, like Kettle & Crane's Monday scoring of about 1,000 renewals. It needs no server and can stop if the file looks wrong.
Batch prediction: Scoring many rows at once on a schedule, as opposed to answering one request at a time.
Behavioral test: A test of something a model must do whatever its score, such as a subscriber who skips more deliveries never being rated less likely to cancel.
Bin: A range of values grouped together so shares of rows can be compared. PSI uses the training data's deciles, ten equal-sized groups, as bins for numbers.
Break-even: The risk at which a call's expected value equals its cost: $10 ÷ $36 ≈ 0.278. Calling anyone above that risk pays off on average.
Business margin: The smallest gain worth the cost of switching models (the `min_gain` setting). It is about the effort of a review, new image and model card, not about noise.

C

CalibratedClassifierCV: scikit-learn tools for recalibration: `FrozenEstimator` marks a model as already trained, and `CalibratedClassifierCV` with `method="sigmoid"` fits only a logistic curve mapping scores to probabilities.
Calibration: How well predicted probabilities match real rates: a risk of 0.278 should mean a 27.8% chance. The break-even threshold is only safe when the model is calibrated.
Called subscriber: A flagged subscriber whom Priya actually called. They get weight 0 in retraining because the call may have changed whether they canceled.
Canary release: Sending a small, stable share of requests (say 10%) to a new model while the rest use the old one. It checks whether the new model breaks anything for real users.
Challenger: A candidate model version tested against the champion. Only a challenger that passes the promotion gate becomes champion.
Champion: The model version that serves, marked by the `champion` alias in the registry.
Concept drift: A change in how inputs relate to the outcome, such as a rival's cheaper box making familiar-looking subscribers leave. Inputs can look normal while the model gets worse, and only labels reveal it.
Config file: A file of settings the code reads, such as paths, hyperparameters and the threshold. Here it is `config.toml`, so settings change without editing code.
Container: A running instance of a Docker image: the service packaged with its packages and model so it behaves the same anywhere.
Continuous integration (CI): Running the tests automatically on every change, on a clean machine. A failing step blocks the change.
Control group: A small share of flagged subscribers deliberately not called, chosen by hashing their ID. Comparing them with called subscribers is how you learn whether calls work, and it keeps retraining honest.
Correlation: A number from −1 to 1 for how closely two columns of numbers move together; 1 means they rise and fall together. It can be used to compare a live model's scores with a shadow model's scores on the same requests.
Cron: A schedule written as minute, hour, day of month, month and weekday; `0 15 * * 1` means 15:00 every Monday, with `*` meaning "any".
curl: A command-line tool for sending HTTP requests, often used to test an API. Common options are `-s` to hide the progress meter, `-X POST` to set the method, `-H` to add a header and `-d` to send a request body.

D

Data drift: A change in the distribution of inputs, such as shorter tenure after a promotion. If the input-to-outcome link still holds, the model copes, so a drift alert is a reason to look, not to retrain.
Data minimization and retention: Keeping personal data to the least that monitoring and retraining need, and deleting logs on a schedule (say after 18 months). The ID in the prediction log still counts as personal data.
Data test: A check on the data rather than the code: are columns present, values in range, nothing impossible? `check_batch` runs these before scoring.
Data versioning: Knowing, and being able to recover, exactly which data a model was trained on. The simplest form stores the data's hash with the model; DVC is a tool that keeps hashes in Git and data in cloud storage.
DataFrame: The table type in pandas, with named columns and numbered rows, used to hold and work with data. For example, `mlflow.search_runs` returns its results as a DataFrame.
Docker Compose: A Docker tool that describes a service's settings (port, environment variable, volume, restart policy) in `compose.yaml` so one command builds and starts it.
Docker layer caching: Docker reuses unchanged build steps. Installing requirements before copying code means editing `app/main.py` rebuilds in seconds.
Dockerfile: A recipe of instructions Docker uses to build an image of the API. The image holds packages, code and model, and is tagged with the model version.

E

Endpoint: A URL path plus HTTP method that a service answers, such as `POST /v1/predict`.
Environment variable: A setting a program reads from its surroundings, like `PREDICTION_LOG`, so a container can change behavior without code changes.
Exit code: The number a program returns when it ends: 0 means success, anything else failure. Schedulers and CI watch it, so a refused file becomes a failed run.
Experiment (MLflow): A named group of MLflow runs, such as `kc-churn`.
Experiment tracking: Recording every training run's settings, scores and outputs in one searchable place so runs can be compared.

F

Fairness monitoring: Checking with every batch of labels how the model treats different groups, such as precision and recall for each `area`. Warning signs are precision falling or recall collapsing for one group.
FastAPI: The Python web framework used for the prediction service. `fastapi run` starts it, and `fastapi dev` restarts it on every save.
Feature store: A system that serves the same features for training and for prediction, solving the need for each subscriber's history at prediction time.
Feedback loop: A model's actions changing the data it later learns from: subscribers who stay because Priya called look like safe cases in the next training data. The control group breaks the loop.
Fixture (pytest): An object pytest builds and passes to any test that names it. Session scope builds it once per run, which matters when a fixture trains a model.

G

Gradient boosting: A model type that builds many small decision trees one after another, each correcting the last. It is the alternative to logistic regression, chosen by `kind = "boosting"`.

H

Hash: A fingerprint of data that changes completely if even one byte changes. It can record exactly which data file a model was trained on, or assign customer IDs to groups (such as control and canary) consistently; Python's built-in `hash()` is unsuitable for this because its values change every time Python starts.
Health check: The `/health` endpoint reports that the service is up and which model version it runs. Docker's `HEALTHCHECK` calls it every 30 seconds and marks the container unhealthy if it stops answering.
Holdout: Labeled data kept out of training and used only to evaluate. In retraining it is the newest labeled week, which the challenger didn't train on.
HTTP status codes: Numbers an API returns to say how a request went. In general 2xx means success, 4xx a mistake by the caller and 5xx a server fault; for example, FastAPI returns 422 when the JSON is valid but its content breaks the schema, with a `loc` field naming the offending field.
Hyperparameter: A model setting chosen before training, such as `C` or `max_iter`, kept in the config file.

I

Image tag: A name for one image version; here it is the model version (`v6`), never "latest", so rolling back means starting the old tag.
Imputation: Filling in missing values before modeling, for example with the column's median for numbers or "unknown" for categories. The same filling must happen in training and serving.

J

joblib: A Python library that saves objects such as models to disk using the pickle format.
JSON: A text format for structured data; Priya's tool sends a subscriber's details as JSON and gets a risk score back.
JSON Lines: A text format with one JSON object per line, ideal for logs because you only append. pandas reads it with `lines=True`.

K

KS statistic: The Kolmogorov-Smirnov statistic: the largest vertical gap between two samples' cumulative distributions, from 0 (identical) to 1 (no overlap). It measures how far a column's distribution has shifted, and its size is more informative than its p-value, which flags tiny differences as significant when samples are large.

L

Label: The known outcome for a row, here whether a subscriber canceled. Labels arrive four weeks after scoring, so performance can only be checked late.
Latency: How long one request takes. Report percentiles such as p50 (median) and p95 rather than the average, which hides the slow requests.
Lifespan (FastAPI): Code that runs once when the service starts and stops. The service loads its model there instead of on every request.
Lock file: A list of every installed package with its exact version, made with `pip freeze`, so the environment can be rebuilt elsewhere.

M

Metric (MLflow): A number a run measured, such as `test_auc`.
MLflow: A free, open-source tool for tracking experiments and managing models. It records runs in `mlflow.db`, keeps artifacts in `mlruns`, and offers a web page via `mlflow ui`.
mlflow-skinny: MLflow without the web interface and database. The service image uses it because serving needs far fewer packages than training.
MLmodel: The file in a model package that describes how to load the model, its signature and your metadata such as the threshold.
MLOps: The practice of running ML models reliably in production, borrowing from software engineering (tests, versions, automation) and adding for the fact that models depend on changing data.
Model card: A short document that travels with a model: what it's for, what it was trained on, how good it is and where it fails. The "Not for" line prevents misuse.
Model package: A folder with a fixed layout holding a model plus its signature, metadata and environment files, so a program knows what it needs to run it.
Model registry: Numbered versions of a named model (here `kc-churn`) plus which one is in use via aliases. Registering a version is not the same as deploying it.
Model test: A test of a trained model's quality or behavior, such as a minimum AUC or a behavioral check.
Model URI: MLflow's address for a model: `models:/kc-churn@champion` means whichever version `champion` points to, and `models:/kc-churn/1` means version 1.
Monitoring: Watching a system's health, data and performance over time, across four layers: service, data quality, drift and performance.

N

Non-root user: Running the container as an ordinary user instead of root, so someone who breaks in through the API can't install software or change system files.

O

One-hot encoding: Turning a category such as `plan` into separate 0/1 columns, one per value, so a model can use it.
Online prediction: Scoring one request in milliseconds while someone waits, like Priya's help desk tool asking for a subscriber's risk.

P

Package (Python): A folder of modules with an `__init__.py` file, so scripts can write `from churn.features import prepare`. Shared code lives in the `churn` package.
Paired bootstrap: Resampling the same rows many times and scoring both models on each sample, to measure the noise in their difference. If the 95% interval of the gain is above 0, the gain is real.
Parameter (MLflow): A setting recorded for an MLflow run, such as `C`, the seed or the data hash.
Percentile: The value below which a given share of measurements fall; p95 is the time that 95% of requests beat.
Pickle: Python's general save format, which can run any code when loaded, so it's unsafe to load from strangers.
Pin: Naming an exact package version, like `scikit-learn==1.9.1`, so the same code gives the same result later.
Pipeline (scikit-learn): A chain of preprocessing steps and a model that works as one object, so the same preparation runs in training and prediction.
Precision: Of the subscribers flagged, the share who really canceled. It tells Priya how often a flag is right.
Prediction drift: A change in the distribution of the model's scores, which sums up the input changes that matter to the model.
Prediction log: The record of every prediction's inputs, score, source and model version, appended after each prediction. Monitoring and retraining rely on it.
Production: Running for real, on new data, for people who rely on it, week after week and unattended.
Promotion gate: The rules a challenger must pass before replacing the champion: checks pass, minimum AUC, calibration, and a real, worthwhile gain.
Proxy variable: A feature that stands in for something sensitive, such as a zip code acting as a proxy for income or race. Before adding one, ask what it adds, check fairness with it and record the decision in the model card.
PSI: Population stability index: a number for how far a column's distribution has moved from the reference. Below 0.1 is stable, 0.1 to 0.25 watch, above 0.25 a real shift.
Pydantic: A Python library that turns a schema into code that validates data, rejecting bad types, out-of-range values and unknown fields. One `Subscriber` class validates both API requests and weekly files.
pytest: A Python testing tool. `python -m pytest -q` runs all tests, and FastAPI's `TestClient` lets tests call the API without a real server.

Q

Quality gate: A model test with a minimum score, such as AUC at least 0.75: below it the build fails and nothing ships.

R

Rate gap: The average predicted risk minus the actual cancel rate. A large gap means the probabilities are off, which breaks the break-even threshold.
Recalibration: Keeping a model's ranking but refitting how its scores map to probabilities on recent labels, so the 0.278 threshold means break-even again.
Recall: Of those who actually canceled, the share the model flagged.
Reference data: The data a new batch is compared with when measuring drift, here the training data scored by the champion. PSI bins and KS comparisons are taken from it.
Refit: Train again, here on all labeled rows after measuring the test score, so the production model learns from every row.
Registered model: A named model in the registry, such as `kc-churn`, which holds numbered versions.
Release: One shipped model: a registry version plus its image tag.
Reproducible: Giving the same result when rerun later. It needs fixed code, packages, settings and data.
Retraining trigger: What starts a retrain: a schedule, drift, or falling performance.
Rollback: Going back to the last version that worked, done by moving the `champion` alias back and starting the previous image tag. Nothing is deleted.
Run (MLflow): One recorded execution of training code in MLflow, with a unique ID, its parameters, metrics and artifacts.
Runner: The fresh virtual machine a CI job runs on; it starts empty each time, so anything that must last, like the prediction log, has to live outside it.

S

S3: Amazon's cloud file storage. The weekly workflow copies the export, labels and prediction log to and from an S3 bucket, though any storage would work.
Sample weight: How much one row counts when fitting or scoring a model. Called subscribers get 0, flagged controls get 10 and others 1.
Schema: A precise description of valid data (allowed values, ranges, optional fields). Pydantic turns one into code used by the API and the batch job.
Secret (GitHub): An encrypted repository setting, such as a password or access key, read in a workflow as `${{ secrets.NAME }}` and never shown in logs.
Seed: The number a random generator starts from; the same seed gives the same "random" numbers. In scikit-learn it is passed as `random_state`.
Serving: Making a model's predictions available to other programs, here through a FastAPI service.
Shadow mode: Scoring every request with a new model too but using only the old model's answer. It shows whether the new model runs and how its scores differ, with no risk to customers.
Signature: The inputs and output a saved model expects: each column's type and whether it may be missing. MLflow stores it in the `MLmodel` file.
skops: A safer format for saving scikit-learn models that only rebuilds object types on an allow list, unlike pickle. Some types must be listed explicitly.
SQLite: A database stored in a single file. MLflow uses `mlflow.db`, a SQLite file, as its default tracking store.
Stand-in: A locally built substitute for something that can't be obtained, such as a Docker base image that couldn't be downloaded from Docker Hub. It lets the remaining steps run unchanged.
Stratify: Splitting data so the train and test halves keep the same share of each label, here about 10% cancelers.

T

Tag (MLflow): A free-form label on a run or model version, such as the Git commit or who approved it.
Threshold: The risk at or above which a subscriber is flagged for a call, 0.278 here because a $10 call pays off when the cancel chance exceeds $10 ÷ $36. It lives in the config file and travels in the model's metadata.
TOML: A simple config format of `[sections]` and `key = value` lines, which Python reads with the standard-library module `tomllib`.
Tracking store: Where MLflow records runs; by default the SQLite file `mlflow.db`, while a team shares an MLflow server instead.
Training-serving skew: Data prepared differently for training and for predictions, such as filling missing ratings with the median in one and 0 in the other. Nothing crashes, but scores are slightly wrong; one shared `prepare` function prevents it.

U

Unit test: A test of one function with known input and output, such as `check_batch` catching a constant column.

V

Virtual environment: An isolated Python setup for one project so its packages don't clash with others. You activate it with `source .venv/bin/activate`.
Volume (Docker): Storage that outlives a container, so the prediction log survives when the container is replaced.

W

Workflow (GitHub Actions): A YAML file in `.github/workflows/` listing when to run and the jobs and steps to run on a fresh runner, used for CI and for the scheduled weekly job.

Y

YAML: A plain-text format of `key: value` lines, indented to show nesting. MLflow package files and GitHub workflows use it.