Cross-validation in scikit-learn explained

cross_val_score splits your data into folds, trains and scores the model once per fold, and returns one score for each. Averaging them gives a far more trustworthy number than a single train/test split.

Run it with 5 folds

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

for seed in [0, 1, 2, 3, 4]:
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.1, random_state=seed)
    print("split", seed, round(model.fit(X_tr, y_tr).score(X_te, y_te), 3))

scores = cross_val_score(model, X, y, cv=5)
print(scores.round(3))
print(round(scores.mean(), 3), round(scores.std(), 3))

Output:

split 0 1.0
split 1 1.0
split 2 0.965
split 3 0.982
split 4 0.93
[0.982 0.982 0.974 0.974 0.991]
0.981 0.007

Why one split can mislead

The same model on the same data scored anywhere from 0.93 to 1.0, only because random_state changed which rows landed in the test set. If you had stopped at split 0, you would have believed the model is perfect. If you had stopped at split 4, you would have thought it was mediocre.

Cross-validation fixes this:

1. The rows are cut into 5 folds.
2. Each fold is the test set exactly once, while the other four train the model.
3. You get 5 scores. Report the mean (0.981) and the standard deviation (0.007), which tells you how much the score moves.

For a classifier, cross_val_score scores accuracy by default. Change it with scoring=, for example scoring="f1". Because the scaler is inside the pipeline, it is refitted on each fold's training part only.

Common mistake

Scaling the data before cross-validation. If you call StandardScaler().fit_transform(X) first and then cross-validate, each test fold has already influenced the scaling. Put the scaler in a pipeline, as above, so it is fitted inside every fold.

Trusting the mean alone. A mean of 0.85 with a standard deviation of 0.10 is a very different story from 0.85 with 0.01. Always look at the spread.

Takeaway

One split gives you one lucky or unlucky number. cross_val_score(model, X, y, cv=5) gives you five, so you can see both the score and how much to trust it.