Ten Data Leakage Mistakes and How to Catch Them

The exam with the answer key stapled to it

Imagine a teacher hands out a maths test, but a few copies accidentally have the answer key stapled to the back page. Students who notice will score extremely well, not because they understand the maths, but because they can read the answer sitting right behind the question. If you only looked at the marks, you would conclude the class was excellent. The moment the answer keys are removed, the real scores appear, and they are much lower. Data leakage in machine learning is the same problem in different clothes: information that should not be available at prediction time ends up influencing the model during training or evaluation, so the model looks accurate during testing and then performs far worse once it meets genuinely new data.

Leakage is one of the most common reasons a model that beats every benchmark in development quietly fails after deployment. It rarely causes an error message. The code runs, the metrics look good, sometimes suspiciously good, and everyone moves on. The problem usually surfaces weeks later when the model meets real, unseen data and accuracy, AUC or F1 collapses. This article walks through ten specific, common ways leakage gets in, with a worked example and a practical check for each one.

What data leakage actually means

At its core, leakage happens whenever information that would not be available at the moment you need to make a prediction is used, directly or indirectly, while building or evaluating the model. It helps to keep two broad families separate in your head. The first is train-test contamination: statistics, parameters or synthetic examples derived from the test set, or from a validation fold during cross-validation, leak into the training process. The model never sees the exact test labels, but it is shown something computed using them, such as a mean, a fitted scaler, a set of selected features, or an oversampled neighbour. The second is target leakage: a feature available at training time accidentally encodes the outcome itself, usually because it was recorded after the event you are trying to predict, or only exists because the event happened. No clever cross-validation scheme fixes target leakage, because the problem lives inside a single row of data, not in how the data is split.

Both families produce the same symptom: evaluation metrics that look better than what the model can actually do once deployed. Spotting that symptom and tracing it back to its source is most of the skill involved in avoiding leakage. In regulated settings such as healthcare or lending, the cost is not just an embarrassing drop in accuracy, it can mean decisions made on a model that never actually learned what it appeared to learn.

Ten specific ways leakage gets in, and how to catch each one

The mistakes below are ordered roughly by how often they show up in real projects, not by severity, because a single-row target leak can be just as damaging as a split-level contamination. For each one you get a plain explanation, a small worked example with concrete numbers, and a way to test for it. The code examples assume a recent scikit-learn release, version 1.0 or later, and numpy; the exact version rarely matters for the patterns shown.

1. Fitting scalers, imputers or encoders before splitting the data

Suppose you have 1000 rows with a feature called annual_income, and you want to standardise it, subtract the mean and divide by the standard deviation, before feeding it into a logistic regression. If you compute that mean and standard deviation using all 1000 rows and only then split into 800 training and 200 test rows, the scaling parameters used for training already reflect the distribution of the 200 test rows. For a single numeric feature the leak is often small, since a mean computed from 1000 rows rarely differs much from one computed from 800, but the same mistake with a more complex transform, such as fitting PCA on the whole dataset, is more serious: the principal components are chosen partly using test rows, so a test row projected onto those components already carries a trace of itself.

How to catch it: check the order of operations in the code. Split first, fit every transformer on the training rows only, then apply that already-fitted transformer to the test rows. The cleanest way to enforce this is to chain preprocessing and model together in a single pipeline object and only ever call fit on that pipeline using training data, never on the full dataset beforehand.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# toy dataset: 1000 rows, 1 numeric feature, binary target
rng = np.random.default_rng(0)
X = rng.normal(loc=50000, scale=15000, size=(1000, 1))
y = (X[:, 0] + rng.normal(0, 10000, 1000) > 55000).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

# correct: the scaler is fit only on X_train, inside the pipeline
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model', LogisticRegression()),
])
pipe.fit(X_train, y_train)
print('test accuracy:', pipe.score(X_test, y_test))

# wrong, do not do this: fitting the scaler on all data first
# scaler = StandardScaler().fit(X)
# X_scaled = scaler.transform(X)
# X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)

2. Target leakage: a feature that only exists because of the outcome

Consider a hypothetical model predicting whether a patient will be readmitted to hospital within 30 days. Suppose the dataset includes a column such as total_charges_for_readmission, a value that is only filled in if a readmission actually happened, or discharge_disposition, which might already reflect how severe the case turned out to be. A similar pattern appears in credit scoring: a column like write_off_amount is nonzero only when a loan has already defaulted. Any feature like this is not a predictor, it is a record of the outcome wearing a different name, and a model trained with it will look almost perfect during testing because the leaking column is present in both the training and test rows, then fail completely once used to make a real forecast where that column cannot yet exist.

How to catch it: for every feature, ask whether its value would already be recorded at the exact moment the prediction is needed in the real world, not just in the static dataset sitting on disk. Watch for suspicious signs such as a single feature carrying almost all of the feature importance, for example one feature at 0.9 while everything else sits below 0.02, or an accuracy or AUC that is implausibly high for a problem domain experts consider hard. Removing the suspected column and watching performance drop sharply to a more believable level is a reliable confirmation.

3. Temporal leakage: letting the future predict the past

Picture a dataset spanning January 2021 to December 2023, with one monthly snapshot per customer, used to predict next month's churn. A feature like total_purchases_in_last_12_months is only safe if it is computed strictly using data available before the prediction month. If the feature pipeline accidentally includes purchases from after the cutoff, or if you take this time-ordered data and run a plain random five-fold cross-validation, rows from December 2022 can end up training a model that is validated on January 2022, even though December's data did not exist yet at the point you are supposedly predicting from.

How to catch it: always sort by date and use a fixed cutoff, or use a splitting tool built for ordered data so that training folds only ever contain rows earlier in time than the validation fold. Also review each feature's as-of date individually and confirm it is computed using information strictly before the prediction timestamp, a check sometimes called a point-in-time audit.

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

# 24 months of data, one row per month, already sorted by time
X = np.arange(24).reshape(-1, 1)
y = np.random.default_rng(1).integers(0, 2, size=24)

tscv = TimeSeriesSplit(n_splits=4)
for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):
    print('fold', fold, 'train months', train_idx.min(), '-', train_idx.max(),
          'test months', test_idx.min(), '-', test_idx.max())

Running this prints, for each fold, a training range that always ends before the test range begins, unlike an ordinary shuffled k-fold split where train and test months would be mixed together.

4. Group leakage: the same real-world entity split across train and test

Suppose a medical imaging dataset has 500 patients contributing 10 scans each, 5000 images in total, used to detect a condition from the scan. If you split randomly by image, roughly 80 percent of every patient's images land in training and 20 percent in test, so nearly every patient is seen during training. The model can then learn patient-specific traits, a particular scanner's calibration, a specific skin tone, a habitual camera angle, rather than the general pattern the condition actually produces, and the test score ends up measuring memorisation of familiar patients rather than genuine generalisation to new ones.

How to catch it: identify the natural grouping key, a patient id, a customer id, a product id or a session id, and make sure a whole group goes entirely to one side of the split. A quick audit after any split is to check whether the same group identifier appears in both the training and test sets; if it does, that is group leakage.

import numpy as np
from sklearn.model_selection import GroupKFold

# 12 images from 4 patients, 3 images each
patient_id = np.array([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4])
X = np.arange(12).reshape(-1, 1)
y = np.random.default_rng(2).integers(0, 2, size=12)

gkf = GroupKFold(n_splits=4)
for fold, (train_idx, test_idx) in enumerate(gkf.split(X, y, groups=patient_id)):
    print('fold', fold, 'test patients:', set(patient_id[test_idx]))

Each printed fold shows a set of test patients that never appears among the training rows for that same fold, which is exactly what you want when rows are not independent of each other.

5. Duplicate or near-duplicate rows on both sides of the split

Datasets built by merging multiple exports, or affected by a logging bug, sometimes contain exact duplicate rows, the same transaction logged twice, or near-duplicate rows, a flipped or lightly cropped copy of the same image saved as a separate file before anyone splits the data. If a duplicate pair is split so that one copy lands in training and its twin lands in test, the model effectively gets to memorise the answer for that row, inflating the test score by exactly the fraction of rows that have a hidden twin on the other side.

How to catch it: run a deduplication pass before splitting, checking for exact duplicate rows and, where relevant, near-duplicates using hashing or a similarity threshold for text or images. As a second line of defence, after splitting, check whether any row in the test set is identical or nearly identical to a row in the training set, which also catches duplicates that slipped in earlier in the pipeline.

6. Applying oversampling such as SMOTE before splitting

Picture a fraud detection dataset with 10,000 rows, 200 of them fraud, a 2 percent positive rate. A common shortcut is to apply SMOTE to the full dataset to balance the classes and only afterwards split into training and test. SMOTE creates synthetic minority rows by interpolating between a real minority example and its nearest minority neighbours, drawn from the entire dataset at that point, including rows that will end up in the test set. A synthetic training row can therefore be built directly from a point that later becomes a test example, giving the model an early look at that example's neighbourhood before it is supposedly unseen.

How to catch it: never resample before splitting. Split first, then resample only the training fold, and do this freshly inside every cross-validation fold rather than once on the whole training set. Libraries built for imbalanced data provide pipeline objects that perform resampling correctly as part of the fit step, so it is automatically redone for each fold.

7. Target encoding computed using the full dataset

Target or mean encoding replaces a categorical value, a city name for instance, with the average of the target variable among rows sharing that category. It is popular because a high-cardinality column with hundreds of cities is awkward to one-hot encode. Here is the leak in numbers: suppose the city Springfield appears in 10 rows, 7 of which end up in training with target values 1, 0, 1, 0, 1, 0, 0, giving a true training mean of about 0.43, and 3 of which end up in test with values 1, 1, 0. If the encoding is computed using all 10 rows together, 5 positive out of 10, the number used is 0.50, a value already shaped by the 3 test rows' own outcomes. That 0.50 is then used as a feature to predict those same 3 rows, which is exactly the information leak.

How to catch it: compute any target-based encoding using training-fold data only, then apply that fixed mapping to validation and test rows, with a sensible fallback, such as the overall training mean, for categories never seen in training. In cross-validation, use an out-of-fold scheme so the encoding applied to each training row is built from the other folds, not from the row's own fold.

8. Feature selection performed before cross-validation

Imagine a dataset with 2000 candidate features, perhaps gene expression measurements or text n-grams, and only 300 rows. A tempting shortcut is to compute the correlation between every feature and the target using the whole dataset, keep the 50 most correlated features, and then run five-fold cross-validation using only those 50. The trouble is that the correlation step already used the target values of rows that later serve as the validation fold within each of the five folds. The model never sees test labels directly, but the shortlist of features it is allowed to use was chosen partly because of those labels, inflating cross-validation performance, sometimes sharply, when the number of candidate features is large relative to the number of rows. With 2000 random noise features and only 300 rows, pure chance alone can make dozens of features look significantly correlated with a random target, and selecting the best 50 of those before cross-validating can produce a deceptively good score built on nothing but noise.

How to catch it: treat feature selection as part of the model rather than a one-off preprocessing step. Put the selection step inside the same pipeline as the estimator, so that for every cross-validation fold, selection is redone using only that fold's training rows, instead of being computed once outside the loop and then reused everywhere.

9. ID columns and metadata that silently encode the label

Datasets assembled by concatenating files from different collection sites or time periods sometimes have the outcome baked into something that looks like harmless metadata. For example, if all the positive cases in a medical dataset came from one hospital's export and all the negative cases came from another hospital's export, a column like hospital_id, or even the row's position if rows were appended in blocks by outcome, can become an almost perfect predictor, purely because of how the file was built rather than any real medical difference. A database auto-increment id can show the same pattern if, say, a batch of fraud cases happened to be entered late on a particular day, making the id range correlate with the label by accident.

How to catch it: inspect feature importances for any id-like, timestamp-like or metadata column with unexpectedly high importance, and ask where that column actually comes from and whether it could vary independently of the outcome once the model meets a new hospital, a new batch, or a new time period. Training the model without the suspected column and watching how much performance drops is a quick, concrete test.

10. Fitting text or image preprocessing on the full corpus, including test data

In text classification, calling a vectoriser's fit_transform method on the entire corpus, training and test documents together, before splitting means the vocabulary, document frequencies and inverse-document-frequency weights are all computed using words that appear in the test documents. In a small dataset this can meaningfully leak information, particularly if a handful of rare words happen to be strongly associated with one class in the test set. Image pipelines have an equivalent mistake: computing per-channel mean and standard deviation for normalisation, or building a shared reference image for alignment, using the entire image set rather than only the training images.

How to catch it: fit the vectoriser, tokenizer or normalisation statistics using only the training split, then call the already-fitted object's transform method, not fit_transform, on the validation and test data. As with the scaling mistake described first in this list, wrapping text or image preprocessing inside a pipeline and only ever calling fit on training data removes the risk by construction.

A checklist for catching leakage before it catches you

Most leakage follows one of the ten patterns above, so a short, repeatable checklist catches the majority of cases before a model goes anywhere near production.

• Split first, transform second: write the code so that the train and test split, whether random, time-based or group-based, happens before any fitting of scalers, encoders, selectors or resamplers
• For every feature, ask whether this exact value would already exist at the moment the prediction is needed, in the real world, not only in this static dataset
• Treat duplicate and near-duplicate rows as a data quality problem to fix before splitting, not something to notice afterwards
• If predictions are needed over time, split by date rather than by random shuffling, and check that every feature's as-of date falls before the cutoff
• If rows belong to a natural group such as a patient, customer, device or product, split by group, never by individual row
• Be suspicious of metrics noticeably better than what domain experience would predict, and of any single feature or metadata column carrying unusually high importance
• Put every data-dependent step, scaling, encoding, selection and resampling, inside one pipeline object, and only call fit on that pipeline using training data
• Re-run the entire pipeline, preprocessing included, inside each cross-validation fold rather than once outside the loop
• Before trusting a final number, check the gap between training and validation performance, and where possible validate on a later, separately collected batch of data rather than only a random hold-out from the same batch

Summary and what comes next

Data leakage is any situation where information that would not be available at prediction time ends up shaping the model, and it almost always shows up as performance that looks better than it reasonably should. Ten common doors it walks through are early scaling or encoding before a split, target leakage from outcome-derived features, shuffling time-ordered data, splitting related rows across train and test, duplicate records, resampling before splitting, target encoding built on the whole dataset, feature selection done before cross-validation, metadata columns that proxy for the label, and text or image preprocessing fit on the full corpus.

The single habit that prevents most of these mistakes is repeated throughout this article: decide on the split first, by time, by group, or plainly at random, and make sure every data-dependent calculation happens after that split, using only the training portion, wrapped in a pipeline that gets rerun for every fold. Once that habit is automatic, the remaining risk is mostly about asking careful questions of where each feature actually comes from and whether it could realistically be known before the prediction is needed.

A good next step is to pick an existing notebook or project and audit it against the checklist above, paying particular attention to the order in which splitting and preprocessing happen. Getting comfortable with a pipeline object, a group-aware splitter and a time-aware splitter is worth the time, since together they enforce the correct order automatically rather than relying on remembering it every time.