The notebook is a comfortable lie

A notebook is a forgiving place to build a model. You can see the whole dataset at once, you can re-run a cell five times until the numbers look right, and you can fix a mistake by editing a cell above without anyone noticing. None of that is available once the model is making predictions for real users. Production is a different environment with different rules: data arrives one row at a time instead of as a neat dataframe, the code that built a feature during training has to be reproduced exactly somewhere else, and nobody is watching over your shoulder to say that a column looks suspicious.

This gap between notebook and production is not a single bug. It is a collection of well-understood failure modes, and almost every one of them has a name and a fix. This article walks through the most common ones, with small worked examples so you can recognise them in your own work, not just in theory.

What actually changes between notebook and production

Before looking at specific failures, it helps to name the structural differences. In a notebook, you typically have the full dataset loaded in memory, you compute features once with pandas or SQL, you split train and test yourself, and you evaluate once at the end. In production, data usually arrives incrementally, often from several systems that were never designed to agree with each other, features must be computed in real time or on a schedule, and the code path that creates a feature for serving is frequently a separate piece of code from the one that created it for training. Any place where these two code paths can disagree is a place where your model's behaviour can quietly change without anyone writing a line of buggy code. It is a mismatch between two correct-looking pieces of code, not a crash.

Failure mode 1: data leakage you did not notice

Data leakage happens when information that would not be available at prediction time sneaks into training, making the model look far better than it will ever perform in reality. It is the single most common reason a model's live performance disappoints people who saw a glowing notebook metric.

Target leakage: the feature that gives away the answer

Imagine you are building a churn model for a subscription product. Out of 1,000 customers, 200 have churned, so the base churn rate is 20 percent. Your dataset includes a column called support_ticket_type, and one of its values is cancellation_request. That value is logged the moment a customer clicks cancel, which is essentially the same moment they churn. If you train a classifier with this column included, cross-validation will report something like 98 percent accuracy, because the model has basically learned to read off the answer from a column that encodes it.

The problem appears the moment you deploy. At the time you actually need a prediction, say seven days before a renewal date, no cancellation ticket exists yet, because the customer has not cancelled. The feature arrives empty or with a placeholder value for every single live prediction. The model, having learned to lean heavily on that one column, effectively loses its strongest signal and falls back to something close to the base rate. A model that looked 98 percent accurate in the notebook can end up barely better than guessing "no churn" for everyone, which by itself would be right 80 percent of the time but catch zero actual churners.

The fix is simple to state and easy to forget in practice: for every feature, ask when the value becomes known relative to the moment you need the prediction. If the answer is "at or after the event I am trying to predict", the feature cannot be used, no matter how predictive it looks.

Preprocessing leakage: fitting on data you should not have seen yet

A subtler version of leakage happens inside preprocessing, not inside a single feature. A common mistake is to scale or normalise the entire dataset before splitting it into train and test sets. The scaler then learns the mean and standard deviation using information from the test set, so the test set is no longer a fair stand-in for unseen data. The effect is usually small on large datasets but can meaningfully inflate reported metrics on smaller ones, and it compounds with any other leakage already present.

The fix is to always fit preprocessing only on the training split, and to apply the already-fitted transformation to the test split and later to production data. The cleanest way to guarantee this in scikit-learn is to put preprocessing and the model together in a single Pipeline object, so there is no way to accidentally call fit on the wrong data.

🐍Python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(42)
X = rng.normal(loc=0, scale=1, size=(1000, 5))
y = (X[:, 0] + rng.normal(scale=0.5, size=1000) > 0).astype(int)

# Split BEFORE any preprocessing happens
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Scaler and model are bound together; fit() only ever sees X_train
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])
pipeline.fit(X_train, y_train)
print("Test accuracy:", pipeline.score(X_test, y_test))

This example uses a synthetic dataset of 1,000 rows with five features, where the label depends mostly on the first feature plus some noise, so you should see test accuracy somewhere in the high 0.80s to low 0.90s range depending on the random seed and library version. The important part is not the exact number but the structure: train_test_split happens first, and StandardScaler.fit is only ever called implicitly on X_train inside pipeline.fit. Compare this with the leaky version, where you would call scaler.fit(X) on the full dataset before splitting. That single misplaced line is enough to let test-set statistics influence training, and it is one of the easiest mistakes to make because it looks completely reasonable in a notebook where you scale the data in one cell and split it in another, with no reminder of which one ran first.

There is a second benefit to using a Pipeline object beyond avoiding leakage: you can save the whole pipeline with a tool such as joblib and load that exact object in production. That guarantees the scaling, encoding and imputation steps used at serving time are bit-for-bit the same ones learned during training, which brings us to the next failure mode.

Failure mode 2: training-serving skew

Training-serving skew happens when a feature is computed one way for training and a slightly different way for serving. Each computation can be entirely correct on its own terms, and that is exactly what makes this failure so hard to catch: nothing crashes, nothing throws a warning, the model simply receives inputs that do not look like what it learned on.

Consider a feature called average_order_value_30d, the average amount a customer spent over the last 30 days, used by a recommendation model. During training, you compute it in a notebook with pandas, grouping historical transactions by customer and by calendar month boundaries, using whatever timezone the raw timestamps happen to be stored in. Suppose for one customer this gives an average of 42.50 dollars. In production, the same feature is computed by a real-time service that maintains a rolling 30-day window, refreshed once a day, and that service happens to exclude the current partial day and convert timestamps to UTC first. For the same customer on the same day, that service might return 39.10 dollars, a difference of roughly 8 percent, purely because of a timezone cutoff and a different definition of "last 30 days".

On its own, an 8 percent difference for one customer sounds minor. The danger is that this kind of systematic offset applies to every customer, every day, in the same direction, because it comes from a structural difference in how the two systems define time windows, not from random noise. The entire distribution of that feature shifts slightly between training and serving. A model that learned fine-grained thresholds on average_order_value_30d during training, for example treating 40 dollars as a meaningful boundary between customer segments, can end up systematically misclassifying customers near that boundary once it is fed the serving version of the feature.

The most reliable fix is to make sure training and serving use the exact same code path to compute a feature, not two independently written implementations that are supposed to agree. This is the core idea behind feature stores: a single definition of each feature, computed once, that both the training pipeline and the serving pipeline read from. If you do not have a feature store, the next best thing is a shared library of feature-computation functions that both pipelines import, plus automated tests that compare offline and online values for a sample of real entities and fail loudly if they diverge beyond a small tolerance.

Failure mode 3: hidden state inside the notebook itself

Notebooks keep every variable alive in the kernel's memory for as long as the session runs, regardless of which cells you have run, in which order, or whether you have since deleted or edited them. This creates a specific trap: you run cell 3, which defines a preprocessing function. You later edit that function in cell 12 and re-run it. The model you trained earlier in cell 8 was trained using the old version of the function, but nothing in the notebook tells you that. Everything still looks consistent because the variables all still have values, they are simply values produced by a sequence of steps that no longer matches the code visible on the page.

A closely related trap is leftover state from a previous experiment. You load a dataframe, filter it down to remove outliers, train a model, get a good score, and move on to try something else, reusing the same variable name for a new dataframe but forgetting that an old cell further down still refers to the filtered version. When you re-run cells out of order later, results no longer reproduce, and it is genuinely hard to tell why.

The test for this is mechanical and worth doing before you trust any notebook result: restart the kernel and run every cell from top to bottom in order, with no manual intervention. If the final metric changes, or if the notebook errors out partway through, you had hidden state that was propping up your earlier result. Once a model is ready to move toward production, its full training logic belongs in a plain script or a small number of functions that can be run start to finish with no notebook session in between, precisely because a script cannot accumulate invisible state the way a long-lived kernel can.

Failure mode 4: the data drifts after you deploy

Even a model with no leakage, no skew and no hidden state problems can degrade over time simply because the world keeps moving and the training data does not. This is usually split into two related ideas. Data drift means the distribution of the input features changes, for example a retailer's customer base shifting toward a younger age group after a marketing campaign. Concept drift means the relationship between the inputs and the target changes, for example a fraud model trained before a new, cheap payment method existed, which fraudsters quickly learn to exploit in ways the training data never showed.

A model trained on transactions from last year has no way to know that this year a new payment option was introduced, or that customer behaviour shifted after a competitor changed its pricing. None of this is a bug in your code; it is a property of deploying any model that was fit on a frozen snapshot of the past into a world that keeps changing. The practical implication is that a model passing its offline test set on day one tells you almost nothing about how it will perform six months later, and silent degradation is the default outcome unless you actively monitor for it, typically by tracking the distribution of incoming features and the model's prediction distribution over time, and comparing them against the distributions seen during training.

Failure mode 5: the environment itself is different

A notebook usually runs inside whatever environment happens to be installed on your laptop or a shared development machine, built up over months of ad hoc pip installs. Production typically runs inside a separate container or server with its own, independently installed set of library versions. If these two environments are not pinned to the same versions, the exact behaviour of a function can differ between them. Libraries change default parameter values, change how they handle missing values or unknown categories, and occasionally change the meaning of an argument between releases. A model trained with one version of a preprocessing library and served with a different version can silently apply a slightly different transformation, with no error raised anywhere, because both versions are individually valid, just not identical.

The defence here is unglamorous but effective: record the exact versions of every library used to train a model, for example in a requirements file or a lock file, and use that exact same set of versions to build the serving environment. Treat the trained model object and the environment it was trained in as a single unit that moves together, not two things you can version separately.

Failure mode 6: scale and latency nobody tested

In a notebook, you usually have the whole test set loaded as one dataframe and you call model.predict once on all of it at the same time, using fast, vectorised operations across thousands of rows. In production, a request usually needs one prediction for one entity, computed within a strict time budget, often tens of milliseconds, and it needs to assemble that single row of features by calling out to one or more other services rather than reading a ready-made dataframe. If one of those upstream services is briefly slow or unavailable, your serving code has to decide what to do: wait, fail the request, or substitute a default value for the missing feature. Each of those choices changes what the model actually sees compared to the complete, clean rows it was trained on, and a model is rarely tested against the case where half its inputs are filled with defaults because a dependency timed out.

This is less a single bug than a reminder that notebook testing checks whether the model learns a useful pattern, while production testing has to check whether the surrounding system can reliably deliver that model the inputs it expects, within the time it is given, under real failure conditions.

A checklist for closing the gap

None of the failure modes above require exotic tools to avoid. Most of them are closed by a small number of consistent habits, worth treating as a checklist before any model moves past the notebook stage.

  • For every feature, write down when its value becomes known, and remove any feature that is only known at or after the moment you need the prediction
  • Split data into train and test before any preprocessing step, and fit all preprocessing only on the training split, ideally inside a single Pipeline object
  • For time-ordered data, split by time rather than randomly, so the test set only ever contains events that happen after everything in the training set
  • Use the same code, not two independently written versions, to compute each feature for both training and serving, through a shared library or a feature store
  • Restart the notebook kernel and run all cells top to bottom before trusting any result, to rule out leftover hidden state
  • Pin the exact library versions used for training and reproduce that same environment for serving, rather than installing whatever is latest at deploy time
  • Monitor the distribution of incoming features and predictions after deployment, and compare them against the distributions seen during training
  • Test the full serving path under realistic failure conditions, including slow or unavailable upstream services, not just the model object in isolation

Summary and what comes next

A model that looks excellent in a notebook and disappoints in production has almost always fallen into one of a handful of well-known traps: a leaked feature that encodes the answer, preprocessing fitted on data it should never have seen, a feature computed differently for training than for serving, hidden state left over from earlier notebook cells, data that has simply drifted since training, a mismatched library environment, or a serving system that cannot deliver inputs the way the model expects. Every one of these is detectable before deployment if you specifically look for it, which is the main shift in mindset this article is arguing for: a good notebook score is evidence that your code ran without crashing, not evidence that your model will behave the same way once it leaves the notebook.

From here, a natural next step is to look at the practical tools built specifically to close this gap: experiment tracking to record exactly which data and code produced a given model, model registries to version trained models alongside their environments, and monitoring systems designed to catch data drift and skew automatically once a model is live.