Two ways for a model to be wrong

Picture four archers shooting arrows at a target, each firing many times. The first archer's arrows land in a tight cluster right on the bullseye. The second archer's arrows also cluster tightly, but the whole cluster sits off to one side, consistently missing the center in the same direction. The third archer's arrows are scattered all over the target, but if you average their positions, the average lands close to the bullseye. The fourth archer's arrows are both scattered and off-center: wide spread, and the spread is not even centered on the target.

These four patterns are a close match for how trained models behave when you imagine retraining them again and again on different random samples of data drawn from the same source. The bullseye is the true relationship you are trying to learn. Each arrow is a prediction your model makes for a particular input, from one particular training run. Bias is how far the average arrow lands from the bullseye: a systematic, repeatable error in a consistent direction. Variance is how spread out the arrows are: how much the prediction would change if you had trained on a slightly different sample. A good model, like the first archer, has both low bias and low variance. Most real models are stuck trading one off against the other, which is exactly what this article is about.

A concrete example: fitting curves to noisy data

The cleanest way to see bias and variance is to fit models of different complexity to the same noisy data and watch what happens. Suppose the true relationship between an input x and an output y is a smooth curve, but every observation also has random noise added to it, the way real measurements do. We will fit three models to samples of this data: a straight line (degree 1 polynomial), a moderately flexible curve (degree 4), and a very flexible curve (degree 15). If you were to plot the data points and all three fitted curves on the same picture, the straight line would cut through the cloud of points ignoring its curvature, the degree-4 curve would trace the curvature nicely, and the degree-15 curve would snake up and down tightly through almost every individual point, including the noisy ones, producing wild wiggles between the points where there is no data to constrain it.

🐍Python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Assumes numpy >= 1.20 and scikit-learn >= 1.0

rng = np.random.default_rng(seed=0)

def true_function(x):
    return 1.0 + 0.5 * x - 0.3 * x**2 + 0.05 * x**3

n_samples = 40
x = rng.uniform(-3, 3, n_samples)
noise = rng.normal(0, 1.0, n_samples)
y = true_function(x) + noise

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.3, random_state=0
)

for degree in [1, 4, 15]:
    poly = PolynomialFeatures(degree=degree)
    x_train_poly = poly.fit_transform(x_train.reshape(-1, 1))
    x_test_poly = poly.transform(x_test.reshape(-1, 1))

    model = LinearRegression()
    model.fit(x_train_poly, y_train)

    train_pred = model.predict(x_train_poly)
    test_pred = model.predict(x_test_poly)

    train_mse = mean_squared_error(y_train, train_pred)
    test_mse = mean_squared_error(y_test, test_pred)

    print(f"degree={degree:>2}  train_mse={train_mse:6.2f}  test_mse={test_mse:6.2f}")

Running this prints one line per degree with the training error and test error. The exact numbers will depend on the random data generated, but the pattern is the important part and should hold whenever you run it with this fixed seed. For degree 1, both train_mse and test_mse will be fairly high and close to each other: the line is too simple to capture the curve, so it is wrong in the same way on both the training data and new data. That is high bias. For degree 4, both errors should drop and stay close to each other, since this model has roughly the right amount of flexibility to match the true cubic-like shape. For degree 15, train_mse should fall dramatically, often close to zero, because the model has enough free parameters to pass through nearly every training point. But test_mse will typically be much higher than train_mse, sometimes worse than the simple line. That gap between a very low training error and a much higher test error is the signature of high variance: the model memorized the particular noisy sample it saw rather than the underlying pattern.

The precise definition

The archer analogy is useful for intuition, but it helps to also have the formal statement, because it tells you exactly what is being traded off against what. For a regression model, if you imagine repeating the entire process of drawing a fresh training set and retraining the model many times, the expected squared error of the model's prediction at a given input x can be split into exactly three non-negative pieces: Bias squared, plus Variance, plus an irreducible noise term. In symbols, writing f hat for the model's prediction and f for the true underlying function: expected squared error equals Bias[f hat(x)] squared, plus Variance[f hat(x)], plus sigma squared, where sigma squared is the variance of the noise in the data itself. Bias is defined as the difference between the average prediction, averaged over all those hypothetical retrainings, and the true value f(x). Variance is defined as how much the prediction varies across those retrainings, around its own average. Sigma squared is the noise floor: the part of the error that comes from randomness in the data generating process itself, which no model, however good, can remove.

A small worked example makes this concrete. Suppose the true value we are trying to predict at some input is 10. Imagine training two different models three times each, on three different random samples, and recording the prediction each time. Model A produces predictions of 7, 7.2 and 6.8. Its average prediction is 7, so its bias is 7 minus 10, which is minus 3, and bias squared is 9. The three predictions are tightly clustered around their own average of 7, so the variance is small, something like 0.03. Model B produces predictions of 9, 13 and 8. Its average prediction is 10, exactly matching the truth, so its bias is 0 and bias squared is 0. But the three predictions are far from their average: the deviations from 10 are minus 1, plus 3 and minus 2, whose squares are 1, 9 and 4, averaging to about 4.67. So model B has zero bias but a variance of roughly 4.67. Model A is like the second archer, consistently off-target but tightly grouped. Model B is like the third archer, scattered but centered correctly on average. Neither is strictly better; which one gives lower total error depends on whether 9 (A's bias squared) or 4.67 (B's variance) is larger, and in this case model B's total error from bias and variance combined is actually smaller.

Seeing the trade-off as model complexity changes

Now imagine sliding a single knob that controls how flexible your model is, from very rigid on the left to very flexible on the right, and at every setting you measure two curves: the error on the data the model was trained on, and the error on fresh data it has never seen. If you were to sketch this, the training error curve would start high on the left and fall steadily as the model becomes more flexible, eventually approaching zero on the far right, because a sufficiently flexible model can always fit its own training data almost perfectly. The test error curve behaves differently. It also starts high on the left, falls for a while as the model gains just enough flexibility to capture the real pattern, reaches a minimum somewhere in the middle, and then rises again on the right as the model becomes flexible enough to start fitting noise instead of signal. The result is a U-shaped test error curve sitting above a steadily decreasing training error curve, with the two curves nearly touching at the bottom of the U and spreading apart as you move further right.

That U shape is the direct visual consequence of bias and variance moving in opposite directions as complexity changes. If you plotted bias squared and variance separately against the same complexity knob, bias squared would trace a steadily falling curve: a rigid model is consistently wrong in the same way, and that systematic error shrinks as the model gains the flexibility to represent the true pattern. Variance would trace a steadily rising curve: a flexible model fits whatever particular noisy sample it is given, so its predictions swing more from one training set to the next. Add those two rising and falling curves together, plus the flat irreducible noise term that does not change with model complexity at all, and you get the U-shaped total error curve. The lowest point of that U is the sweet spot: not the most complex model, not the simplest, but the one where the marginal reduction in bias from adding flexibility is exactly balanced by the marginal increase in variance.

What drives bias and variance in real models

  • Model family: a straight line can only ever represent straight relationships, so it tends to have high bias and low variance no matter how it is trained. An unconstrained deep decision tree can represent almost any pattern, including noise, so it tends to have low bias and high variance.
  • Regularization strength: techniques like L1 or L2 penalties in linear models, or weight decay in neural networks, shrink the model's capacity to chase every detail in the training data. Turning regularization up increases bias and decreases variance; turning it down does the reverse.
  • Neighbors in k-nearest neighbors: with k equal to 1, the prediction for a point is just its single nearest training example, which changes a lot between samples, giving low bias but high variance. With a large k, the prediction is an average over many neighbors, smoothing out noise at the cost of blurring real local structure, giving higher bias but lower variance.
  • Tree depth and pruning: a deep, unpruned tree can carve out a separate region for almost every training point, which is low bias and high variance. Limiting depth or pruning branches trades some of that flexibility for stability.
  • Ensemble methods: bagging, as used in random forests, trains many high-variance, low-bias models on different bootstrap samples and averages their predictions, which cancels out a lot of the variance while leaving bias largely unchanged. Boosting does roughly the opposite: it starts from simple, high-bias models and builds up complexity gradually by fitting each new model to the errors of the previous ones, mainly reducing bias.
  • Amount of training, for iterative learners: training a neural network for very few epochs tends to leave it underfit with higher bias; training for very many epochs without any check on overfitting tends to push it toward higher variance, which is exactly why early stopping is a common safeguard.
  • Feature engineering: adding genuinely informative features that capture real structure in the problem can reduce bias, because the model now has the information it needs to represent the true relationship. Adding noisy or irrelevant features, especially with a limited amount of data, tends to increase variance, because the model has more opportunities to fit coincidental patterns.

Bagging and boosting are worth dwelling on because they show the trade-off is not fixed once and for all, it can be managed by how you combine models. A single deep decision tree is a classic high-variance, low-bias model: retrain it on a slightly different sample and the tree can look quite different. A random forest trains hundreds of such trees on different bootstrap samples of the data and different random subsets of features, then averages their predictions. Averaging many noisy but roughly unbiased estimates cancels out much of the noise, the way averaging many archers' shots, each individually scattered but centered on the bullseye, gives you a combined estimate close to the bullseye with much less spread than any single archer. Boosting methods such as gradient boosted trees start from the opposite end: a single shallow tree is biased, since it is too simple to capture the full pattern, and boosting repeatedly fits new shallow trees to the mistakes of the ensemble so far, gradually reducing bias. Both families end up controlling the trade-off, just by attacking different ends of it.

Diagnosing bias and variance from learning curves

A learning curve is a plot with the size of the training set on the horizontal axis and prediction error on the vertical axis, with two lines: one for error measured on the training set itself, and one for error measured on a held-out validation set, at each training set size. You train the model repeatedly on growing subsets of your data, for example on the first 50, 100, 200, 500 and 1000 examples, recording both errors each time. The shape of these two curves, and the gap between them, tells you which problem you have without needing to inspect the model's internals at all.

If both curves converge quickly to roughly the same, disappointingly high error level as the training set grows, and stay there, you are looking at high bias. The model is too simple or too constrained to capture the pattern, and no amount of additional data will fix that, because the problem is not a lack of examples, it is a lack of capacity. The remedy has to involve the model itself: adding features, loosening regularization, choosing a more flexible model family, or training longer if the model has not converged.

If instead the training error stays low throughout while the validation error stays noticeably higher, with a persistent gap between the two lines that only narrows slowly as the training set grows, you are looking at high variance. The model has enough capacity to fit the training data closely but is not generalizing, because it is partly fitting noise specific to the sample it saw. Here, more data genuinely helps, since a larger and more representative training set leaves less room for the model to fit sample-specific quirks. Simplifying the model, adding regularization, or using an averaging ensemble like bagging also helps, by directly reducing the sensitivity to the particular sample drawn.

Seeing it empirically: a bootstrap experiment

The bias and variance decomposition is usually described in terms of an expectation over many hypothetical retrainings, which can feel abstract. You can make it concrete by actually simulating that process: draw many different training samples from a known true function plus known noise, fit a model to each one, and look at how the resulting predictions behave across all those fits. Because we control the true function and the noise level in this simulation, we know the exact answer and can check that the pieces really do add up.

🐍Python
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Assumes numpy >= 1.20 and scikit-learn >= 1.0

rng = np.random.default_rng(seed=1)

def true_function(x):
    return 1.5 * x**2 - 0.5 * x + 1.0

noise_std = 0.3
n_train = 30
n_bootstrap = 300
x_test = np.linspace(-2, 2, 50)

for degree in [1, 2, 10]:
    predictions = np.zeros((n_bootstrap, len(x_test)))

    for b in range(n_bootstrap):
        x_sample = rng.uniform(-2, 2, n_train)
        y_sample = true_function(x_sample) + rng.normal(0, noise_std, n_train)

        poly = PolynomialFeatures(degree=degree)
        x_sample_poly = poly.fit_transform(x_sample.reshape(-1, 1))
        x_test_poly = poly.transform(x_test.reshape(-1, 1))

        model = LinearRegression()
        model.fit(x_sample_poly, y_sample)
        predictions[b] = model.predict(x_test_poly)

    mean_prediction = predictions.mean(axis=0)
    true_values = true_function(x_test)

    bias_squared = np.mean((mean_prediction - true_values) ** 2)
    variance = np.mean(predictions.var(axis=0))
    noise_variance = noise_std ** 2

    print(
        f"degree={degree:>2}  bias^2={bias_squared:7.4f}  "
        f"variance={variance:7.4f}  "
        f"bias^2+variance+noise={bias_squared+variance+noise_variance:7.4f}"
    )

This code draws 300 fresh training samples of 30 points each from the same true quadratic function corrupted by noise with standard deviation 0.3, so the known irreducible noise variance is 0.3 squared, which is 0.09. For each of the 300 samples it fits a polynomial of a given degree and records its predictions at a fixed grid of 50 test points. Averaging those 300 predictions at each test point gives an estimate of the model's average behavior, which is used to compute bias squared against the true function. The spread of those 300 predictions around their own average, at each test point, gives the variance. Because the true function here is genuinely quadratic, you should see degree 1, which is only a straight line, produce the largest bias squared of the three, since a line simply cannot represent a curve, while its variance stays fairly small. Degree 2 matches the true function's shape, so its bias squared should be close to zero and its variance modest, making its total close to the noise floor of 0.09. Degree 10 should show bias squared close to zero as well, since a flexible enough polynomial can approximate a quadratic, but its variance should be noticeably larger than degree 2's, because with only 30 points to fit 10 free coefficients plus an intercept, small changes in which 30 points you happen to draw swing the fitted curve substantially, especially near the edges of the input range. The exact numbers will depend on the random draws even with the seed fixed across different library versions, but this ordering, lowest total error at degree 2, is what the bias-variance decomposition predicts and is the point of running the experiment.

Practical remedies

  • If you have diagnosed high bias: add more or better features that capture real structure in the problem, reduce regularization strength, switch to a more flexible model family, increase the depth or capacity of the model you already have, or train for longer if the model has not finished converging.
  • If you have diagnosed high variance: collect more training data if you can, simplify the model by reducing its capacity, increase regularization strength, remove noisy or redundant features, use an averaging ensemble method such as bagging or a random forest, apply early stopping during iterative training, and always tune hyperparameters using a separate validation set or cross-validation rather than the test set.

Common mistakes

  • Confusing the statistical variance of the data with the variance of the model. A noisy dataset has high variability in its values, but that is the irreducible noise term, sigma squared, which no model choice changes. Model variance is about how much the model's predictions change across different training samples, a separate quantity you can actually influence.
  • Tuning hyperparameters on the test set. If you repeatedly check test set performance while adjusting model settings, you are effectively fitting to the test set, which makes its reported accuracy an overly optimistic estimate of real-world performance. Use a separate validation set, or cross-validation, for all tuning decisions, and touch the test set only once, at the very end.
  • Assuming more data always fixes overfitting. More data reduces variance, because it leaves less room for the model to latch onto sample-specific noise, but it does nothing for bias. If a linear model is underfitting a genuinely curved relationship, doubling the training set size will not help, because the model's capacity, not the amount of data, is the limiting factor.
  • Expecting an error of zero. The irreducible noise term sigma squared sets a floor below which no model can go, no matter how sophisticated, because it reflects genuine randomness or unmeasured factors in the data generating process. A model that seems to beat this floor on a particular dataset is usually overfitting to that dataset's specific noise rather than genuinely generalizing.
  • Drawing conclusions from a single train and test split. A single split can look favorable or unfavorable purely by chance, especially with a small dataset. Cross-validation, which repeats the split several times and averages the results, gives a more reliable read on whether a model's apparent error is a real pattern or sampling noise.
  • Letting data leak from the test set into preprocessing. Scaling features, imputing missing values, or selecting features using statistics computed from the full dataset before splitting into train and test lets information from the test set influence training, which inflates apparent performance and hides the true variance of the model.

Summary and what comes next

Every prediction error a model makes can be traced back to three sources: bias, the systematic error from a model that is too simple or constrained to represent the true pattern; variance, the error from a model that is too sensitive to the particular sample of data it happened to be trained on; and irreducible noise, the floor set by genuine randomness in the data that no model can remove. These three pieces add together to form the total expected error, and because bias tends to fall and variance tends to rise as you give a model more flexibility, there is usually a sweet spot of complexity where their sum is smallest rather than at either extreme. Learning curves, which compare training error and validation error as the training set grows, are the most practical tool for telling which of the two problems you are facing, and that diagnosis points directly at the right fix: more capacity and better features for high bias, more data and more constraint for high variance. From here, natural next steps are to study cross-validation in more depth, since it is the standard way to measure these errors reliably, and to look at regularization techniques and ensemble methods such as bagging and boosting, since they are the main practical tools for shifting a model along the bias-variance trade-off once you know which direction you need to move.