Accuracy Is Not Enough: Five Metrics Stories

Why accuracy feels like the obvious choice

Accuracy is the fraction of predictions a model gets right. If a model makes 1,000 predictions and 950 of them match the true label, accuracy is 95 percent. It is easy to compute, easy to explain to a non-technical stakeholder, and it matches the everyday idea of being correct. That simplicity is also the trap. Accuracy treats every mistake as equally bad and every class as equally common, and in most real problems neither of those things is true.

The best way to see where accuracy breaks down is not through a formal proof but through small stories with numbers you can check by hand. Each of the five stories below uses a tiny, concrete example. None of the numbers are real measurements from a real system; they are constructed to make the arithmetic clean and the lesson obvious. The code blocks let you reproduce every number yourself.

Story one: the fraud detector that does nothing

Imagine a bank with 10,000 transactions a day, of which 100 are fraudulent. Fraud is rare, about 1 percent of the traffic. A lazy model that simply predicts legitimate for every single transaction will be right 9,900 times out of 10,000, for an accuracy of 99 percent. It will also catch exactly zero fraud cases. A model that never does its job can look nearly perfect by this measure, which is why this failure mode is sometimes called the accuracy paradox.

Now compare that lazy model to a real model that flags 80 of the 100 fraud cases correctly, but in doing so also wrongly flags 500 legitimate transactions as fraud. Its accuracy is lower than the lazy model's, yet it is obviously the more useful system, because it actually stops fraud. The code below builds both confusion matrices and computes the standard metrics so you can see the numbers side by side.

import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score

# 10,000 transactions: 100 fraud (label 1), 9,900 legitimate (label 0)
y_true = np.array([1] * 100 + [0] * 9900)

# Lazy model: predicts 'legitimate' (0) for everything
y_pred_lazy = np.array([0] * 10000)

# Real model: catches 80 of the 100 fraud cases,
# but also flags 500 legitimate transactions as fraud
y_pred_real = np.array(
    [1] * 80 + [0] * 20 +      # within the 100 fraud cases
    [1] * 500 + [0] * 9400     # within the 9,900 legitimate cases
)

for name, y_pred in [("lazy", y_pred_lazy), ("real", y_pred_real)]:
    acc = accuracy_score(y_true, y_pred)
    prec = precision_score(y_true, y_pred, zero_division=0)
    rec = recall_score(y_true, y_pred, zero_division=0)
    f1 = f1_score(y_true, y_pred, zero_division=0)
    print(name, "accuracy=%.3f precision=%.3f recall=%.3f f1=%.3f" % (acc, prec, rec, f1))

print(confusion_matrix(y_true, y_pred_real))

Running this prints an accuracy of 0.990 for the lazy model and 0.948 for the real model, confirming that the lazy model scores higher on accuracy alone. But look at recall, the fraction of actual fraud cases the model catches: 0.000 for the lazy model versus 0.800 for the real model. Precision, the fraction of fraud alarms that are genuine, is 0.138 for the real model, meaning only about 1 in 7 flagged transactions is truly fraud. The F1 score, which is the harmonic mean of precision and recall, lands around 0.235 for the real model and 0.000 for the lazy one, reflecting that usefulness, not raw correctness, is what matters here.

The lesson is not that precision and recall are always better than accuracy. It is that when classes are imbalanced, accuracy can be high for reasons that have nothing to do with the thing you actually care about, which in this case is catching fraud.

Story two: the screening test and the rare disease

Medical screening introduces a second, subtler trap: even a test that looks good on paper can be nearly useless once you account for how rare the condition is. Suppose a disease affects 20 out of 1,000 people, a base rate of 2 percent. A screening test has 95 percent sensitivity, meaning it correctly identifies 95 percent of people who truly have the disease, and 90 percent specificity, meaning it correctly clears 90 percent of people who do not.

Those numbers sound strong. Here is what they mean in absolute terms for 1,000 people: of the 20 who are sick, the test correctly flags 19 and misses 1. Of the 980 who are healthy, the test correctly clears 882 but wrongly flags 98 as sick, because 10 percent of a large healthy group is still a lot of people. The code below turns this into a confusion matrix.

import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score, accuracy_score

# 1,000 patients: 20 have the disease (label 1), 980 do not (label 0)
y_true = np.array([1] * 20 + [0] * 980)

# Test: 95% sensitivity on the sick group, 90% specificity on the healthy group
y_pred = np.array(
    [1] * 19 + [0] * 1 +       # 20 sick patients: 19 caught, 1 missed
    [1] * 98 + [0] * 882       # 980 healthy patients: 98 false alarms, 882 cleared
)

acc = accuracy_score(y_true, y_pred)
sensitivity = recall_score(y_true, y_pred)          # true positive rate
ppv = precision_score(y_true, y_pred)               # positive predictive value

print("accuracy=%.3f sensitivity=%.3f precision(PPV)=%.3f" % (acc, sensitivity, ppv))
print(confusion_matrix(y_true, y_pred))

Accuracy here is 90.1 percent, which again looks reassuring. Sensitivity, the fraction of sick people the test finds, is 95 percent, matching the stated performance of the test. But precision, also called positive predictive value in medical contexts, is only about 16.2 percent. In plain terms, if your test comes back positive, there is roughly an 84 percent chance you do not actually have the disease. This is sometimes called the base rate problem: when a condition is rare, even a specific test generates many false alarms relative to true cases, simply because there are so many more healthy people than sick ones to misclassify.

This is also where the cost of different errors starts to matter more than any single number. A missed cancer diagnosis (a false negative) can be far more costly than an unnecessary follow-up test (a false positive), so clinicians often deliberately favour higher sensitivity even if it drops precision. No single metric captures that trade-off; you need to look at both and decide, in context, which error you would rather make.

Story three: the search results nobody asked for

Accuracy assumes there is a clean label to compare against for every item, and that every item matters equally. Ranking problems break both assumptions. Think of a search engine returning results for the query best remote work laptop. There is no single correct answer to check against; instead there is a ranked list, and what matters is whether the relevant results appear near the top, because almost nobody scrolls to result 50.

Suppose out of the top 10 results returned, 4 are genuinely relevant and 6 are not, and of those 4 relevant results, 3 happen to appear within the top 5 positions. Two metrics commonly used here are precision at k, the fraction of relevant items within the top k results, and recall at k, the fraction of all relevant items in the whole collection that appear within the top k. In this example, precision at 5 is 3 out of 5, or 60 percent, while precision at 10 is 4 out of 10, or 40 percent. Precision actually drops as k grows, because the extra results added between position 5 and position 10 happened to be irrelevant.

A global accuracy figure would not even make sense here, since there is no fixed right-or-wrong label for the whole ranked list. What a search or recommendation team usually tracks instead is precision at k for the k values users actually see, sometimes combined with measures that also reward putting the best results first rather than merely including them somewhere in the list. The general point carries over from fraud detection and screening: the metric has to match the shape of the decision being made, not just whether a prediction happens to be technically right.

Story four: the spam filter and shifting costs

Most classifiers do not output a hard yes or no; they output a probability, and a threshold turns that probability into a decision. Moving the threshold changes precision and recall in opposite directions, and the right threshold depends on which mistake is more expensive, not on which threshold gives the highest accuracy or even the highest F1.

Take a spam filter processing 1,000 emails a day, of which 100 are spam and 900 are legitimate. At the default threshold of 0.5, the model flags 90 emails as spam, catching 70 true spam messages while missing 30, and wrongly flags 20 legitimate emails as spam. Lower the threshold to 0.3, making the filter more aggressive, and it now flags 130 emails as spam, catching 90 true spam messages and missing only 10, but at the cost of wrongly flagging 40 legitimate emails.

import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

# 1,000 emails: 100 spam (label 1), 900 legitimate (label 0)
y_true = np.array([1] * 100 + [0] * 900)

# Threshold 0.5: catches 70 of 100 spam, wrongly flags 20 of 900 legitimate emails
y_pred_t05 = np.array(
    [1] * 70 + [0] * 30 +
    [1] * 20 + [0] * 880
)

# Threshold 0.3 (more aggressive): catches 90 of 100 spam, wrongly flags 40 legitimate emails
y_pred_t03 = np.array(
    [1] * 90 + [0] * 10 +
    [1] * 40 + [0] * 860
)

for name, y_pred in [("threshold 0.5", y_pred_t05), ("threshold 0.3", y_pred_t03)]:
    p = precision_score(y_true, y_pred)
    r = recall_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    print(name, "precision=%.3f recall=%.3f f1=%.3f" % (p, r, f1))

At threshold 0.5, precision is about 77.8 percent and recall is 70 percent, giving an F1 of roughly 73.7 percent. At threshold 0.3, precision drops to about 69.2 percent but recall rises to 90 percent, giving a higher F1 of roughly 78.2 percent. Judged purely by F1, the lower threshold looks better. But think about what the false positives actually mean: a legitimate email, perhaps a job offer or an invoice, silently routed to a spam folder that the user rarely checks. If that cost is high, a team might deliberately choose the threshold 0.5 setting, accepting more spam getting through in exchange for fewer important emails getting lost, even though it scores lower on F1.

This is why practitioners often look at the full precision-recall curve, which plots precision and recall across every possible threshold, rather than reporting a single number at a single threshold. The curve shows the full menu of trade-offs; picking a point on it is a business decision, not a purely statistical one.

Story five: predicting house prices, one mansion at a time

Accuracy, precision and recall are all built for classification, where predictions are categories. Regression problems, where the prediction is a number such as a price or a temperature, need a different family of metrics, and they have their own blind spots.

Suppose a model predicts the sale price of five houses, and the errors, in thousands of dollars, turn out to be 2, minus 3, 1, minus 2, and 50. The first four houses are small misses, off by one to three thousand dollars. The fifth is a large, unusual property where the model is off by 50,000 dollars, perhaps because the model has rarely seen a house that size.

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Errors in thousands of dollars for five houses (one is a large outlier)
y_true = np.array([300, 450, 220, 500, 900])   # actual prices ($1,000s)
y_pred = np.array([298, 453, 219, 502, 850])   # predicted prices ($1,000s)

mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)   # computed manually so the code works across sklearn versions
r2 = r2_score(y_true, y_pred)

print("MAE=%.2f RMSE=%.2f R2=%.3f" % (mae, rmse, r2))

Running this gives a mean absolute error, MAE, of 11.6 thousand dollars, which is simply the average size of the mistake regardless of direction. The root mean squared error, RMSE, comes out to about 22.4 thousand dollars, nearly double the MAE. That gap is not a bug; it is RMSE doing exactly what it is designed to do, which is to square each error before averaging, so a 50,000 dollar miss contributes far more than ten separate 5,000 dollar misses would. If large errors are especially costly in your situation, for example because a 50,000 dollar misvaluation could sink a deal, RMSE reflects that danger better than MAE does. If you simply want to know the typical size of a mistake and do not want one outlier property to dominate the picture, MAE gives a more representative number.

R squared, the proportion of variance in the outcome explained by the model, is a third common regression metric, and it has its own trap. R squared compares your model's errors to the errors of a naive model that always predicts the average value. A high R squared can look impressive on a dataset with a few very large, very predictable houses, even if the model is systematically bad at the mid-range houses that most buyers actually care about. R squared can also turn negative if a model performs worse than simply predicting the average every time, which is a useful sanity check but is sometimes forgotten. As with classification, the right move is to look at more than one regression metric and to plot the actual errors rather than trusting a single summary number.

Common mistakes to watch for

• Reporting accuracy alone on an imbalanced dataset without checking the class distribution first.
• Optimising for F1 or accuracy without asking which error, a false positive or a false negative, is actually more costly in the real decision being made.
• Treating precision, recall or any single number as fixed properties of a model, when in fact they move together as the decision threshold changes.
• Applying classification metrics to ranking problems, or expecting a single confusion matrix to summarise a ranked list.
• Using R squared alone for regression and missing that a model can look strong on variance explained while still being badly wrong for the cases that matter most.
• Computing a metric on the training set and quoting it as if it reflects how the model will behave on new, unseen data.

A short checklist before you trust a metric

• Check how common each class or outcome actually is before reading too much into accuracy.
• Write down, in plain language, what a false positive and a false negative cost in your situation, before picking a metric to optimise.
• For probabilistic classifiers, look at a range of thresholds, not just the default, and pick the one that matches the real costs.
• For ranking or recommendation systems, measure performance at the positions users actually see, not across the whole list.
• For regression, report more than one metric, for example both MAE and RMSE, and look at the distribution of errors, not just the average.

Summary and what comes next

Each of these five stories used the same trick: construct a small, concrete example, compute the numbers by hand or with a few lines of code, and let the arithmetic show where a single metric hides something important. A fraud detector can have 99 percent accuracy and catch nothing. A screening test can have 95 percent sensitivity and still be wrong most of the time it raises an alarm. A search ranking has no single right answer to check against. A spam filter's best F1 score is not automatically its best threshold. A regression model's RMSE and MAE can disagree by a factor of two because of one unusual data point.

None of this means accuracy is useless; on a balanced dataset with roughly equal costs for every mistake, it is a perfectly reasonable summary. The habit worth building is simpler: before quoting a metric, ask what decision it is meant to support, how the classes or outcomes are distributed, and what each type of error actually costs. From there, a natural next step is learning to read a confusion matrix and a precision-recall curve fluently, since most of the metrics in this article are just different views of the same underlying counts.