Accuracy Is Not Enough: Five Metrics Stories

Why accuracy feels like the obvious choice

Accuracy is the fraction of predictions a model gets right. If a model makes 1,000 predictions and 950 of them match the true label, accuracy is 95 percent. It is easy to compute, easy to explain to a non-technical stakeholder, and it matches the everyday idea of being correct. That simplicity is also the trap. Accuracy treats every mistake as equally bad and every class as equally common, and in most real problems neither of those things is true.

The best way to see where accuracy breaks down is not through a formal proof but through small stories with numbers you can check by hand. Each of the five stories below uses a tiny, concrete example. None of the numbers are real measurements from a real system; they are constructed to make the arithmetic clean and the lesson obvious. The code blocks let you reproduce every number yourself.

Story one: the fraud detector that does nothing

Imagine a bank with 10,000 transactions a day, of which 100 are fraudulent. Fraud is rare, about 1 percent of the traffic. A lazy model that simply predicts legitimate for every single transaction will be right 9,900 times out of 10,000, for an accuracy of 99 percent. It will also catch exactly zero fraud cases. A model that never does its job can look nearly perfect by this measure, which is why this failure mode is sometimes called the accuracy paradox.

Now compare that lazy model to a real model that flags 80 of the 100 fraud cases correctly, but in doing so also wrongly flags 500 legitimate transactions as fraud. Its accuracy is lower than the lazy model's, yet it is obviously the more useful system, because it actually stops fraud. The code below builds both confusion matrices and computes the standard metrics so you can see the numbers side by side.

import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score

# 10,000 transactions: 100 fraud (label 1), 9,900 legitimate (label 0)
y_true = np.array([1] * 100 + [0] * 9900)

# Lazy model: predicts 'legitimate' (0) for everything
y_pred_lazy = np.array([0] * 10000)

# Real model: catches 80 of the 100 fraud cases,
# but also flags 500 legitimate transactions as fraud
y_pred_real = np.array(
    [1] * 80 + [0] * 20 +      # within the 100 fraud cases
    [1] * 500 + [0] * 9400     # within the 9,900 legitimate cases
)

for name, y_pred in [("lazy", y_pred_lazy), ("real", y_pred_real)]:
    acc = accuracy_score(y_true, y_pred)
    prec = precision_score(y_true, y_pred, zero_division=0)
    rec = recall_score(y_true, y_pred, zero_division=0)
    f1 = f1_score(y_true, y_pred, zero_division=0)
    print(name, "accuracy=%.3f precision=%.3f recall=%.3f f1=%.3f" % (acc, prec, rec, f1))

print(confusion_matrix(y_true, y_pred_real))

Running this prints an accuracy of 0.990 for the lazy model and 0.948 for the real model, confirming that the lazy model scores higher on accuracy alone. But look at recall, the fraction of actual fraud cases the model catches: 0.000 for the lazy model versus 0.800 for the real model. Precision, the fraction of fraud alarms that are genuine, is 0.138 for the real model, meaning only about 1 in 7 flagged transactions is truly fraud. The F1 score, which is the harmonic mean of precision and recall, lands around 0.235 for the real model and 0.000 for the lazy one, reflecting that usefulness, not raw correctness, is what matters here.

The lesson is not that precision and recall are always better than accuracy. It is that when classes are imbalanced, accuracy can be high for reasons that have nothing to do with the thing you actually care about, which in this case is catching fraud.

Story two: the screening test and the rare disease

Medical screening introduces a second, subtler trap: even a test that looks good on paper can be nearly useless once you account for how rare the condition is. Suppose a disease affects 20 out of 1,000 people, a base rate of 2 percent. A screening test has 95 percent sensitivity, meaning it correctly identifies 95 percent of people who truly have the disease, and 90 percent specificity, meaning it correctly clears 90 percent of people who do not.

Those numbers sound strong. Here is what they mean in absolute terms for 1,000 people: of the 20 who are sick, the test correctly flags 19 and misses 1. Of the 980 who are healthy, the test correctly clears 882 but wrongly flags 98 as sick, because 10 percent of a large healthy group is still a lot of people. The code below turns this into a confusion matrix.

import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score, accuracy_score

# 1,000 patients: 20 have the disease (label 1), 980 do not (label 0)
y_true = np.array([1] * 20 + [0] * 980)

# Test: 95% sensitivity on the sick group, 90% specificity on the healthy group
y_pred = np.array(
    [1] * 19 + [0] * 1 +       # 20 sick patients: 19 caught, 1 missed
    [1] * 98 + [0] * 882       # 980 healthy patients: 98 false alarms, 882 cleared
)

acc = accuracy_score(y_true, y_pred)
sensitivity = recall_score(y_true, y_pred)          # true positive rate
ppv = precision_score(y_true, y_pred)               # positive predictive value

print("accuracy=%.3f sensitivity=%.3f precision(PPV)=%.3f" % (acc, sensitivity, ppv))
print(confusion_matrix(y_true, y_pred))

Accuracy here is 90.1 percent, which again looks reassuring. Sensitivity, the fraction of sick people the test finds, is 95 percent, matching the stated performance of the test. But precision, also called positive predictive value in medical contexts, is only about 16.2 percent. In plain terms, if your test comes back positive, there is roughly an 84 percent chance you do not actually have the disease. This is sometimes called the base rate problem: when a condition is rare, even a specific test generates many false alarms relative to true cases, simply because there are so many more healthy people than sick ones to misclassify.

This is also where the cost of different errors starts to matter more than any single number. A missed cancer diagnosis (a false negative) can be far more costly than an unnecessary follow-up test (a false positive), so clinicians often deliberately favour higher sensitivity even if it drops precision. No single metric captures that trade-off; you need to look at both and decide, in context, which error you would rather make.

Story three: the search results nobody asked for

Accuracy assumes there is a clean label to compare against for every item, and that every item matters equally. Ranking problems break both assumptions. Think of a search engine returning results for the query best remote work laptop. There is no single correct answer to check against; instead there is a ranked list, and what matters is whether the relevant results appear near the top, because almost nobody scrolls to result 50.

Suppose out of the top 10 results returned, 4 are genuinely relevant and 6 are not, and of those 4 relevant results, 3 happen to appear within the top 5 positions. Two metrics commonly used here are precision at k, the fraction of relevant items within the top k results, and recall at k, the fraction of all relevant items in the whole collection that appear within the top k. In this example, precision at 5 is 3 out of 5, or 60 percent, while precision at 10 is 4 out of 10, or 40 percent. Precision actually drops as k grows, because the extra results added between position 5 and position 10 happened to be irrelevant.

A global accuracy figure would not even make sense here, since there is no fixed right-or-wrong label for the whole ranked list. What a search or recommendation team usually tracks instead is precision at k for the k values users actually see, sometimes combined with measures that also reward putting the best results first rather than merely including them somewhere in the list. The general point carries over from fraud detection and screening: the metric has to match the shape of the decision being made, not just whether a prediction happens to be technically right.

Story four: the spam filter and shifting costs

Most classifiers do not output a hard yes or no; they output a probability, and a threshold turns that probability into a decision. Moving the threshold changes precision and recall in opposite directions, and the right threshold depends on which mistake is more expensive, not on which threshold gives the highest accuracy or even the highest F1.

Take a spam filter processing 1,000 emails a day, of which 100 are spam and 900 are legitimate. At the default threshold of 0.5, the model flags 90 emails as spam, catching 70 true spam messages while missing 30, and wrongly flags 20 legitimate emails as spam. Lower the threshold to 0.3, making the filter more aggressive, and it now flags 130 emails as spam, catching 90 true spam messages and missing only 10, but at the cost of wrongly flagging 40 legitimate emails.

import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

# 1,000 emails: 100 spam (label 1), 900 legitimate (label 0)
y_true = np.array([1] * 100 + [0] * 900)

# Threshold 0.5: catches 70 of 100 spam, wrongly flags 20 of 900 legitimate emails
y_pred_t05 = np.array(
    [1] * 70 + [0] * 30 +
    [1] * 20 + [0] * 880
)

# Threshold 0.3 (more aggressive): catches 90 of 100 spam, wrongly flags 40 legitimate emails
y_pred_t03 = np.array(
    [1] * 90 + [0] * 10 +
    [1] * 40 + [0] * 860
)

for name, y_pred in [("threshold 0.5", y_pred_t05), ("threshold 0.3", y_pred_t03)]:
    p = precision_score(y_true, y_pred)
    r = recall_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    print(name, "precision=%.3f recall=%.3f f1=%.3f" % (p, r, f1))

At threshold 0.5, precision is about 77.8 percent and recall is 70 percent, giving an F1 of roughly 73.7 percent. At threshold 0.3, precision drops to about 69.2 percent but recall rises to 90 percent, giving a higher F1 of roughly 78.2 percent. Judged purely by F1, the lower threshold looks better. But think about what the false positives actually mean: a legitimate email, perhaps a job offer or an invoice, silently routed to a spam folder that the user rarely checks. If that cost is high, a team might deliberately choose the threshold 0.5 setting, accepting more spam getting through in exchange for fewer important emails getting lost, even though it scores lower on F1.

This is why practitioners often look at the full precision-recall curve, which plots precision and recall across every possible threshold, rather than reporting a single number at a single threshold. The curve shows the full menu of trade-offs; picking a point on it is a business decision, not a purely statistical one.