What we mean by an outlier

Over the last few parts of this path you built a toolkit for describing a column of numbers: a centre (part 3), a spread (part 4), percentiles and box plots (part 5), the overall shape of the distribution (part 6), and, for bell-shaped data, the 68-95-99.7 rule (part 7). All of that assumes the numbers behave themselves and cluster together in a recognisable way. Real data is rarely that polite. Almost every dataset you will work with contains at least one value that sits apart from the rest, and knowing what to do about it is its own skill.

An outlier, in plain terms, is an observation that is unusually distant from the other observations in the same dataset. That definition is deliberately loose, because deciding how far counts as unusual is exactly the practical problem this article solves. The second, harder problem is deciding what an unusual value means once you have found it: a typo, a broken sensor, a genuinely rare event, or a sign that you are mixing two different things together. Outliers matter because they distort the very summaries you have learned to trust. The mean and standard deviation are pulled hard by extreme values, while the median and IQR barely move, which is the robustness idea you first met when comparing mean and median in part 3.

A small dataset to work with

Suppose a small delivery company logs the time, in minutes, between an order being placed and it arriving, for 15 orders on a weekday afternoon. The recorded times, sorted from shortest to longest, are: 18, 18, 19, 19, 20, 20, 20, 21, 21, 22, 22, 23, 23, 24, 95. Fourteen of the fifteen deliveries cluster tightly between 18 and 24 minutes. One took 95 minutes. Does that single order belong with the rest, statistically speaking, or does it stand apart? We will answer that with two methods you can apply to almost any numeric column, then spend the second half of the article on the harder question of what the answer actually means.

Finding outliers two ways

Method 1: the IQR rule

Part 5 introduced quartiles and the box plot, and the box plot already carries a built-in test for flagging unusual points: anything plotted beyond the whiskers. To find Q1 and Q3 for our 15 values, split the sorted list at the median and take the median of each half, exactly as in part 5. The median of all 15 values is the 8th one, 21. The lower half, the first seven values (18, 18, 19, 19, 20, 20, 20), has a median of 19, so Q1 is 19. The upper half, the last seven values (21, 22, 22, 23, 23, 24, 95), has a median of 23, so Q3 is 23. That gives an interquartile range of IQR = 23 - 19 = 4.

The standard rule, which traces back to the way box plots were originally designed, flags anything more than 1.5 times the IQR beyond Q1 or Q3. Here that means: lower fence = 19 - 1.5 x 4 = 13, upper fence = 23 + 1.5 x 4 = 29. Any value below 13 or above 29 is a candidate outlier. None of our deliveries falls below 13, but 95 is far above 29, so it is flagged clearly. If you drew the box plot for this data, that 95-minute order would appear as a single dot well past the upper whisker.

The 1.5 multiplier is a widely used convention, not a law of nature. Some analysts use 3 instead of 1.5 to mark only the most extreme points as outliers, reserving the smaller multiplier for merely unusual ones. What makes the IQR rule genuinely useful is that it inherits the robustness of quartiles: Q1 and Q3 depend on the rank of values, not their exact size. If that 95-minute delivery had instead taken 950 minutes, Q1 and Q3 would not move at all, because they are determined by the middle-ranked values, not the extreme one. This is the same resistant-to-extremes property that made the median a safer centre than the mean back in part 3.

Method 2: the z-score rule

The second common method uses the mean and standard deviation from part 4, together with the intuition from part 7's 68-95-99.7 rule: in a roughly normal distribution, about 99.7 percent of values lie within 3 standard deviations of the mean, so a point further away than that is genuinely rare under that assumption. This gives a natural test: convert each value to a z-score, z = (x - mean) / standard deviation, and flag anything with a large absolute z-score, commonly above 3, sometimes above 2 for a stricter check.

For our 15 delivery times, the mean is 385 divided by 15, about 25.67 minutes, and the sample standard deviation works out to about 19.27 minutes. The z-score for the 95-minute order is (95 - 25.67) / 19.27, about 3.60, comfortably past the usual threshold of 3. Every other order has a small z-score: 18 minutes gives about -0.40, and 24 minutes gives about -0.09, all well within one standard deviation of the mean. Both methods agree on this dataset: the 95-minute delivery is the outlier, nothing else is.

You can check both calculations with a short script. It only needs Python's built-in statistics module, nothing to install.

🐍Python
import statistics

data = [18, 19, 21, 22, 20, 23, 19, 24, 21, 20, 95, 22, 18, 23, 20]

sorted_data = sorted(data)

# Q1 = median of the lower 7 values, Q3 = median of the upper 7 values
q1 = statistics.median(sorted_data[:7])
q3 = statistics.median(sorted_data[8:])
iqr = q3 - q1
lower_fence = q1 - 1.5 * iqr
upper_fence = q3 + 1.5 * iqr

mean = statistics.mean(data)
sd = statistics.stdev(data)  # sample standard deviation, divides by n - 1

print("Q1, Q3, IQR:", q1, q3, iqr)
print("Fences:", lower_fence, upper_fence)
print("Mean, SD:", round(mean, 2), round(sd, 2))

outliers_iqr = [x for x in data if x < lower_fence or x > upper_fence]
z_scores = [round((x - mean) / sd, 2) for x in data]

print("Outliers by IQR rule:", outliers_iqr)
print("Z-scores:", z_scores)

Running this prints Q1 = 19, Q3 = 23, IQR = 4, fences of 13 and 29, a mean of about 25.67, a standard deviation of about 19.27, an IQR-flagged outlier list containing only 95, and a z-score list in which 95 stands at roughly 3.6 while every other value sits between about -0.4 and 0. Both tools point to the same single order.

The z-score rule's blind spot: masking

The z-score method has a subtle weakness worth understanding before you rely on it. The standard deviation you use to judge whether a point is extreme is itself calculated from all the data, including the extreme point. A single large outlier can inflate the standard deviation so much that it raises the very bar used to detect outliers, including itself and any other unusual points nearby. This effect is sometimes called masking.

You can see this directly in our data. If you remove the 95-minute order and recompute, the mean drops from about 25.67 to about 20.71, which is expected since an extreme value pulls the mean. But look at the standard deviation: without that one point, it falls from about 19.27 minutes to about 1.90 minutes, roughly a tenfold drop. Squaring the deviations before averaging them, as the standard deviation formula does, means a single far-away point dominates the sum: that one order alone accounts for about 92 percent of the total squared deviation used to compute the standard deviation on the full dataset. In other words, almost the entire "spread" the z-score method reports is actually just one data point advertising itself. With a bigger inflated standard deviation, a second moderately unusual value sitting, say, 40 minutes out might no longer cross a z-score threshold of 3, even though it clearly does not belong with the rest. That is masking in action: the outlier hides evidence of itself, and can hide other outliers too.

A more robust alternative: median and MAD

A fix used in practice replaces the mean and standard deviation with the median and the median absolute deviation, or MAD, both of which resist being dragged around by a single extreme value. For our data, the median is 21. The absolute deviations from the median are 3, 3, 2, 2, 1, 1, 1, 0, 0, 1, 1, 2, 2, 3, and 74 (for the 95-minute order). The median of those absolute deviations is 2, so the MAD is 2. A commonly used modified z-score is 0.6745 x (x - median) / MAD, where 0.6745 is a scaling constant chosen so the result is comparable to an ordinary z-score when the data is normally distributed. For the 95-minute order, that gives 0.6745 x (95 - 21) / 2, which is about 25, far beyond the usual flagging threshold of roughly 3.5 for this method. Compare that to the ordinary z-score of 3.6: the modified version gives a much more decisive signal, precisely because the median and MAD barely noticed the outlier in the first place, so there was nothing to mask it with.

Comparing the two methods

  • The IQR rule is robust: it is built from quartiles, which depend on rank rather than magnitude, so it works reasonably well even on skewed distributions (recall the asymmetric shapes from part 6). It is a sensible default when you are exploring a new dataset and do not yet know its shape.
  • The z-score rule is intuitive and ties directly into the 68-95-99.7 rule from part 7, but it assumes the bulk of the data is roughly symmetric and bell-shaped. On skewed data, the mean and standard deviation are themselves distorted by the skew, which can produce both missed outliers and false alarms.
  • The z-score rule is also vulnerable to masking, as shown above, especially in small samples or when more than one outlier is present.
  • The modified z-score, using median and MAD, offers a more robust middle ground and is used in some automated anomaly-flagging systems, at the cost of being slightly less familiar to read.

A practical starting point: when you are unsure about the shape of a distribution, or the sample is small, lean on the IQR rule and a box plot, since you already know how to build both from part 5. Treat the z-score method as a second opinion, most trustworthy when you have good reason to think the data is close to normal, for example repeated physical measurements of the same quantity.

An outlier is a flag, not a verdict

This is the central idea behind the title of this article: finding an outlier is only step one. The IQR rule or the z-score rule can tell you that a value is numerically unusual relative to the rest of the sample. Neither one tells you why. That question can only be answered by looking outside the number itself, at where it came from.

  • Data entry error: someone typed 95 instead of 9.5, a decimal point slipped, or a digit was duplicated. This is correctable if you can trace the original source.
  • Measurement or instrument error: a GPS trace glitched and logged extra transit time, or a clock on one device was out of sync with the others. This may need correcting, or discarding with a clear note of why.
  • A genuine rare event: the driver really was stuck behind an accident or a stopped train, and the order genuinely took 95 minutes. The value is correct and belongs in the dataset, even though it is extreme.
  • A mixed population: perhaps this particular order was a long-distance delivery outside the usual radius, a fundamentally different kind of trip being recorded in the same column as ordinary local deliveries. Folding it into one summary compares two different processes as if they were one.
  • A process change: in a time-ordered dataset, an apparent outlier can also be the first sign of a shift in the underlying process, rather than an isolated accident. That possibility becomes easier to check once you start looking at data over time, a topic that goes beyond this article.

Notice that the number 95 alone cannot distinguish between these explanations. You need the order's address, the delivery log notes, the GPS trace, or a conversation with the driver. Statistics narrows the search by telling you where to look; it does not do the looking for you.

What to do once you understand it

  • If it is a correctable error, fix it using the original source if one exists. If you cannot verify the correct value, treat it as missing rather than guessing a replacement.
  • If it is an error you cannot correct or verify, it is often reasonable to exclude just that record from the analysis, but write down that you did it and why, so the decision can be checked later by someone else.
  • If it is a genuine rare event belonging to the same population, keep it. Report statistics that are not overly sensitive to it, such as the median and IQR, alongside or instead of the mean and standard deviation, and say plainly that an unusual value is included.
  • If it belongs to a genuinely different population or process, consider analysing it separately rather than blending it into one summary. For example, report typical in-city delivery time and long-distance delivery time as two separate figures rather than one average that represents neither well.
  • When outliers are a normal feature of the kind of data you work with, such as incomes, transaction sizes, or session lengths, make the median and IQR your everyday summary rather than the mean and standard deviation, extending the lesson from part 3 about choosing the right centre.
  • A documented middle ground between keeping and deleting is winsorizing or trimming: capping or removing values beyond a chosen percentile, for example limiting anything above the 99th percentile to that percentile's value. This reduces the influence of extremes while keeping the sample size, but it is a deliberate adjustment and should always be disclosed, not hidden inside the analysis.

One option to avoid in every case: deleting a data point purely because it makes a chart look tidier or a mean look more flattering, without being able to say why the value occurred. That quietly changes the question your analysis is actually answering, and it is one of the easiest ways to mislead people without intending to.

Common mistakes

  • Treating outlier detection as a one-off cleaning step to run once and forget, rather than a check you repeat each time new data arrives.
  • Deleting flagged values before investigating whether they are errors or genuine signal. Sometimes the outlier is the most interesting observation in the dataset, such as the delivery that reveals a routing problem worth fixing.
  • Applying the z-score rule to visibly skewed data, where the mean and standard deviation are already distorted by the skew (see part 6), producing both missed outliers and false alarms.
  • Treating the 1.5 or 3 multiplier in the IQR rule as an exact law rather than a convention. Some genuinely skewed, well-behaved data will flag many points this way, which may simply mean the rule needs adjusting for that context, not that the data is full of errors.
  • Forgetting that an outlier is defined relative to the sample it sits in. A 35-minute delivery would be unremarkable among rural deliveries but stands out among these 15 city orders.
  • Assuming a far-from-average value must be wrong. Rare and real are not the same thing as impossible, and the explanations list above exists precisely because more than one of them is usually plausible until you check.

Summary and what's next

An outlier is a value that sits unusually far from the rest of a dataset. Two concrete tools find candidates: the IQR rule, built on the quartiles from part 5, which is robust and works reasonably well even on skewed data; and the z-score rule, built on the mean and standard deviation from part 4 and the 68-95-99.7 intuition from part 7, which is simple but assumes roughly normal data and can be fooled by the very outlier it is trying to catch through the masking effect. Finding a candidate only answers whether a value is statistically unusual. Deciding what it means, an error, a rare but real event, or evidence that you are mixing two different populations, requires looking at where the data came from, and the right response, correcting, keeping, separating, or carefully excluding, follows from that answer rather than from the number alone.

Part 9 moves from describing one variable at a time to describing how two variables move together, introducing correlation and covariance, and examining why a strong correlation does not mean what people often assume it means.