Why this part matters

Every mean, median, standard deviation, percentile and correlation you have calculated in this path so far was computed from a specific set of numbers you had in front of you. But almost never do those numbers represent everything you actually care about. If you compute the average delivery time from last month's orders, you care about delivery times in general, not just last month. If you measure the correlation between study hours and exam scores for one classroom, you probably want to say something about students more broadly. The gap between the data you have and the thing you actually want to know about is the subject of this part.

This gap is where a huge amount of bad analysis comes from, and it is usually not a math mistake. The arithmetic can be perfect and the conclusion can still be wrong, because the data never represented what it was supposed to represent. Understanding populations, samples and sampling bias is what lets you ask the right question before you trust any number: who or what is actually in this dataset, and who or what is missing.

Population and sample, defined

A population is the complete set of things you want to learn about. It could be every customer of a company, every transaction in a year, every patient with a certain condition, every product on a shelf, or every possible measurement a sensor could ever record. A sample is the subset of the population that you actually observe and measure.

Consider a company with 50,000 registered users. If you want to know the average age of all 50,000 users, that full group is your population. If you only have survey responses from 400 of them, those 400 responses are your sample. The number you compute from the sample, the sample mean, is an estimate of a number you did not directly observe, the population mean. These two numbers usually get different symbols so you never confuse them: the population mean is often written as the Greek letter mu, and the sample mean is written as x-bar, the same x-bar notation used back in part 3 when we first defined the mean.

This distinction explains something you may have noticed in part 4. When we computed variance for a sample, the formula divided by n minus 1 rather than by n. That small correction, called Bessel's correction, exists precisely because a sample tends to underestimate the true spread of the population it came from, and dividing by a slightly smaller number nudges the estimate back up to be closer to the population value on average. The correction is a direct consequence of the idea in this part: a sample is not the population, and estimates computed from a sample carry a built-in tendency to be slightly off unless you adjust for it.

Why we sample instead of measuring everyone

If sampling introduces all this extra uncertainty, why not just measure the whole population every time? In practice there are a few recurring reasons.

  • Cost and time: surveying 50,000 customers takes far longer and costs far more than surveying 400 of them, and the extra accuracy from the larger number is often not worth the extra cost.
  • The population is effectively infinite or unmeasurable: if you want to know the average reaction time of all humans, or the failure rate of a manufacturing process over all time, there is no finite list to measure completely.
  • The measurement is destructive: to test the breaking strength of a batch of glass bottles, you break them. You cannot test every bottle and still have bottles left to sell.
  • The population does not exist yet: a machine learning model trained today is meant to work on future data it has not seen, so the training set is always a sample from a population that is partly still in the future.

Sampling is therefore not a shortcut you take when you are lazy, it is usually the only realistic option. The question is never whether to sample, it is how to sample well.

What makes a sample good: representativeness

A good sample is representative, meaning that its mix of characteristics looks like the population's mix of characteristics. If a company's user base is 60 percent on the free plan and 40 percent on the paid plan, a representative sample of 400 users should have roughly 240 free-plan users and 160 paid-plan users, not 390 and 10.

The list or process you use to pick the sample is called the sampling frame. In the user survey example, the sampling frame might be the company's email list. If that email list is missing 5,000 users who signed up with a different email provider and whose messages always went to spam, those 5,000 users can never be selected, no matter how carefully you pick names from the list afterward. This kind of gap, where part of the population has no chance of being selected, is called undercoverage, and it is one of the most common and hardest to notice causes of a bad sample, because the data looks perfectly clean once you have it. Nothing in the 400 rows you end up with tells you that 5,000 people were silently excluded before you even started.

Common ways to draw a sample

There are several standard ways to go from a sampling frame to an actual sample, and the choice affects both how representative the sample is and how practical it is to collect.

Simple random sampling

Every member of the population has an equal chance of being chosen, and selections are independent of each other. If you number all 50,000 users from 1 to 50,000 and use a random number generator to pick 400 unique numbers, that is simple random sampling. It is the baseline against which other methods are compared, and it is what most statistical formulas assume by default unless stated otherwise.

Systematic sampling

You pick every k-th item from a list. If the population has 50,000 users and you want 400, you could pick every 125th user starting from a random point in the first 125. This is easy to implement and usually close to random, but it can go badly wrong if the list has a hidden pattern that lines up with your interval. If a list of factory output is ordered by shift, and every 125th item happens to fall at the start of each night shift, you might end up sampling only night-shift products.

Stratified sampling

You split the population into groups, called strata, based on a characteristic you care about, then sample from each group in proportion to its size. Suppose a company has 100 employees: 70 in sales, 20 in engineering and 10 in management. A simple random sample of 10 employees could, just by bad luck, include zero people from management, since management is small. Stratified sampling avoids this by deliberately drawing from each group: 7 from sales, 2 from engineering and 1 from management, matching the 70-20-10 split in the full population. This guarantees that every group is represented in roughly the right proportion, which simple random sampling does not guarantee on any single draw.

Cluster sampling

Instead of sampling individuals directly, you randomly select whole groups, or clusters, and then measure everyone or everything within the chosen clusters. If a retail chain has 50 stores, you might randomly pick 5 stores and survey every customer who visits those 5 stores that week, rather than trying to survey a random customer across all 50 stores. This is often far cheaper logistically, since you only need staff in 5 locations instead of 50, but it works best when the clusters are fairly similar to each other. If the 5 chosen stores happen to be in wealthier neighborhoods than average, the sample will be biased even though the stores were chosen randomly.

Convenience sampling

You sample whoever is easiest to reach: people walking past you on the street, the first 100 rows in a spreadsheet, users who happen to respond quickly to an email. This is cheap and fast, which is why it is used constantly in practice, but it carries no guarantee of representativeness at all. It is the method most likely to produce the kind of bias discussed in the next section.

Sampling bias: when the sample systematically misrepresents the population

Sampling bias happens when the way a sample is collected makes certain members of the population more or less likely to be included, in a way that is connected to the thing you are trying to measure. The key word is systematic. Random variation from sample to sample is expected and manageable; a systematic tilt toward one kind of answer is not, and it does not go away no matter how carefully you compute statistics afterward.

A well documented historical example is a 1936 United States presidential poll conducted by a magazine, which mailed questionnaires using names drawn from telephone directories and car registration lists. At the time, owning a telephone or a car was strongly associated with being wealthier than the average voter, so the sampling frame itself, before any question was even asked, already leaned toward one segment of the population. The poll's prediction ended up badly wrong relative to the actual election result. The lesson is not about politics, it is about method: a very large sample size does not protect you if the sampling frame itself excludes or underrepresents part of the population you care about.

A small worked example of bias versus randomness

To see the effect in numbers rather than history, imagine a population of 12 students with these exam scores out of 100.

🐍Python
population = [55, 60, 62, 65, 68, 70, 72, 75, 78, 82, 88, 95]
population_mean = sum(population) / len(population)

# a biased sample: only the five highest scorers
biased_sample = [75, 78, 82, 88, 95]
biased_mean = sum(biased_sample) / len(biased_sample)

# a more spread-out sample, closer to simple random sampling
spread_sample = [60, 65, 75, 82, 95]
spread_mean = sum(spread_sample) / len(spread_sample)

print("population mean:", population_mean)
print("biased sample mean:", biased_mean)
print("spread sample mean:", spread_mean)

This code assumes nothing beyond standard Python, there is no library to install. Running it gives a population mean of 72.5. The biased sample, built from only the five highest scorers, gives a mean of 83.6, almost 11.1 points higher than the true population value. The second sample, which deliberately spreads across the low, middle and high scores, gives a mean of 75.4, much closer to 72.5. Both samples have exactly 5 of the 12 students, so sample size is not the difference here. The difference is entirely about which students were included. The first sample systematically overestimates because it only ever looks at the top of the distribution, no matter how many times you recompute the mean from that same group.

This is the essential signature of bias: it is a property of how the data was collected, not of how much data you have, and it is also a property of which values are left out, something that connects back to part 8 on outliers. A biased sample is not the same thing as a sample full of outliers, but both share the trait that the numbers you see do not tell the whole story unless you know what was excluded before you ever opened the file.

Common types of sampling bias

Sampling bias shows up under several names depending on how it happens. Recognising the pattern matters more than memorising the label.

  • Selection bias: the method of choosing who or what enters the sample favors certain outcomes, as in the exam score example where only top scorers were chosen.
  • Undercoverage: part of the population has no chance, or a much lower chance, of being included, as with the email list that silently drops users whose messages go to spam.
  • Non-response bias: everyone had a chance to be included, but the people who actually respond differ systematically from those who do not. A customer satisfaction survey sent to every customer still suffers from this if only the very happy and the very angry customers bother to fill it in, while the mildly satisfied majority stays silent.
  • Voluntary response bias: a close relative of non-response bias, where people opt themselves into the sample, such as online product reviews or call-in polls. People with strong opinions, usually at the extremes, are far more likely to volunteer than people with an average, unremarkable experience.
  • Survivorship bias: the sample only includes things that made it to the point of being observed, missing the things that dropped out earlier. Studying only companies that are still operating today to learn what makes a company successful ignores every company that followed similar strategies and failed, which can make risky strategies look safer than they are.
  • Convenience bias: sampling whoever or whatever is easiest to reach, such as only surveying people who walk past a particular table in a particular mall at a particular time of day, which quietly limits the sample to one kind of location, schedule and demographic.

Bigger samples do not fix bias

It is tempting to think that collecting more data solves these problems, and for one specific kind of problem it does. A simple random sample's mean will tend to get closer to the true population mean, and bounce around less from sample to sample, as the sample size grows. This is the intuition behind what is usually called the law of large numbers: random noise in a sample tends to average out as you add more observations, because overestimates and underestimates from different individuals tend to cancel each other.

But bias is not random noise, it is a systematic tilt, and averaging more of the same tilted data does not cancel a tilt, it just repeats it more confidently. Going back to the exam score example, imagine instead of 5 top scorers you took the top 50, or the top 5,000, from a much larger population of students, always选择ing only the highest scorers available. The sample mean would still sit above the true population mean, consistently, no matter how large that biased sample grew. A useful mental picture is a kitchen scale that is miscalibrated to always read 2 kilograms heavier than the true weight. Weighing more items on that same broken scale does not reveal the true weight of anything, it just produces more readings that are all wrong in the same direction. Fixing bias requires fixing how the sample is collected, not collecting more of it.

Where this shows up in everyday data work

Sampling and its biases are not just a concern for survey statisticians, they come up constantly in ordinary data science and analytics work.

  • A/B testing: if one version of a webpage is only shown to users who already opted into early-access features, the comparison between versions is contaminated by who got assigned to which group, not just by the webpage design itself.
  • Logged or app data: analytics collected only from users who still have the app installed miss everyone who already uninstalled it, which is a direct form of survivorship bias when you are trying to understand dissatisfaction.
  • Training data for machine learning models: a model is only as representative as the data it was trained on. A loan approval model trained only on applicants who were previously approved, because only they have a known repayment outcome, never learns from the people who were rejected and might have repaid fine. This is a well known form of selection bias in credit modelling, and no amount of additional approved-applicant data fixes it, echoing the point from the previous section.
  • Dashboard metrics: a daily active users count drawn from users in a single time zone's working hours systematically undercounts users elsewhere, even if the underlying database logs every user in the world.

In every one of these cases, the fix is the same first step used throughout this part: before trusting a mean, a correlation, or a model's accuracy, ask what the sampling frame was and who or what had no chance of being included.

Common mistakes

  • Treating a convenience sample as if it were a random sample, simply because it was easy to collect and nobody questioned it.
  • Assuming a larger sample automatically means a more trustworthy conclusion, when the real issue is how the sample was selected, not how big it is.
  • Using the sample mean and the population mean interchangeably in writing or in code, which hides from readers whether a number is a known fact about everyone or an estimate from a subset.
  • Ignoring non-response entirely, for example reporting survey results based only on the people who answered without mentioning what fraction of those invited actually responded.
  • Forgetting that a sample collected for one purpose may be badly suited to a different question. A dataset of customers who called support is a fine sample for studying support call patterns, but a poor sample for studying overall customer satisfaction, because it systematically contains people who had a problem.

Summary and what is next

A population is everything you want to know about, and a sample is the subset you actually measure. Sampling is necessary for practical reasons of cost, time and feasibility, but it only produces trustworthy estimates when the sample is representative of the population, which depends entirely on how it was collected: simple random sampling, systematic sampling, stratified sampling and cluster sampling all try to achieve this in different ways, while convenience sampling usually does not. Sampling bias, in its many forms including selection bias, undercoverage, non-response bias, voluntary response bias and survivorship bias, is a systematic distortion that more data cannot fix, unlike ordinary random variation which does shrink as a sample grows. Every statistic covered earlier in this path, the mean, the spread, the percentiles, the correlation, inherits whatever bias was baked into the sample before a single calculation was done.

With populations and samples now in place, the path moves from concepts to practice. Part 11 puts everything from parts 1 through 10 to work on a real dataset using pandas, computing and interpreting these statistics directly in code. Part 12 then builds a full statistical profile of a city's housing prices, bringing the whole path together in one project.