Recap and goal

Over the last ten parts of this path you built a vocabulary for describing data without touching a single line of code: types of data, centre, spread, percentiles, distribution shape, the normal distribution, outliers, correlation, and the difference between a population and a sample. This part is where that vocabulary turns into working code. You will load one real dataset into pandas and compute, in order, everything the earlier parts taught you to look for. By the end you will have a small, reusable routine for describing any table of numbers, and in Part 12 you will scale that routine up into a full project on housing prices.

The dataset for this session is the iris flower dataset, originally measured by the botanist Edgar Anderson and made famous by the statistician Ronald Fisher in a 1936 paper on classification. It records four measurements, in centimetres, from 150 iris flowers belonging to three species. It is small, it is real, and it ships with scikit-learn, so you can load it without downloading anything from the internet. That matters for a hands-on article: every reader should get the exact same numbers.

Setup: the libraries and the dataset

You need four libraries: pandas for the table itself, numpy for a couple of numeric checks, matplotlib for plots, and scikit-learn only to fetch the bundled iris data (you will not do any machine learning with it). These examples were written against pandas 2.x, numpy 1.x, matplotlib 3.x and scikit-learn 1.x. The exact minor version of each rarely matters for the code shown here, but if a method name has changed in a much newer release, check that library's changelog.

💻Bash
pip install pandas numpy matplotlib scikit-learn

If you are working in a Jupyter notebook, run that command in a terminal first, then restart the kernel if the libraries were just installed. From here on, run the code blocks in order inside the same notebook or script. Later blocks assume the dataframe called df, created in the next step, is still in memory.

Loading a real dataset into a DataFrame

🐍Python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris

iris = load_iris()

df = pd.DataFrame(iris.data, columns=iris.feature_names)
df.columns = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
df['species'] = pd.Categorical.from_codes(iris.target, iris.target_names)

print(df.head())

load_iris() returns a scikit-learn Bunch object, which is a dictionary-like container. Its data attribute is a numpy array with 150 rows and 4 columns, and feature_names holds the original column labels, which look like 'sepal length (cm)'. The code above renames them to simple, code-friendly names and adds a species column built with pd.Categorical.from_codes, which turns the numeric labels 0, 1 and 2 into the actual species names using iris.target_names. This is the same categorical data type you met in Part 2: species is a nominal, unordered category, while the four measurement columns are continuous.

If you run this, df.head() shows the first five rows, and all five say setosa. That is not a coincidence or a bug: the iris dataset is stored grouped by class, with the 50 setosa rows first, then the 50 versicolor rows, then the 50 virginica rows. Keep that ordering in mind, because it means you cannot assume a random sample just by taking the first few rows of this particular file, a reminder of the sampling ideas from Part 10.

First look: shape, types and missing values

🐍Python
print(df.shape)
df.info()
print(df.dtypes)
print(df.isna().sum())

df.shape returns (150, 5): 150 rows, one per flower, and 5 columns, four measurements plus species. df.info() prints each column's name, how many non-null values it has, and its dtype. You should see four columns of dtype float64 and one of dtype category. df.dtypes gives the same type information in a tidier form if you only need that.

df.isna().sum() counts missing values in each column. For this dataset every count should be zero, because the file was curated for teaching, but checking is a habit worth keeping even when you expect a clean answer. Real datasets you meet later in this path, and certainly the housing dataset in Part 12, will not be this tidy, and a silent NaN can quietly change a mean or a correlation without raising an error.

Counting a categorical column

🐍Python
print(df['species'].value_counts())
print(df['species'].value_counts(normalize=True))

value_counts() builds a frequency table: how many rows fall into each category. For species you should see exactly 50 for setosa, 50 for versicolor and 50 for virginica, which is a well-known and deliberate property of this dataset, three perfectly balanced groups. Passing normalize=True turns the counts into proportions, so each species shows roughly 0.333, or one third. This is the categorical-data counterpart of the mean: instead of an average value, you get the share of each label, exactly the kind of summary Part 2 said was appropriate for nominal data, where mean and median do not make sense.

Centre: mean, median and mode in pandas

🐍Python
col = 'sepal_width'
print('mean:', df[col].mean())
print('median:', df[col].median())
print('mode:', df[col].mode().tolist())

mean(), median() and mode() do exactly what Part 3 described. The mean adds every value and divides by the count. The median sorts the values and takes the middle one (or the average of the two middle ones, for an even count). The mode is the most frequent value or values, which is why mode() returns a small Series rather than a single number: a column can have more than one tied mode. For sepal_width you should find the mean and median sitting close together, both a little above 3.0 cm, which is a sign of a roughly symmetric distribution for that column, consistent with what you will see in its histogram later in this article.

Now repeat the comparison on a column where mean and median disagree more noticeably.

🐍Python
col = 'petal_length'
print('mean:', df[col].mean())
print('median:', df[col].median())
print(df[col].describe())

For petal_length the mean comes out close to 3.76 cm, while the median is noticeably higher, close to 4.35 cm. Part 3 taught you to read a mean-median gap as a clue about skew. Here the full explanation is more interesting than plain skew: petal length is not one smooth distribution, it is two distinct clusters sitting side by side. The 50 setosa flowers have short petals, roughly between 1.0 and 1.9 cm, and the 100 versicolor and virginica flowers have much longer petals, roughly between 3.0 and 6.9 cm, with a gap in between where almost no flowers fall. Averaging across that gap produces a mean that does not describe any real flower well, a direct illustration of the warning in Part 10 about describing a mixture of subpopulations as if it were one population.

Spread: variance, standard deviation, range and IQR

🐍Python
col = 'sepal_length'
print('variance:', df[col].var())
print('std:', df[col].std())
print('range:', df[col].max() - df[col].min())

q1 = df[col].quantile(0.25)
q3 = df[col].quantile(0.75)
print('IQR:', q3 - q1)

var() and std() compute the variance and standard deviation exactly as Part 4 defined them: how far, on average, values sit from the mean, with the standard deviation expressed in the original units so it is directly comparable to the data. The range is simply the maximum minus the minimum, and the IQR, the interquartile range, is the 75th percentile minus the 25th percentile, the width of the middle half of the data that Part 4 and Part 5 both introduced as a spread measure that ignores extreme values.

There is one detail here that trips up a lot of people moving between numpy and pandas: the two libraries use different default formulas for variance and standard deviation.

🐍Python
sample = df[col].values

print('pandas std (default):', df[col].std())
print('numpy std (default):', np.std(sample))
print('numpy std with ddof=1:', np.std(sample, ddof=1))

pandas' std() and var() divide by n minus 1 by default, the sample formula explained in Part 4 and justified in Part 10 by the fact that a sample's own mean is slightly closer to its data points than the true population mean would be, which makes the plain n-divided version underestimate spread. numpy's std() and var(), by contrast, divide by n by default, the population formula. The two will give slightly different numbers unless you explicitly pass ddof=1 to numpy, which tells it to divide by n minus 1 instead. For 150 rows the difference is small, but for a dataset of 10 or 20 rows it can matter, and mixing the two libraries without noticing the default can produce a standard deviation that looks wrong in a report when it is really just using a different convention. Always check which default your tool is using, or set ddof explicitly so the choice is visible in your code rather than hidden in a default.

Percentiles, quartiles and a worked example

🐍Python
print(df['sepal_length'].quantile([0.1, 0.25, 0.5, 0.75, 0.9]))

quantile() accepts a single number between 0 and 1, or a list of them, and returns the value below which that fraction of the data falls, exactly the percentile idea from Part 5. Passing [0.1, 0.25, 0.5, 0.75, 0.9] gives you the 10th, 25th, 50th, 75th and 90th percentiles in one call. The 50th percentile is always the same number you get from median().

It is worth checking this against a manual calculation once, on numbers small enough to sort by hand, so the method never feels like a black box. The first 10 rows of this dataset, in the order they were loaded, have these sepal_length values: 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9.

🐍Python
sample10 = df['sepal_length'].head(10)
print('original order:', sample10.tolist())
print('sorted:', sorted(sample10.tolist()))
print('pandas median:', sample10.median())

Sorted, those ten values become 4.4, 4.6, 4.6, 4.7, 4.9, 4.9, 5.0, 5.0, 5.1, 5.4. With an even count of 10, the median is the average of the 5th and 6th sorted values, which are both 4.9, so the median is 4.9. That matches what sample10.median() reports. This is a small check, but doing it once by hand is the fastest way to trust the method when you later run it on a column of 30,000 rows that you cannot sort by eye.

The box plot: seeing spread and centre together

🐍Python
df.boxplot(column='sepal_length', by='species', grid=False)
plt.title('Sepal length by species')
plt.suptitle('')
plt.ylabel('cm')
plt.show()

This draws one box plot per species, side by side, for the sepal_length column. Part 5 explained how to read one: the box spans the IQR, from the 25th to the 75th percentile, the line inside the box marks the median, the whiskers extend to the most extreme points that are not flagged as outliers, and any point beyond the whiskers is drawn separately as a dot. With the plot split by species using the by argument, you can see at a glance that virginica tends to have the longest sepals, setosa the shortest, and versicolor sits in between, with the three boxes barely overlapping. A single overall box plot, without the species split, would hide this entirely and just describe one wide, misleading spread, another reminder of Part 10's lesson about subpopulations.

The all-in-one view: .describe()

🐍Python
print(df.describe())
📃Plain Text
       sepal_length  sepal_width  petal_length  petal_width
count        150.00       150.00        150.00       150.00
mean           5.84         3.05          3.76         1.20
std            0.83         0.43          1.76         0.76
min            4.30         2.00          1.00         0.10
25%            5.10         2.80          1.60         0.30
50%            5.80         3.00          4.35         1.30
75%            6.40         3.30          5.10         1.80
max            7.90         4.40          6.90         2.50

When you run describe() yourself you will see more decimal places, the table above rounds to two for readability, but the values should be close to these for every numeric column. This single call reproduces everything done by hand so far: count is how many non-missing values pandas found, mean and std are the centre and spread from Part 3 and Part 4, min and max give the range, and the 25%, 50% and 75% rows are exactly the quartiles from Part 5. Treat describe() as a shortcut once you understand what each row means, not as a replacement for understanding it.

You can change which percentiles appear.

🐍Python
print(df.describe(percentiles=[0.1, 0.9]))

This swaps the default 25% and 75% rows for 10% and 90%, while the 50% row, the median, always stays because pandas treats it as mandatory. Use this when the middle half of the data is less interesting to you than the tails, for instance when you specifically want to see how extreme the top and bottom 10 percent of values are.

describe() behaves differently on a categorical column.

🐍Python
print(df['species'].describe())

For species you get count, unique, top and freq instead of mean and std, because none of the arithmetic summaries make sense for a nominal category, exactly the distinction Part 2 drew between data types. unique should read 3, and freq will show 50. Because all three species are perfectly tied at 50 rows each, the top value you see is just whichever one pandas happens to list first internally, not a genuinely more common category. Whenever counts are this close, check value_counts() directly rather than trusting top and freq to mean

Shape of the distribution: histograms, skew and kurtosis

🐍Python
df.hist(column=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'], bins=20, figsize=(8, 6))
plt.tight_layout()
plt.show()

This draws a histogram for each of the four measurement columns, using 20 bins each. Look at the shapes rather than the exact bar heights. sepal_width should look close to a single bell-shaped bump, roughly what Part 6 called a symmetric, unimodal distribution. petal_length should look very different: two separate humps with a gap between them, one small hump around 1 to 2 cm and a wider one from about 3 to 7 cm. That shape is the bimodal pattern from Part 6, and it is exactly what produced the mean-median mismatch you saw earlier, caused by the setosa cluster sitting apart from the other two species.

🐍Python
numeric_cols = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
print(df[numeric_cols].skew())
print(df[numeric_cols].kurt())

skew() returns a single number per column measuring the asymmetry Part 6 described: close to zero for a roughly symmetric shape, positive for a right-hand tail, negative for a left-hand tail. kurt() returns a measure related to how heavy the tails are compared with a normal distribution, the idea previewed briefly before Part 7's deeper look at the normal distribution. Run both on this data and compare the numbers against the histograms you just drew rather than trusting a number on its own. petal_length is the column to watch closely: its skew value will be some single number, but that one number cannot tell you the column is bimodal, only that it leans one way on average. This is the practical version of a point statisticians often make with a famous constructed example called Anscombe's quartet, four small datasets that share almost identical means, variances and correlations while looking completely different when plotted. The lesson carries over directly here: a skew or kurtosis number is a useful summary, never a substitute for looking at the histogram.

Finding outliers with the IQR rule

🐍Python
col = 'sepal_width'
q1 = df[col].quantile(0.25)
q3 = df[col].quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr

print(f'Q1={q1}, Q3={q3}, IQR={iqr}')
print(f'Lower fence={lower}, Upper fence={upper}')

outliers = df[(df[col] < lower) | (df[col] > upper)]
print(outliers[[col, 'species']])

This is the IQR rule from Part 8, applied directly: anything more than 1.5 times the IQR below Q1 or above Q3 is flagged as a potential outlier. Using the quartile values from the describe() table above, Q1 for sepal_width is about 2.8 and Q3 is about 3.3, so the IQR is about 0.5. That puts the lower fence at roughly 2.05 and the upper fence at roughly 4.05. Notice something interesting: the minimum value in the whole column is 2.0, which is just below the lower fence of 2.05, so it gets flagged too, even though 2.0 is not an extreme or suspicious number for a flower's sepal width, it is simply the smallest value in an otherwise ordinary dataset. On the high end, the overall maximum of 4.4 cm sits comfortably above the upper fence and will also be flagged, along with possibly one or two nearby values.

This is exactly the situation Part 8 warned about: the IQR rule is a mechanical flag, not a verdict. Flagging a point is the start of a judgement call, not the end of one. Here, there is no reason to think these flowers were mismeasured or belong to a different dataset; they are simply the naturally occurring extremes of a real biological measurement. Removing them would not clean the data, it would just throw away true variation. Keep that distinction in mind whenever you run this rule on a new dataset: ask what an outlier means before deciding what to do with it.

Correlation between measurements

🐍Python
corr_matrix = df[numeric_cols].corr()
print(corr_matrix)

corr() computes the Pearson correlation coefficient, from Part 9, between every pair of numeric columns and lays the results out in a grid. The diagonal is always 1.0, because every column is perfectly correlated with itself. Off the diagonal, you should see petal_length and petal_width very strongly correlated, typically above 0.9, which makes biological sense: a flower with a longer petal tends to have a wider one too, since petals generally scale together as the flower grows. Petal measurements also correlate fairly strongly with sepal_length. sepal_width behaves differently: across the whole dataset it correlates weakly and slightly negatively with the other measurements.

That last result is worth pausing on, because it connects directly to Part 9's warning that correlation can mislead and to Part 10's lesson about populations and samples. Compute the same correlation separately within each species.

🐍Python
for species, group in df.groupby('species'):
    corr = group['sepal_length'].corr(group['sepal_width'])
    print(species, round(corr, 2))

When you run this, the correlation between sepal_length and sepal_width within each individual species comes out positive, often noticeably so, even though the overall correlation across all 150 flowers combined was weak and negative. Nothing contradictory is happening: each species occupies a different region of the sepal_length and sepal_width space, and when you combine all three regions into one scatter of points, the across-species pattern can point in a different direction from the within-species pattern. This exact iris example is used in a number of statistics courses precisely because it shows, concretely, that a single correlation number computed on mixed groups can describe a pattern that does not exist in any one of those groups. Whenever your data has a natural category like species, customer segment or region, always check whether grouping changes the correlation before reporting a single overall figure.

Comparing groups: groupby reveals hidden structure

🐍Python
group_means = df.groupby('species')[numeric_cols].mean()
print(group_means)

group_stds = df.groupby('species')[numeric_cols].std()
print(group_stds)

groupby('species') splits the dataframe into three smaller tables, one per species, and the chained .mean() computes the mean of each numeric column within each group separately. For petal_length you should see three clearly different averages: roughly 1.46 cm for setosa, roughly 4.26 cm for versicolor, and roughly 5.55 cm for virginica. Compare that to the single overall mean of about 3.76 cm from earlier, a number that does not sit close to any of the three real group averages. This is the concrete version of what Part 10 called treating a mixture of populations as if it were one: the overall figure describes an average flower that, in a sense, does not exist.

Look at the standard deviations too. Each species' own petal_length standard deviation is much smaller than the overall 1.76 cm computed earlier, often by a factor of three or four for setosa in particular, whose petals are quite uniform in size. That gap is the signature of mixed populations: spread computed across groups that differ in their centres will always look larger than the spread within any single group, because part of what you are measuring is not natural variation around one centre, it is the distance between different centres.

🐍Python
print(df.groupby('species')['petal_length'].describe())

Chaining describe() after a groupby produces the full count, mean, std, min, quartiles and max for each group in one table, rather than one column at a time. This is usually the fastest way to compare three or more groups on the same measurement, and it is the technique you will lean on most heavily in Part 12, where the groups will be neighbourhoods or property types instead of flower species.

Putting it together: a small descriptive-report function

Every technique in this article answers one question: what does this column, or this group of rows, actually look like? It is worth wrapping the common ones into a single function you can point at any new dataframe, so you never have to retype the same six or seven lines from scratch.

🐍Python
def describe_dataset(data: pd.DataFrame, group_col: str = None) -> None:
    print('Shape:', data.shape)
    print()

    print('Missing values per column:')
    print(data.isna().sum())
    print()

    categorical_cols = data.select_dtypes(include=['object', 'category']).columns
    numeric_cols = data.select_dtypes(include=['number']).columns

    for col in categorical_cols:
        print(f'--- {col} (categorical) ---')
        print(data[col].value_counts())
        print()

    if len(numeric_cols) > 0:
        print('--- Numeric summary ---')
        print(data[numeric_cols].describe())
        print()

        print('Skew (watch for values far from zero, and check the histogram too):')
        print(data[numeric_cols].skew())
        print()

        print('Correlation matrix:')
        print(data[numeric_cols].corr())
        print()

    if group_col is not None and len(numeric_cols) > 0:
        print(f'--- Mean of numeric columns, grouped by {group_col} ---')
        print(data.groupby(group_col)[numeric_cols].mean())


describe_dataset(df, group_col='species')

This function does not introduce anything new, it simply runs the steps from this article in a fixed order: shape, missing values, category counts, numeric summary, skew, correlation, and group means if you tell it which column holds the groups. Point it at any dataframe with a mix of categorical and numeric columns and you get a quick first orientation before you decide what to investigate further. It is intentionally simple: it prints to the screen rather than returning a tidy object, and it does not try to guess which columns matter most. Treat it as a starting template, not a finished tool. In Part 12 you will extend an approach very like this one to a housing dataset, adding specific questions about price and location on top of this general scaffold.

Common mistakes when describing data with pandas

  • Trusting describe() without ever plotting the data. A skew value near zero or a tidy-looking mean can hide a bimodal shape, as petal_length showed here. Always pair a numeric summary with at least a histogram or a box plot.
  • Mixing up pandas and numpy standard deviation defaults. pandas' std() and var() use n minus 1 by default, matching the sample formula from Part 4 and Part 10. numpy's std() and var() use plain n by default. Set ddof explicitly when it matters, especially on small samples.
  • Running mean() or std() on a numeric-looking column that is really a category. If species had been left as 0, 1 and 2 instead of converted to a category, calling df['species'].mean() would run without error and produce a meaningless number, an average species code.
  • Treating top and freq from describe() as proof of a dominant category when groups are closely tied. Check value_counts() directly whenever the counts could plausibly be close.
  • Computing one overall mean, standard deviation or correlation across rows that actually belong to different groups, then reporting it as if it described a single population. Always check whether splitting by a known category, like species here, changes the picture, as it did for the sepal_length and sepal_width correlation.
  • Reporting statistics to more decimal places than the sample size justifies. A mean computed from 150 rows, or worse from 10, does not become more trustworthy by printing eight decimal digits; round to a sensible precision for the audience.

Summary and what's next

You took a real, publicly available dataset and worked through every idea from Parts 2 to 10 using pandas: value_counts() for categorical frequencies, mean(), median() and mode() for centre, var(), std(), range and quantile() for spread, describe() as a shortcut once you understand its parts, hist(), skew() and kurt() for shape, a manual IQR calculation for outliers, corr() for relationships between columns, and groupby() to see how all of the above can change once you account for real subgroups in the data. Along the way you saw two concrete traps that matter well beyond this dataset: the ddof default mismatch between pandas and numpy, and the way mixing distinct subpopulations can produce a mean, a spread or a correlation that describes none of the groups it was computed from.

Part 12 is the project that closes out this path: a full statistical profile of a city's housing prices, built with the same tools you just practised here, but applied to messier, real-world data where missing values, skew and outliers will need actual decisions rather than tidy classroom answers.