Polars Under the Hood: Why It Beats pandas, and How to Use It on Real Pipelines

Every data team has the same afternoon at some point. A notebook that ran fine on 200,000 rows is pointed at 200 million. The fan spins up, the kernel dies, and somebody says, "we need a bigger machine." Sometimes that is true. More often the machine was never the problem. The tool was doing far more work than the question needed, on one core, with three copies of the data sitting in memory.

This is Part 3 of the MLHub series. Part 1 covered agents and MCP, and Part 2 covered LoRA and QLoRA fine-tuning. Both of those are about the exciting middle of a machine learning system. This part is about the unglamorous edges: the code that reads, cleans, joins and reshapes data before a model ever sees it, and the code that turns model output back into tables. In most real projects that code is where the time and the money go.

Polars is a DataFrame library written in Rust, and it has become the default answer to "pandas is too slow." But "it's written in Rust" is not an explanation, and it is certainly not a reason to rewrite a working pipeline. So in this article we will go from the bottom up. First, why pandas struggles. Then what Polars actually does differently: the memory layout, the CPU tricks, the threading model and the query optimiser. Then a hands-on pipeline in both libraries side by side, a migration cheat-sheet, a look at the wider Rust-flavoured toolchain (DuckDB, uv, Ruff and a tiny PyO3 extension), and finally the boring-but-vital production parts: Parquet layout, schema checks, testing, benchmarking and the ways pipelines break.

Everything here is runnable. The dataset is generated with NumPy inside the article, and the two timing scripts are for you to run on your own hardware. I will not quote benchmark numbers as facts, becuase your CPU, your file layout and your pandas version will change them. Where I show numbers or plan output, I will say so.

Code in this article targets Python 3.12, Polars 1.x, pandas 2.x and NumPy 2.x. Polars still moves quickly, so if a method name looks different on your machine, check the Polars documentation for your installed version.

Why pandas hits a wall

pandas is a wonderful library, and it is worth saying so before listing its limits. It made Python a serious language for data work. The API is expressive, the ecosystem is enormous, and for data that fits comfortably in memory it is often fast enough. The limits below are not bugs. They are consequences of design choices made in a different decade, for different hardware.

Most operations run on one core

The core of pandas leans on NumPy and on Cython. NumPy arithmetic is fast, but it runs on a single thread. A groupby in pandas hashes the keys, splits the rows and aggregates them, mostly on one core. Your laptop may have 8 or 16 cores, and a pandas job will typically keep one of them busy while the rest idle. Some operations release the GIL and some libraries add parallelism on top, but as a general rule pandas does not spread one query over all your cores.

Think of a supermarket with sixteen checkout lanes where every customer is sent to lane one. Lane one is very well run. It is still lane one.

Eager evaluation

In pandas every line runs immediately and produces a full result. Consider this chain:

df = pd.read_csv("events.csv")
df = df[df["status"] == "delivered"]
df = df[["city", "subtotal"]]
totals = df.groupby("city")["subtotal"].sum()

By the time the filter runs, the whole file has already been read, every column of it. The filter creates a new frame. The column selection creates another. pandas never gets to notice that you only needed two columns and one status, because nobody told it the full plan up front. Each step is executed as written, and each step's output has to be built in full.

This is like a cook who buys every ingredient in the shop, carries them home, and only then reads the recipe.

Strings as Python objects

Numeric columns in pandas are compact NumPy arrays. String columns, historically, are not. The default object dtype is an array of pointers, and each pointer leads to a separate Python str object somewhere else on the heap. A million short strings means a million small allocations, each carrying object header overhead, scattered across memory. Scanning them means chasing pointers, which the CPU hates. Operations like .str.lower() run a Python-level loop over those objects.

pandas has been moving away from this. The nullable and Arrow-backed dtypes (dtype_backend="pyarrow", or pd.ArrowDtype) have been available for a while, and pandas 3.0 moves the defaults towards a dedicated string dtype and Copy-on-Write. If you are on a recent version and your strings are Arrow-backed, some of this section is softer than it used to be. Check df.dtypes on your own data rather than assuming.

Copies, copies, copies

Many pandas operations return a new frame. Some return a view, some return a copy, and historically it was hard to tell which without reading the docs and the source. That ambiguity is where the infamous SettingWithCopyWarning came from. Copy-on-Write, the default in recent versions, makes behaviour predictable, but it does not make the copies free.

This is why people quote the rule of thumb from pandas' creator, Wes McKinney, who wrote in 2017 that you should have roughly five to ten times as much RAM as the size of your dataset. It is a rule of thumb, not a law. A file that is 2 GB on disk as compressed CSV might be 6 GB as parsed columns, and the intermediate frames in a long chain can multiply that again. You can hit it with data that "should" fit.

The mental model that ties this together

Put those four together and you get the shape of the wall. One core does the work. Every step materialises its result. Strings are slow. Memory use is a multiple of the data. None of these are fatal on small data. All of them scale badly.

Polars was designed from the start against these four points. It uses all cores, it can plan before it executes, it stores strings in a columnar format, and it tries hard not to copy. To see how, we need to look at memory.

Apache Arrow: how columnar memory is laid out

Polars stores data in a layout that follows the Apache Arrow columnar format. Arrow is a specification for how to lay out a table in memory so that different tools (Polars, DuckDB, pandas with PyArrow, Spark, and many more) can share it without converting.

The word "columnar" means each column is stored as its own contiguous block. A row-oriented layout, which is what a Python list of dicts or a database row store gives you, keeps all the fields of one record together and then the next record after it.

Row-oriented (each record together):
  [id=1, city=Berlin, fare=12.5] [id=2, city=Pune, fare=8.0] [id=3, city=Berlin, fare=21.0]

Column-oriented (each column together):
  id   : [1, 2, 3]
  city : [Berlin, Pune, Berlin]
  fare : [12.5, 8.0, 21.0]

Think of a filing cabinet. The row layout is one folder per customer with all their paperwork inside. The column layout is one drawer per kind of paperwork: all the invoices in one drawer, all the addresses in another. If your question is "what is the total of all invoices," the second cabinet lets you open one drawer and never touch the rest.

What a column actually is

An Arrow column is a small number of buffers, each one a plain block of bytes.

For a fixed-width type such as a 64-bit float, there are two:

1. A values buffer: the numbers, back to back, 8 bytes each.
2. A validity bitmap: one bit per row, where 1 means "present" and 0 means "null."

That bitmap is how Arrow represents missing data. There is no special sentinel value, and no need to turn an integer column into floats just to hold a NaN. A column of one million integers with some nulls costs 8 MB of values plus 125 KB of bitmap. If a column has no nulls, the bitmap can be left out entirely.

values   : [ 12.5 ][ 0.0 ][ 21.0 ][ 8.0 ]        (4 x 8 bytes, contiguous)
validity : 1 0 1 1                               (one bit per row; row 1 is null)

For strings, Arrow uses three buffers in its classic layout:

1. The validity bitmap.
2. An offsets buffer: for each row, where its string starts in the data buffer (and the last entry marks the end).
3. A data buffer: all the string bytes glued together in one block.

values  : "Berlin", "Pune", "Berlin"
data    : B e r l i n P u n e B e r l i n
offsets : 0, 6, 10, 16

To read row 1, look at offsets 1 and 2 (6 and 10), and take bytes 6 to 10 from the data buffer. No pointer chasing, no per-string object. Three strings, one block.

Polars goes a step further than the classic layout. Modern Polars uses Arrow's string view variant, sometimes called "German strings" after the research system that introduced the idea. Each string gets a fixed 16-byte view. Short strings (12 bytes or fewer) live directly inside the view, and longer ones store a 4-byte prefix plus a pointer into a buffer. That makes comparisons and filters faster, because you can often decide "not equal" from the inline prefix without following anything. The offsets picture above is still the best mental model. Just know that the production layout has a few extra tricks.

Why this layout is fast: a worked example

Take a table of 1,000,000 rows and 8 columns, all 64-bit floats. Each row is 8 x 8 = 64 bytes. Now ask for the sum of one column.

A CPU does not read memory a byte at a time. It reads in cache lines, typically 64 bytes. In the row layout, the value you want is 8 bytes out of every 64-byte row. To read it, the CPU must fetch the whole cache line, and 56 bytes of that line are columns you did not ask for. Across a million rows that is about 64 MB of memory traffic to use 8 MB of useful data.

In the column layout, the column you want is a contiguous 8 MB block. Every cache line the CPU fetches contains eight useful numbers. Total traffic is about 8 MB. That is roughly an 8x reduction in memory movement for this query, and scans like this are usually limited by memory bandwidth, not arithmetic.

There is a second win. Because the values are contiguous and the same type, the CPU can use SIMD (single instruction, multiple data). A modern x86 core with AVX2 can add four 64-bit floats in a single instruction. With AVX-512 it is eight. ARM NEON handles two. Compilers and libraries generate this code automatically when the loop is simple, uniform and over contiguous memory, which is exactly what a columnar aggregation is. In the row layout the values are 64 bytes apart, and the CPU must gather them one at a time.

A third, smaller benefit: same-typed neighbours compress well. Runs of similar numbers, or repeated strings, compress far better than mixed record data. This matters when we get to Parquet.

The specific ratios depend on the CPU, so I will not promise you a number. But you can measure the effect yourself in about forty lines of NumPy.

Try it: rows versus columns in NumPy

This script builds the same 8-column table three ways: a Python list of dicts (the "obviously row-oriented" version), a NumPy structured array (records packed together in memory, so a row layout), and eight separate NumPy arrays (a columnar layout). Then it sums one column from each and times it, using warm-up runs and the median of several repeats.

import statistics
import time

import numpy as np

N = 1_000_000          # rows for the NumPy layouts
N_PY = 200_000         # list of dicts is memory hungry, so use fewer rows
NAMES = [f"c{i}" for i in range(8)]

rng = np.random.default_rng(0)

# 1. Columnar: one contiguous float64 array per column.
columns = {name: rng.random(N) for name in NAMES}

# 2. Row-oriented: one structured array; each record is 64 contiguous bytes.
rows = np.empty(N, dtype=[(name, "f8") for name in NAMES])
for name in NAMES:
    rows[name] = columns[name]

# 3. Python objects: a list of dicts.
records = [
    {name: float(columns[name][i]) for name in NAMES}
    for i in range(N_PY)
]


def bench(fn, repeat=7, warmup=2):
    """Median wall-clock seconds for fn(), after warm-up runs."""
    for _ in range(warmup):
        fn()
    samples = []
    for _ in range(repeat):
        t0 = time.perf_counter()
        fn()
        samples.append(time.perf_counter() - t0)
    return statistics.median(samples)


t_col = bench(lambda: columns["c3"].sum())
t_row = bench(lambda: rows["c3"].sum())
t_py = bench(lambda: sum(r["c3"] for r in records))

print(f"columnar numpy   : {t_col / N * 1e9:8.2f} ns per row")
print(f"structured numpy : {t_row / N * 1e9:8.2f} ns per row")
print(f"list of dicts    : {t_py / N_PY * 1e9:8.2f} ns per row")

# Sanity check: all three layouts hold the same numbers.
assert np.isclose(columns["c3"][:N_PY].sum(), sum(r["c3"] for r in records))

I am deliberately not printing sample output, since anything I typed here would be invented. What you should see on almost any machine is the ordering: the contiguous column is fastest, the structured array (same numbers, strided access) is slower, and the list of dicts is dramatically slower per row. If you want to go one level deeper, change 8 columns to 32 and watch the gap between the first two rows widen, because each record gets longer while your column stays the same size in useful bytes.

Two honest caveats. NumPy's structured-array sum is not the only way to write a row store, and a database engine with a row layout can be very clever. And for tiny data, all of this vanishes into the noise. The point is direction, not a leaderboard.

Using every core: Rust, rayon-style work stealing and the GIL

Columnar memory gives each core less to read. The next step is to use all the cores at once.

Polars is written in Rust and does its parallel work inside Rust code, on its own thread pool built on rayon, a Rust data-parallelism library. Two consequences follow. First, Polars is not bound by Python's GIL for its heavy lifting, because the work happens outside the interpreter and the GIL is released while it runs. Second, Rust's ownership rules make it far safer to write parallel code that touches shared data, which is a big reason a small team could build a parallel engine quickly without a swamp of race conditions.

Work stealing in plain English

Suppose you have 16 cores and a big task to split up. The naive approach is to cut the work into 16 equal pieces and hand one to each core. The trouble is that "equal" rarely means equal in time. One piece hits a cold cache or a bunch of long strings, and everyone else waits for it.

Work stealing fixes this. Cut the job into many small tasks and give each worker its own queue. A worker takes tasks from its own queue while it has some. When a worker runs dry, it looks at a busy neighbour and steals a task from the back of that neighbour's queue. Nobody sits idle while there is work anywhere.

The picture is checkout lanes again, but this time the cashiers are allowed to walk over to a long line and take a few customers when their own line is empty.

Where the parallelism comes from

Polars finds parallelism in three places.

1. Across expressions. In df.select(a_expr, b_expr, c_expr) or with_columns(...), expressions that do not depend on each other run in parallel.
2. Inside an operation. A group_by splits the rows by hash of the key into partitions, and each thread builds and aggregates its own hash tables. Sorts, joins and scans of big files are likewise split into chunks.
3. Across files and row groups. Reading a Parquet dataset with many files, or a file with many row groups, decodes those pieces on different threads.

You can see the size of the pool with pl.thread_pool_size(), and you can cap it with the POLARS_MAX_THREADS environment variable, which must be set before Polars is imported. That variable matters more than it sounds: when you benchmark, or when you run inside a container with a CPU quota, you want the thread count to match the cores you actually own.

The catch: Python callbacks

The one thing that drags Polars back to single-threaded speed is calling Python for each row. map_elements (Polars' apply) runs your Python function once per value, and the GIL forces that to happen one call at a time. It also blocks the optimiser from reasoning about your code, because it cannot see inside a Python function. We will come back to this in the migration section. The short version is that if you write a lambda, you have left the fast path.

The expression API: describe the computation, don't perform it

Polars has one central idea in its API: the expression. pl.col("subtotal") * 1.2 does not compute anything. It is a small tree that says "take the column named subtotal, multiply by 1.2." You hand such trees to a context (select, filter, with_columns, group_by().agg()), and the engine decides how to run them.

import polars as pl

expr = (pl.col("subtotal") * 1.2).alias("with_tax")
print(expr)

That prints the expression tree rather than any data. This has three practical payoffs:

• Expressions are composable. You can store them in variables, build lists of them, and reuse them across queries.
• They are inspectable. The engine can rewrite col("a") + 0 or fold 2 * 3 before running.
• They are parallelisable. Independent expressions have no side effects, so they can be scheduled on any thread.

A useful habit: whenever you reach for a Python loop or apply, ask whether there is an expression for it. There usually is, and it is usually quite a bit faster.

The lazy engine: logical plan, optimiser, physical plan

Eager mode (pl.read_parquet(...), then method after method) runs each step as you call it, the way pandas does. Lazy mode does not. You call pl.scan_parquet(...) or df.lazy(), and every method afterwards just adds a node to a logical plan. Nothing is read or computed until you call .collect().

The pipeline then has three stages:

1. Logical plan. A tree describing what you asked for: scan this file, filter these rows, join with that table, group and aggregate.
2. Optimiser. Rewrites the tree into an equivalent one that does less work.
3. Physical plan. The optimised tree is turned into concrete operators (a hash join, a parallel group-by, a Parquet reader with pushed-down filters) and executed across threads.

What the optimiser does

Here are the rewrites that matter most in day-to-day work.

Projection pushdown. Only the columns that the final result needs are ever read. If your file has 60 columns and the query touches four of them, the Parquet reader decodes four columns. Because Parquet is columnar on disk too, the bytes for the other 56 are never even read from storage. Think of asking a librarian for three pages of a book and having them photocopy exactly those, rather than lugging the whole book to your desk.

Predicate pushdown. Filters are moved as close to the source as possible. Instead of loading everything and then filtering, the scan itself skips data that cannot match. With Parquet, the reader consults per-row-group min/max statistics and skips whole row groups that cannot contain a match. With Hive-style partitioned directories, it skips whole folders. This is like telling the warehouse "only send me the red boxes" rather than receiving every box and sorting them on your own floor.

Slice pushdown. A .head(10) at the end can stop the scan early.

Common subexpression elimination (CSE). If the same expensive expression appears several times (say, the same window computation used in three output columns), Polars computes it once and reuses the result. Plans that read the same source twice can also share that scan.

Simplification. Constant folding, removal of redundant casts, and rewriting of expressions into cheaper equivalents.

None of these are magic. A careful person writing pandas by hand could do most of them too, by selecting columns early and filtering first. The difference is that the optimiser does it every time, for every query, without anyone remembering to.

Reading a plan with .explain()

You can print the optimised plan of any lazy query. Later in this article we will build a real query and call explain() on it. As a preview, the plan for that query looks roughly like the following. The exact text differs between Polars versions, so treat this as an illustration of shape, not a byte-exact copy:

SORT BY [col("revenue")]
  AGGREGATE
    [len().alias("orders"), col("subtotal").sum().alias("revenue"), col("delivery_min").median().alias("median_minutes")] BY [col("city"), col("cuisine")]
    FROM
    LEFT JOIN:
    LEFT PLAN ON: [col("restaurant_id")]
      Parquet SCAN [orders.parquet]
      PROJECT 5/10 COLUMNS
      SELECTION: [(col("status")) == ("delivered")]
    RIGHT PLAN ON: [col("restaurant_id")]
      Parquet SCAN [restaurants.parquet]
      PROJECT 2/3 COLUMNS

Three phrases are worth learning to spot. PROJECT 5/10 COLUMNS means projection pushdown worked: only 5 of the 10 columns are read. SELECTION attached to the scan means predicate pushdown worked: the filter travels with the reader. And if you ever see a FILTER node above a scan instead, something blocked the pushdown, often a Python callback in the filter expression.

Streaming: data larger than memory

A normal .collect() builds the whole result in RAM, and intermediate results can be larger than the final one. For data that does not fit, Polars has a streaming engine. Instead of processing whole columns at once, it processes the data in batches, and it keeps only running state (such as hash tables of group totals) in memory.

result = lazy_query.collect(engine="streaming")

Older Polars releases used collect(streaming=True) for a previous implementation. The newer streaming engine is the one you should reach for on current versions, and it has been improving release by release. Two honest notes: not every operation is supported in streaming mode yet, and unsupported parts fall back to the in-memory engine, so a plan that "should" stream may still spike memory. Check the plan and watch the memory graph the first time you run a new query on big data.

The other tool for huge outputs is sink_parquet. It runs the query and writes the result straight to disk in batches, never holding the full result in memory:

(
    pl.scan_parquet("raw/*.parquet")
    .filter(pl.col("status") == "delivered")
    .select("order_id", "city", "subtotal")
    .sink_parquet("clean/delivered.parquet", compression="zstd")
)

Read that pipeline as a conveyor belt: read a batch, filter it, write it, and take the next one. It can process a dataset many times larger than RAM on a modest machine.

Hands-on: a food-delivery pipeline in pandas and Polars

Enough theory. Let's build something. We will simulate a food-delivery marketplace with two tables: two million orders accross five cities and 5,000 restaurants, plus a restaurants table with cuisine and rating. The data is synthetic, and I have made it a bit realistic on purpose: a few restaurants get most of the orders (skew), some delivery times are missing, and customers write messy notes.

Generate the data once

Install what you need first:

pip install "polars>=1.0" "pandas>=2.0" numpy pyarrow

Then run this script. It writes two Parquet files that both libraries will read.

import numpy as np
import polars as pl

rng = np.random.default_rng(42)

N_ORDERS = 2_000_000
N_RESTAURANTS = 5_000
N_COURIERS = 800
START = 1_735_689_600            # 2025-01-01 00:00:00 UTC, epoch seconds
DAYS = 90

CITIES = ["Berlin", "Lisbon", "Toronto", "Pune", "Nairobi"]
CUISINES = ["pizza", "sushi", "burgers", "curry", "salad", "tacos", "noodles"]
NOTES = [
    "", "no onions", "  Extra SPICY please ", "LEAVE AT DOOR",
    "call when outside", "Allergic to nuts!", "extra napkins",
]
STATUSES = ["delivered", "cancelled", "refunded"]


def pick(values, n, p=None):
    """Sample n items from a Python list and return a Polars String Series."""
    idx = rng.choice(len(values), size=n, p=p)
    return pl.Series(values).gather(idx)


restaurants = pl.DataFrame({
    "restaurant_id": np.arange(N_RESTAURANTS),
    "cuisine": pick(CUISINES, N_RESTAURANTS),
    "rating": np.clip(rng.normal(4.2, 0.4, N_RESTAURANTS), 1.0, 5.0).round(1),
})

# Heavy skew: a Zipf draw sends most orders to a handful of restaurants.
restaurant_id = (rng.zipf(1.3, N_ORDERS) - 1) % N_RESTAURANTS

subtotal = np.round(rng.lognormal(mean=3.2, sigma=0.5, size=N_ORDERS), 2)
delivery_min = 10 + rng.gamma(shape=4.0, scale=5.0, size=N_ORDERS)
delivery_min[rng.random(N_ORDERS) < 0.02] = np.nan     # 2% missing

orders = pl.DataFrame({
    "order_id": np.arange(N_ORDERS),
    "ts_s": START + rng.integers(0, DAYS * 86_400, N_ORDERS),
    "restaurant_id": restaurant_id,
    "courier_id": rng.integers(0, N_COURIERS, N_ORDERS),
    "city": pick(CITIES, N_ORDERS, p=[0.30, 0.15, 0.25, 0.20, 0.10]),
    "items": rng.integers(1, 9, N_ORDERS),
    "subtotal": subtotal,
    "delivery_min": pl.Series(delivery_min).fill_nan(None),   # NaN -> real null
    "status": pick(STATUSES, N_ORDERS, p=[0.93, 0.05, 0.02]),
    "note": pick(NOTES, N_ORDERS),
})

orders.write_parquet("orders.parquet", compression="zstd", row_group_size=250_000)
restaurants.write_parquet("restaurants.parquet")
print(orders.shape, restaurants.shape)

A quick note on fill_nan(None). NumPy has no null, only NaN. We convert those NaN values into proper nulls at the boundary. This will matter in the traps section, where we see why Polars separates the two.

Each order picks its city independently of its restaurant, which is not realistic (a restaurant lives in one city). That is fine for a demo, but it is worth remembering that synthetic data can hide the very join and skew problems you would meet on real data.

Read, filter, add columns

First the pandas version.

import pandas as pd

orders_pd = pd.read_parquet("orders.parquet")
restaurants_pd = pd.read_parquet("restaurants.parquet")

orders_pd["ts"] = pd.to_datetime(orders_pd["ts_s"], unit="s")
delivered_pd = orders_pd[orders_pd["status"] == "delivered"]

And the same in Polars.

import polars as pl

orders = pl.read_parquet("orders.parquet").with_columns(
    pl.from_epoch(pl.col("ts_s"), time_unit="s").alias("ts")
)
restaurants = pl.read_parquet("restaurants.parquet")

delivered = orders.filter(pl.col("status") == "delivered")

Three differences are already visible. Polars refers to columns through pl.col("name") rather than indexing the frame. It adds columns with with_columns instead of assigning into the frame, so the frame is never modified in place. And there is no index anywhere. A Polars frame is just columns and rows.

Aggregating with group_by().agg()

Revenue, order count and average delivery time per city per day. In pandas:

daily_pd = (
    delivered_pd.assign(day=delivered_pd["ts"].dt.floor("D"))
    .groupby(["city", "day"], as_index=False)
    .agg(
        orders=("order_id", "count"),
        revenue=("subtotal", "sum"),
        avg_minutes=("delivery_min", "mean"),
    )
    .sort_values(["city", "day"])
)

In Polars:

daily = (
    delivered
    .group_by("city", pl.col("ts").dt.truncate("1d").alias("day"))
    .agg(
        pl.len().alias("orders"),
        pl.col("subtotal").sum().alias("revenue"),
        pl.col("delivery_min").mean().alias("avg_minutes"),
    )
    .sort("city", "day")
)
print(daily.head(3))

The Polars call passes a list of expressions to .agg, and each one is a full expression, so you can use any transformation before the aggregation. pl.len() counts rows in each group. Note the .sort(...) at the end: unlike pandas, group_by in Polars does not guarantee output order, since ordering would cost time and most queries do not need it. You can pass maintain_order=True if you do.

Joins

# pandas
joined_pd = orders_pd.merge(restaurants_pd, on="restaurant_id", how="left")

# Polars
joined = orders.join(
    restaurants, on="restaurant_id", how="left", validate="m:1"
)

The validate="m:1" argument is a small gift. It says "many orders to one restaurant." If restaurants accidentally has two rows for the same restaurant_id, Polars raises an error rather than quietly duplicating every matching order. We will see later how much pain that one argument prevents.

Window functions with .over()

A window function computes something per group but keeps every row. In SQL it is SUM(x) OVER (PARTITION BY city). In pandas it is groupby(...).transform(...). In Polars it is .over(...).

Share of city revenue that each order represents:

# pandas
joined_pd["city_revenue"] = joined_pd.groupby("city")["subtotal"].transform("sum")
joined_pd["share_of_city"] = joined_pd["subtotal"] / joined_pd["city_revenue"]

# Polars
joined = joined.with_columns(
    (pl.col("subtotal") / pl.col("subtotal").sum().over("city")).alias("share_of_city")
)

Time since the same courier's previous order, a useful feature for a fatigue or utilisation model:

# pandas
sorted_pd = orders_pd.sort_values("ts")
sorted_pd["since_prev"] = sorted_pd.groupby("courier_id")["ts"].diff()

# Polars
gaps = (
    orders.sort("ts")
    .with_columns(pl.col("ts").diff().over("courier_id").alias("since_prev"))
    .with_columns(pl.col("since_prev").dt.total_minutes().alias("gap_minutes"))
)

And the top three restaurants by revenue in each city, a ranking window:

city_rest = delivered.group_by("city", "restaurant_id").agg(
    pl.col("subtotal").sum().alias("revenue")
)

top3 = (
    city_rest
    .filter(pl.col("revenue").rank("ordinal", descending=True).over("city") <= 3)
    .sort("city", "revenue", descending=[False, True])
)

The pandas version of the last one is city_rest.sort_values(...).groupby("city").head(3), which works but hides the ranking idea inside a sort-then-slice trick.

Rolling aggregations

Seven-day rolling revenue per city, on the daily table we built:

# pandas (sorted by city, day first)
daily_pd["revenue_7d"] = daily_pd.groupby("city")["revenue"].transform(
    lambda s: s.rolling(7).mean()
)

# Polars
daily = daily.with_columns(
    pl.col("revenue").rolling_mean(window_size=7).over("city").alias("revenue_7d")
)

Both leave the first six days of each city empty, because a full seven-day window does not exist yet. For time-based windows on irregular data, Polars offers group_by_dynamic, which buckets by a time interval. This counts orders per city per hour:

hourly = (
    orders.sort("ts")
    .group_by_dynamic("ts", every="1h", group_by="city")
    .agg(pl.len().alias("orders"))
)

The frame must be sorted by the time column for group_by_dynamic to make sense, which is why the sort comes first.

Strings, dates and conditionals

Customer notes are messy: stray spaces, mixed case. String operations in Polars live under .str, and they run over the Arrow string buffers in native code.

orders = orders.with_columns(
    pl.col("note").str.strip_chars().str.to_lowercase().alias("note_clean")
).with_columns(
    pl.col("note_clean").str.contains("spicy").alias("wants_spicy"),
    pl.col("note_clean").str.len_chars().alias("note_len"),
)

Why two with_columns calls? Because expressions inside one call are evaluated together against the same input frame (that is what lets them run in parallel), so the second group cannot see the note_clean the first group creates. It is the price of parallelism, and it takes about a day to stop tripping over.

Date and time handling lives under .dt:

orders = orders.with_columns(
    pl.col("ts").dt.hour().alias("hour"),
    (pl.col("ts").dt.weekday() >= 6).alias("is_weekend"),   # 1 = Monday ... 7 = Sunday
    pl.col("ts")
    .dt.replace_time_zone("UTC")
    .dt.convert_time_zone("Europe/Berlin")
    .alias("ts_berlin"),
)

Notice the comment about weekday(). Polars follows ISO numbering, Monday is 1 and Sunday is 7. pandas' dayofweek runs from 0 to 6. That single off-by-one has silently broken more than one weekend feature. Also note the two-step time zone dance: our timestamps are naive (no zone attached), so we first declare them to be UTC with replace_time_zone, and only then convert to Berlin local time.

For conditionals, pl.when().then().otherwise() is the vectorised if/else, and you can chain further when clauses:

orders = orders.with_columns(
    pl.when(pl.col("subtotal") < 15).then(pl.lit("small"))
    .when(pl.col("subtotal") < 40).then(pl.lit("medium"))
    .otherwise(pl.lit("large"))
    .alias("basket")
)

print(orders.group_by("basket").agg(pl.len()).sort("basket"))

The pandas equivalent is np.select([cond1, cond2], ["small", "medium"], default="large"), which works well but lives outside the DataFrame API.

List and struct columns

Polars has first-class nested types, which pandas approximates with object columns holding Python lists or dicts. A List column is stored in Arrow's list layout (a flat values buffer plus offsets), and there is a full .list namespace of vectorised operations on it. A Struct is a column of named fields.

This groups by restaurant, collects its three biggest baskets into a list, and captures the most recent order as a struct:

per_restaurant = (
    delivered.group_by("restaurant_id")
    .agg(
        pl.len().alias("n_orders"),
        pl.col("subtotal").sort(descending=True).head(3).alias("top3_baskets"),
        pl.struct(
            pl.col("ts").alias("last_ts"),
            pl.col("subtotal").alias("last_subtotal"),
        )
        .sort_by("ts")
        .last()
        .alias("latest"),
    )
    .with_columns(pl.col("top3_baskets").list.mean().alias("top3_avg"))
    .unnest("latest")
)
print(per_restaurant.head(3))

Here top3_baskets is a List(Float64) column. .list.mean() averages inside each list without any Python loop. unnest expands the struct into ordinary columns last_ts and last_subtotal. If you ever find yourself writing df["col"].apply(lambda xs: sum(xs) / len(xs)) in pandas, this is the alternative.

The same report, lazily

Now the version that is the reason to learn Polars. Same idea, but nothing is read until collect():

lazy_query = (
    pl.scan_parquet("orders.parquet")
    .filter(pl.col("status") == "delivered")
    .join(pl.scan_parquet("restaurants.parquet"), on="restaurant_id", how="left")
    .group_by("city", "cuisine")
    .agg(
        pl.len().alias("orders"),
        pl.col("subtotal").sum().alias("revenue"),
        pl.col("delivery_min").median().alias("median_minutes"),
    )
    .sort("revenue", descending=True)
)

print(lazy_query.explain())      # inspect the optimised plan
report = lazy_query.collect()    # now the work happens
print(report.head())

The explain() output is the one previewed earlier. Of the ten columns in orders.parquet, five are read (restaurant_id, city, subtotal, delivery_min, status), and from restaurants.parquet two of three. The status == "delivered" filter is attached to the scan so that Parquet statistics can skip row groups when possible (with random data like ours, most row groups contain some delivered orders, so the skipping will be modest. On real time-sorted data it is spectacular).

The equivalent pandas run reads everything, keeps everything, and does each step on one core. That is not a criticism of pandas as much as a description of what "eager" means.

Migrating from pandas

You do not need to rewrite everything. The safest migrations are incremental: pick the slowest or most memory-hungry step in a pipeline, port that, and convert to and from pandas at the edges. Here is a cheat-sheet for the port.

Cheat-sheet: pandas to Polars

Double-check argument names against your installed version. The table is meant to point you to the right method, not replace the docs.

Common traps

There is no index. No row labels, no MultiIndex, no .loc. If you relied on the index for alignment, for example adding two Series that align on their labels, you now do it with an explicit join or with matching row order. It feels like a loss for about a week. After that, many people find they were spending a lot of effort fighting the index.

Null is not NaN. Polars distinguishes a missing value (null) from the floating-point "not a number" (NaN). Aggregations skip nulls but propagate NaN. So if you load a NumPy array full of NaN and take the mean, you get NaN back. That tiny experiment shows the whole story:

import polars as pl

s = pl.Series("x", [1.0, 2.0, float("nan"), None])
print(s.mean())              # nan   (the NaN poisons the mean)
print(s.fill_nan(None).mean())  # 1.5   (NaN turned into null, then skipped)
print(s.is_null().sum(), s.is_nan().sum())

The habit to adopt: convert NaN to null at the boundary, as we did with fill_nan(None) in the data generator. pl.from_pandas does this for you by default (nan_to_null=True).

Immutability. There is no df["a"] = ... and no inplace=True. Every operation returns a new frame, and cheaply, because columns are shared rather than copied where possible. Write pipelines as chains and reassign the name.

Lazy versus eager. A LazyFrame has no data. You cannot index it or print its rows. It gives you a plan until you call collect(). New users sometimes call collect() after every step "to see what happened," which quietly turns a lazy pipeline into an eager one. Use .head(5).collect() to peek, or .explain() to look at the plan.

apply is an anti-pattern. In pandas, df.apply(f, axis=1) is slow but common. Polars has pl.col("a").map_elements(f, return_dtype=pl.Float64) and it is the same trap: Python called per row, one thread, no optimiser visibility. Before using it, check for an expression, and for anything with conditions, try when/then/otherwise. If you truly need custom Python, map_batches receives a whole Series at a time, which is usually a much better fit than per-row calls.

Strict types. Polars does not silently turn integers into floats when you introduce a missing value, and casts fail loudly by default (strict=True) if a value does not fit. That is a feature in production. Just expect more explicit .cast(...) calls.

Order is not guaranteed. group_by output order is arbitrary unless you pass maintain_order=True or sort afterwards. Tests that compare exact row order will flake. We will fix that in the testing section.

Comparisons with null are null. pl.col("a") == None does not do what you hope. Use is_null(). And filter drops rows where the predicate is null, as SQL does.

Interoperability: getting data in and out

The bridge between the two libraries is Arrow, which makes it cheap but not free.

import pandas as pd
import polars as pl

pdf = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})

df = pl.from_pandas(pdf)        # pandas -> Polars
back = df.to_pandas()           # Polars -> pandas (needs pyarrow)
arrow_table = df.to_arrow()     # Polars -> pyarrow.Table
df2 = pl.from_arrow(arrow_table)
arr = df.select("a").to_numpy() # Polars -> NumPy

What is zero-copy, and what is not? Numeric columns without nulls can often be shared with NumPy or Arrow without copying. Anything else (strings, nulls that need to become NaN, converting to pandas' object dtype) requires a conversion. Do not assume to_pandas() on a large frame is instantaneous or that it will not double your memory; it often does. You can ask for Arrow-backed pandas columns with df.to_pandas(use_pyarrow_extension_array=True), which avoids some of the conversion cost.

The practical guidance: convert once, at the boundary of a stage, not in a loop. Doing to_pandas() in the middle of a lazy chain also forces a full collect(), which is one of the failure modes we will list at the end. Also, many plotting and ML libraries (scikit-learn, matplotlib, seaborn) now accept Polars input directly or through the Arrow interface, so check before you convert.

The wider Rust-flavoured toolchain

Polars is the most visible member of a broader shift. A lot of the Python tooling you use daily has quietly moved to compiled languages, and it is worth knowing which tool does what.

DuckDB: not Rust, but in the same conversation

DuckDB is often mentioned alongside Polars, so let's be accurate: DuckDB is written in C++, not Rust. It belongs in this discussion because it took the same design decisions. It is an in-process (no server) analytical database with columnar storage, a vectorised execution engine that processes batches of values at a time, multi-threaded parallelism, and first-class Parquet support. It speaks SQL where Polars speaks method chains, and the two share Arrow as a common currency.

That makes them complementary rather than rivals. If your team thinks in SQL, or a query has ten joins and window clauses, DuckDB can be the more natural fit. If you are building feature logic with lots of programmatic column generation, Polars expressions are easier to compose. Moving data between them is cheap:

import duckdb
import polars as pl

orders = pl.read_parquet("orders.parquet")

top_cities = duckdb.sql("""
    SELECT city, COUNT(*) AS orders, SUM(subtotal) AS revenue
    FROM orders
    WHERE status = 'delivered'
    GROUP BY city
    ORDER BY revenue DESC
""").pl()     # result comes back as a Polars DataFrame

print(top_cities)

DuckDB can find the local variable orders by name and query it in place, and it can query Parquet files directly, for example FROM 'orders.parquet'. Use whichever reads best for the step at hand.

uv and Ruff: the Astral tools

uv and Ruff are written in Rust by Astral, and they are not data libraries. They speed up the work around your data code.

• uv is a package and project manager. It resolves and installs dependencies far faster than pip in most setups, creates virtual environments, manages Python versions, and writes a lockfile (uv.lock) so your environment is reproducible. A typical start: uv init pipeline, uv add polars pyarrow, uv run python job.py. For a one-off script, uv run --with polars script.py runs it in a temporary environment.
• Ruff is a linter and formatter that replaces a pile of separate tools (flake8 and its plugins, isort, and a Black-compatible formatter) with one fast binary. ruff check . and ruff format . are usually all you need. It even has a rule set aimed at pandas code (the PD rules) that flags patterns like .values and inplace=True.

Speed matters here for a reason beyond impatience. When a linter runs in 100 milliseconds instead of ten seconds, people run it on every save and in every pre-commit hook, and that changes behaviour.

Two other Rust-backed libraries you probably already use without noticing: pydantic-core (the validation engine inside Pydantic v2) and Hugging Face's tokenizers and safetensors. The pattern repeats: a hot loop moved from Python to Rust, behind an interface that still looks like Python.

Writing your own Python extension with PyO3 and maturin

Sometimes you have a numeric routine that no library offers and that Python is too slow for. PyO3 lets you write it in Rust and call it from Python, and maturin builds and packages it. Here is a great-circle distance function, which is a realistic building block for delivery apps.

Create a project with maturin new geofast (choose PyO3 bindings), or lay out these files by hand. First Cargo.toml:

[package]
name = "geofast"
version = "0.1.0"
edition = "2021"

[lib]
name = "geofast"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.25", features = ["extension-module"] }

Use whatever recent PyO3 version maturin new gives you; the API below matches the modern Bound style. Then pyproject.toml:

[build-system]
requires = ["maturin>=1.5,<2.0"]
build-backend = "maturin"

[project]
name = "geofast"
version = "0.1.0"
requires-python = ">=3.9"

And src/lib.rs:

use pyo3::prelude::*;

const EARTH_RADIUS_KM: f64 = 6371.0088;

/// Great-circle distance between two (lat, lon) points in degrees.
#[pyfunction]
fn haversine_km(lat1: f64, lon1: f64, lat2: f64, lon2: f64) -> f64 {
    let (phi1, phi2) = (lat1.to_radians(), lat2.to_radians());
    let dphi = (lat2 - lat1).to_radians();
    let dlambda = (lon2 - lon1).to_radians();

    let a = (dphi / 2.0).sin().powi(2)
        + phi1.cos() * phi2.cos() * (dlambda / 2.0).sin().powi(2);
    2.0 * EARTH_RADIUS_KM * a.sqrt().asin()
}

/// Distances from one origin to many destinations.
#[pyfunction]
fn haversine_many(lat: f64, lon: f64, lats: Vec<f64>, lons: Vec<f64>) -> Vec<f64> {
    lats.iter()
        .zip(lons.iter())
        .map(|(la, lo)| haversine_km(lat, lon, *la, *lo))
        .collect()
}

#[pymodule]
fn geofast(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(haversine_km, m)?)?;
    m.add_function(wrap_pyfunction!(haversine_many, m)?)?;
    Ok(())
}

Build it into your active virtual environment and call it:

uv pip install maturin
maturin develop --release
python -c "import geofast; print(geofast.haversine_km(52.52, 13.405, 48.8566, 2.3522))"

Berlin to Paris should come out at roughly 878 km, which is a useful sanity check that you did not mix up latitude and longitude. Always build with --release: a debug Rust build can be slower than plain Python and people are regularly surprised by that.

One honest limitation. Vec<f64> arguments are converted from Python lists, which copies. For large arrays, the numpy crate for PyO3 lets you accept a NumPy array without copying. And if what you really want is a custom operation inside a Polars expression, look at Polars expression plugins (the pyo3-polars crate), which let your Rust function run inside the engine on Arrow buffers and keep the optimiser and multi-threading.

Is it worth it? Only after measurement. If a vectorised NumPy or Polars expression solves the problem, use that. Reach for Rust when there is genuine branching, recursion or state per element that cannot be vectorised, and then you get performance without leaving the Python ecosystem.

Production reality

A fast query on a sample is a demo. A pipeline that runs every night for a year, on data you do not control, is a different thing. These are the areas I would check before trusting Polars (or anything) in production.

Parquet layout decides half your performance

Parquet is columnar on disk, with structure worth understanding:

• A file is split into row groups, each holding a chunk of rows (typically hundreds of thousands to a million).
• Inside a row group, every column is stored as its own column chunk, itself made of compressed pages.
• The file footer stores metadata, including, for each column chunk, statistics such as min, max and null count.

Predicate pushdown uses those statistics. A query with WHERE ts >= '2025-03-01' can skip any row group whose maximum ts is earlier than that, without decompressing a byte. This only works well if the data is sorted or clustered by the column you filter on. If timestamps are randomly scattered, every row group's min/max spans the whole range and nothing can be skipped. So sort your data by the most common filter column before writing. Our synthetic data is random, and real data usually is not.

Row group size is a trade-off. Small groups give finer skipping but more metadata and less compression. Very large groups reduce parallelism, since roughly one row group is one unit of parallel work when reading. A few hundred thousand to a few million rows is a sane range, and I would tune it against your own queries.

Compression codecs. Polars' write_parquet supports several, including snappy, lz4, gzip, brotli and zstd. zstd is a strong default: good ratios and fast decompression. snappy and lz4 favour speed over size. gzip is mostly there for compatibility. Test on your data, because the gap depends on how repetitive it is.

Partitioning splits a dataset into directories by a column value, the classic being date=2025-03-01/ or city=Berlin/. Engines skip directories that do not match the filter. Partition by low-cardinality columns that queries commonly filter on, and avoid over-partitioning: ten thousand tiny files are slower than fifty medium ones. Here is a simple Hive-style writer and reader:

from pathlib import Path
import polars as pl

orders = pl.read_parquet("orders.parquet")

for (city,), part in orders.partition_by("city", as_dict=True).items():
    out = Path("dataset") / f"city={city}"
    out.mkdir(parents=True, exist_ok=True)
    part.drop("city").write_parquet(out / "part-0.parquet", compression="zstd")

berlin = (
    pl.scan_parquet("dataset/**/*.parquet", hive_partitioning=True)
    .filter(pl.col("city") == "Berlin")
    .select(pl.len())
    .collect()
)
print(berlin)

The city value is recovered from the directory name, and the filter on it means the other four directories are never opened.

Enforce the schema at the boundary

Most production data bugs are not about performance. They are about somebody upstream renaming a column, changing an integer to a string or sending negative prices. The strategy is to check at the boundary where data enters your pipeline and fail early with a clear message.

Polars makes the schema of a lazy frame available without reading the data, via collect_schema():

import polars as pl

EXPECTED = {
    "order_id": pl.Int64,
    "restaurant_id": pl.Int64,
    "city": pl.String,
    "subtotal": pl.Float64,
    "status": pl.String,
}


def validate(lf: pl.LazyFrame) -> pl.LazyFrame:
    actual = lf.collect_schema()

    missing = [c for c in EXPECTED if c not in actual]
    wrong = {
        c: (str(actual[c]), str(t))
        for c, t in EXPECTED.items()
        if c in actual and actual[c] != t
    }
    if missing or wrong:
        raise ValueError(f"Schema drift. Missing: {missing}. Wrong types (got, want): {wrong}")

    # Value rules: cheap, and they run inside the same optimised query.
    bad = (
        lf.filter((pl.col("subtotal") < 0) | pl.col("city").is_null())
        .select(pl.len())
        .collect()
        .item()
    )
    if bad:
        raise ValueError(f"{bad} rows violate value rules")

    return lf.select([pl.col(c).cast(t, strict=True) for c, t in EXPECTED.items()])


clean = validate(pl.scan_parquet("orders.parquet"))

Splitting the job into a structural check (names and types, which costs nothing) and a value check (which scans data) keeps the cheap check first. For richer rules, libraries such as Pandera have Polars support, and dataclass-style tools exist too. The principle is the same either way: a contract at the door, so that downstream code can trust its inputs.

Out-of-memory strategies

When a job does not fit, work down this list in order.

1. Use fewer columns and rows. Lazy scanning with projection and predicate pushdown often solves it on its own.
2. Use better dtypes. pl.Int32 instead of pl.Int64 when values fit, pl.Float32 for features, pl.Categorical or pl.Enum for low-cardinality strings such as city or status. Halving the width halves the memory and the scan time.
3. Stream it. collect(engine="streaming") or sink_parquet, described earlier.
4. Aggregate before you join. Joining two large tables and then aggregating is far more expensive than aggregating each side to the join grain first.
5. Process in slices. Loop over partitions or files, write out intermediate results, and combine at the end. It is unfashionable and it works.
6. Then, and only then, buy a bigger machine.

Testing with small frames

Pipelines are functions from frames to frames, so they are easy to test: build a tiny input by hand, run the function, and compare against a hand-written expected output. polars.testing.assert_frame_equal does the comparison with helpful diffs.

import polars as pl
from polars.testing import assert_frame_equal


def daily_revenue(df: pl.DataFrame) -> pl.DataFrame:
    return (
        df.filter(pl.col("status") == "delivered")
        .group_by("city")
        .agg(pl.col("subtotal").sum().alias("revenue"))
    )


def test_daily_revenue_ignores_cancelled_orders():
    df = pl.DataFrame({
        "city": ["Berlin", "Berlin", "Pune"],
        "status": ["delivered", "cancelled", "delivered"],
        "subtotal": [10.0, 99.0, 5.5],
    })

    expected = pl.DataFrame({"city": ["Berlin", "Pune"], "revenue": [10.0, 5.5]})

    assert_frame_equal(
        daily_revenue(df),
        expected,
        check_row_order=False,   # group_by order is not guaranteed
    )

Useful options are check_row_order, check_column_order, check_dtypes, and the float tolerances rtol and atol. Do not compare floats for exact equality. Also test the ugly cases explicitly: an empty frame, a frame that is all nulls, a single-row group, and a key that appears in one table but not the other. Those are where pipelines actually fail.

Benchmarking properly

Plenty of "X is 50x faster than Y" claims online come from an unfair setup rather than a real difference. A benchmark you would trust needs these things:

• Same work. Same input file, same output, and a check that the results agree before you time anything.
• Warm-up. The first run pays for imports, page-cache misses and lazy initialisation. Run it once or twice and throw the result away.
• Repeat and take the median. Single runs are noisy, and the mean is dragged by outliers.
• Same thread count. Polars will use all your cores by default and pandas mostly will not. That may be the legitimate real-world difference, but report it, and also run once with POLARS_MAX_THREADS=1 to separate "better algorithm" from "more cores."
• Same dtypes. Comparing Arrow strings to Python object strings is comparing two data layouts as much as two libraries.
• Include the I/O in a way that matches reality. If production reads from disk, do not benchmark from a warm in-memory frame.
• Measure memory too. Peak RSS often matters more than seconds.

Here is a script that follows the rules for our daily report. Run it as python bench.py, then again as POLARS_MAX_THREADS=1 python bench.py.

import statistics
import time

import pandas as pd
import polars as pl


def bench(fn, repeat=7, warmup=2):
    for _ in range(warmup):
        fn()
    samples = []
    for _ in range(repeat):
        t0 = time.perf_counter()
        fn()
        samples.append(time.perf_counter() - t0)
    return statistics.median(samples)


def pandas_job():
    df = pd.read_parquet("orders.parquet", columns=["ts_s", "city", "status", "subtotal"])
    df = df[df["status"] == "delivered"].copy()
    df["day"] = pd.to_datetime(df["ts_s"], unit="s").dt.floor("D")
    return df.groupby(["city", "day"])["subtotal"].sum()


def polars_job():
    return (
        pl.scan_parquet("orders.parquet")
        .filter(pl.col("status") == "delivered")
        .group_by("city", pl.from_epoch(pl.col("ts_s"), time_unit="s").dt.truncate("1d").alias("day"))
        .agg(pl.col("subtotal").sum())
        .collect()
    )


# 1. Correctness first: do both libraries agree on the total?
total_pd = float(pandas_job().sum())
total_pl = float(polars_job()["subtotal"].sum())
assert abs(total_pd - total_pl) < 1e-6 * max(1.0, abs(total_pd)), (total_pd, total_pl)

# 2. Then time.
print(f"pandas : {bench(pandas_job):.3f} s (median)")
print(f"polars : {bench(polars_job):.3f} s (median)")
print(f"polars threads: {pl.thread_pool_size()}")

Notice I gave pandas the fair advantage of columns=[...] in read_parquet, so that it also skips the columns it does not need. Comparing against a naively written pandas job would inflate the gap. Report what you measured, on which machine, with which versions.

Feeding Polars data into an ML training loop

Polars is the pre-processing stage, and PyTorch or scikit-learn is the consumer. The handover is a NumPy array, and it is worth doing carefully: exactly once, with the right dtype, after nulls have been dealt with.

import numpy as np
import polars as pl
import torch
from torch.utils.data import DataLoader, Dataset

FEATURES = ["items", "log_subtotal", "hour", "is_weekend", "rest_rating"]
TARGET = "delivery_min"

features = (
    pl.scan_parquet("orders.parquet")
    .filter(pl.col("status") == "delivered")
    .join(pl.scan_parquet("restaurants.parquet"), on="restaurant_id", how="left")
    .with_columns(pl.from_epoch(pl.col("ts_s"), time_unit="s").alias("ts"))
    .select(
        pl.col("items").cast(pl.Float32),
        pl.col("subtotal").log1p().cast(pl.Float32).alias("log_subtotal"),
        pl.col("ts").dt.hour().cast(pl.Float32).alias("hour"),
        (pl.col("ts").dt.weekday() >= 6).cast(pl.Float32).alias("is_weekend"),
        pl.col("rating").cast(pl.Float32).alias("rest_rating"),
        pl.col(TARGET).cast(pl.Float32),
    )
    .drop_nulls()
    .collect()
)


class DeliveryDataset(Dataset):
    def __init__(self, df: pl.DataFrame):
        self.x = torch.from_numpy(np.ascontiguousarray(df.select(FEATURES).to_numpy(), dtype=np.float32))
        self.y = torch.from_numpy(np.ascontiguousarray(df[TARGET].to_numpy(), dtype=np.float32))

    def __len__(self):
        return len(self.y)

    def __getitem__(self, i):
        return self.x[i], self.y[i]


loader = DataLoader(DeliveryDataset(features), batch_size=1024, shuffle=True)
xb, yb = next(iter(loader))
print(xb.shape, yb.shape)

Every feature is an expression, so the whole preparation is one optimised query, and only the columns and rows that survive are ever materialised. Polars also has DataFrame.to_torch() for a direct tensor conversion, which is handy for quick work. I prefer the explicit NumPy path here because it makes the dtype and memory layout visible.

Two cautions. Do not call .to_numpy() on a huge frame if your dataset is bigger than RAM; instead write features to sharded Parquet files and let each epoch iterate over the shards (or use slices of the frame with iter_slices). And be careful with leakage: computing a window or group statistic, such as the average delivery time per restaurant, over the whole dataset before splitting into train and validation puts validation information into the training features. Split first, or compute those aggregates only from the training window.

How pipelines fail

The interesting production incidents are seldom "Polars is slow." They are these.

Unexpected schema change. An upstream team turns subtotal from a float into a string with a currency symbol, or renames city to market. Without a check you find out from a downstream chart that looks strange. With the validate function above, you find out at 02:00 from a clear exception with the two schemas printed side by side.

Null explosion in joins. Two related things happen here. First, a left join against a table with unmatched keys fills columns with null, and a later aggregate quietly skips those rows, so your totals shrink without an error. After every join, count nulls in the columns you brought in. Second, a duplicated key in the dimension table multiplies rows in the fact table: a row in orders matches two restaurants rows and is counted twice, so revenue inflates. Use validate="m:1" and it fails on the spot. As a rule, know the cardinality of every join you write. Also remember that by default, null keys do not match each other in Polars joins, unlike in some pandas merges.

Skewed group keys. In our data, a few restaurants receive most orders. In a group_by, that means some partitions are much heavier than others, and a hot key can dominate the time or memory of a step. Polars copes well with mild skew because of work stealing. Extreme skew shows up as one core pegged while others idle, or as memory spikes in streaming mode. Mitigations: pre-filter, pre-aggregate in two stages (aggregate by key and hour, then by key), or treat the hot keys separately. Look at df["key"].value_counts(sort=True).head() before you go looking for exotic causes.

Accidental full collect. The lazy pipeline is only lazy until something forces it. The usual culprits are .collect() called too early "just to check," passing a LazyFrame into a function that calls .to_pandas(), and map_elements or Python lambdas inside a filter, which block pushdown so the whole file is read before the filter runs. The defences are cheap: read explain() for your critical queries, look for PROJECT n/m COLUMNS and a SELECTION on the scan, and keep a single collect() (or sink_*) at the very end of each pipeline stage.

Memory that looks fine until Friday. A job that fits comfortably on Tuesday's data fails on the month-end file. Set memory limits on your containers to fail early, monitor peak RSS in the job logs, and keep an eye on data growth the same way you keep an eye on run time.

The pattern in all five is the same: make assumptions explicit, check them cheaply, and fail loudly and early.

What to build next

Here are four small projects that turn this article into skills, in rising order of effort:

1. Port one real pandas notebook. Pick your slowest cell, rewrite it as a lazy Polars query, assert that the results match with assert_frame_equal, and run the benchmark script above on it.
2. Build a nightly feature job. scan_parquet on raw events, features as expressions, sink_parquet to a partitioned folder, with the validate function at the front and a test with tiny frames.
3. Wrap a bottleneck in Rust. Take a per-element routine that will not vectorise, build it with maturin, and compare it honestly with NumPy and a Polars expression.
4. Run the same analysis in DuckDB SQL and in Polars. Compare readability, plans (EXPLAIN in DuckDB, .explain() in Polars) and speed on your data. You will learn more about both from that than from any benchmark table.

The theme running through this whole article is a simple one. Fast data code is mostly code that does less: reads fewer bytes, touches fewer columns, copies less and lets all the cores work at once. Polars, DuckDB and friends are not faster because of language logos. They are faster because their designs make "do less" the default.

In Part 4 we leave the tables behind and go to pixels: vision transformers, how they turn an image into a sequence of patches, and how to shrink and deploy them on edge devices where memory and power are tight. It is the same spirit as this part, finding the smallest amount of computation that gets the job done, applied to a very different kind of data.