From Hand-Made Features to Learned Features

Eight parts in, one quiet assumption

Over the last eight parts you built a working toolbox. You learned how a sensor turns light into numbers (part 2), how those numbers can be arranged as RGB or HSV or greyscale (part 3), how OpenCV loads and shows them (part 4), how small grids of numbers called kernels can blur or sharpen an image (part 5), how gradients turn into edges, contours and shape measurements (part 6), how thresholds turn an image into a mask that isolates what matters (part 7), and how all of this runs frame by frame on live video (part 8).

Every one of those tools shares something in common that we have not named yet. In each case, a person looked at the problem and decided what to measure: which kernel to use, which colour range counts as skin or grass or a red sign, which threshold separates foreground from background, which property of a contour tells you it is roughly circular. The computer did the arithmetic, but a human chose what the arithmetic should be about. This article gives that pattern a name, looks at where it starts to struggle, and introduces the idea that replaced it in many modern applications: instead of a human deciding what to measure, the measurements themselves are learned from examples.

This is a concept article, not a training session. We are not going to train a neural network here, and the three hands-on projects coming next in parts 10 to 12 will still be built entirely from the hand-made tools of parts 1 to 8. The goal of this part is to make sure you understand clearly what those tools are examples of, and what the alternative looks like, so that when you meet deep learning later you recognise it as a continuation of the same story rather than something unrelated.

What exactly is a feature

A raw image is a grid of numbers. A modest 640 by 480 colour photo already has 921,600 pixels, each with three channel values, which is close to 2.8 million numbers. Almost none of those numbers are directly useful for answering a question like is there a face here, or is this a coin, or has something moved in this scene. A feature is a number, or a small set of numbers, computed from the image that is more directly useful for a particular task than the raw pixels are, precisely because it throws away information that does not matter and keeps information that does.

Take a concrete example. Suppose the task is to tell whether a photo contains a stop sign. The raw pixel grid technically contains the answer, in the sense that everything the computer could ever know about the image is in there somewhere, but it is buried under enormous irrelevant detail about sky colour, road texture, tree branches and camera noise. A single number such as the fraction of pixels that are a strong red is a feature: far smaller than the original image, and far more directly connected to the question being asked. Choosing what to measure, and writing the formula or algorithm that measures it, is called feature engineering.

The hand-made features you already have

Looking back, almost every technique from parts 2 to 8 is a hand-made feature, or a step towards building one. Here is the same material, relabelled.

• Raw pixel brightness (part 2): the simplest possible feature, one number per location, with no processing at all.
• Hue and saturation (part 3): a human decided that separating colour from brightness is more useful for many tasks than red, green and blue mixed together.
• A blurred copy of the image (part 5): a hand-chosen kernel that averages nearby pixels, used as a feature that is more robust to noise than the raw pixel.
• Gradient magnitude from a Sobel kernel (parts 5 and 6): a hand-chosen pair of 3 by 3 number grids, derived from calculus, that responds strongly where brightness changes quickly.
• Contour area, perimeter and number of corners (part 6): hand-written formulas applied to the outline of a shape, used to decide whether something is round, square or octagonal.
• A binary mask from thresholding (part 7): a single hand-chosen cutoff value that turns a continuous image into a yes or no decision per pixel.
• Frame difference between consecutive video frames (part 8): a hand-written rule stating that large pixel differences between frames mean motion.

In every case, the formula existed before you ever ran it on a single image. Nobody looked at thousands of labelled photos of stop signs and backed out the number 100 as the right lower bound for red hue, or the number 8 as the right corner count. A human reasoned about the problem using knowledge of geometry, optics or colour theory, and wrote the rule down directly. That is what hand-made, or hand-crafted, means here.

Three more classic hand-made features

Corners

An edge, as you saw in part 6, is a place where brightness changes a lot in one direction but not much along the edge itself. A corner is different: it is a place where brightness changes a lot no matter which direction you look. One classic way to find corners, often associated with the name Harris, works by imagining a small window placed over a point in the image and sliding it slightly in several directions. Over a flat region, sliding the window barely changes what is inside it. Over an edge, sliding it along the edge changes almost nothing, but sliding it across the edge changes a lot. Over a true corner, sliding the window in any direction changes what is inside it substantially. That difference in behaviour, measured mathematically rather than by literally sliding a window around, is what a corner detector reports as a corner score at each pixel. Corners are useful because they tend to be stable, recognisable points you can track across frames or match between two photos of the same object taken from different angles.

Colour histograms

A colour histogram summarises an image, or a region of one, by counting how many pixels fall into each of several colour bins, ignoring where in the image each pixel sits. Imagine a tiny 2 by 2 patch with hue values, on OpenCV's 0 to 179 scale from part 3, of 10, 15, 100 and 105. If we define four bins covering 0 to 45, 45 to 90, 90 to 135 and 135 to 180, then two pixels land in the first bin and two land in the third, giving the histogram 2, 0, 2, 0. A different 2 by 2 patch with hues 12, 8, 102 and 98 produces exactly the same histogram, even though none of the individual values match. That is the whole point: the histogram captures that the patch has two dark reddish pixels and two cyan-ish pixels, and discards exactly which pixel held which value and where it was positioned. A full colour histogram with more bins, computed over a whole image, becomes a compact feature vector that can be compared between images using simple distance measures.

Histograms of oriented gradients

A more elaborate hand-made feature, often abbreviated HOG, combines the gradient idea from part 6 with the histogram idea above. The image is divided into small cells, perhaps 8 by 8 pixels each. Within each cell, the gradient direction and magnitude is computed at every pixel, exactly as a Sobel filter would, and a histogram is built of how many pixels point in each direction range, for example 0 to 20 degrees, 20 to 40 degrees, and so on. Concatenating the histograms from every cell across the image produces one long descriptor vector that captures the overall pattern of edges and their orientations, which turns out to describe the outline of a shape quite well regardless of small changes in lighting. This kind of descriptor became well known for detecting people in photographs, because the rough outline of a standing human body, arms and legs, produces a fairly consistent pattern of edge orientations even across different clothing and backgrounds.

Turning features into decisions: a worked example

Features on their own are not decisions. The classic computer vision pipeline extracts one or more hand-made features and then applies a rule, or a simple classifier trained on those feature numbers, to reach a decision. Here is a small, self-contained example that detects something resembling a stop sign using exactly the tools from parts 6 and 7: a colour mask and a corner count from a contour.

import cv2
import numpy as np

# Build a synthetic "stop sign": a red octagon on a grey background
img = np.full((300, 300, 3), 200, dtype=np.uint8)

center = (150, 150)
radius = 80
n_sides = 8
angles = np.linspace(0, 2 * np.pi, n_sides, endpoint=False) + np.pi / 8
points = np.array([
    [center[0] + radius * np.cos(a), center[1] + radius * np.sin(a)]
    for a in angles
], dtype=np.int32)

cv2.fillPoly(img, [points], color=(0, 0, 200))  # BGR: a red shape

# Hand-made feature 1: fraction of red pixels
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lower_red = np.array([0, 100, 100])
upper_red = np.array([10, 255, 255])
red_mask = cv2.inRange(hsv, lower_red, upper_red)
red_fraction = np.sum(red_mask > 0) / red_mask.size

# Hand-made feature 2: number of corners in the largest contour
contours, _ = cv2.findContours(red_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
largest = max(contours, key=cv2.contourArea)
perimeter = cv2.arcLength(largest, True)
approx = cv2.approxPolyDP(largest, 0.02 * perimeter, True)
num_corners = len(approx)

print(f"Red pixel fraction: {red_fraction:.2f}")
print(f"Number of corners: {num_corners}")

# Hand-made rule: is this a stop sign?
is_stop_sign = red_fraction > 0.05 and num_corners == 8
print("Looks like a stop sign:", is_stop_sign)

This uses nothing beyond cvtColor, inRange, findContours and approxPolyDP from earlier parts, with findContours in OpenCV 4.x returning just two values as shown. Given the geometry in the code, the red octagon covers a bit under a fifth of the 300 by 300 image, so red_fraction should print as roughly 0.2, comfortably above the 0.05 threshold. The corner count should usually print as 8, but it is worth running the code and trying different values for the epsilon argument to approxPolyDP, the 0.02 multiplier, since that number controls how aggressively the contour is simplified. With a slightly different epsilon you may see 7 or 9 instead. That sensitivity is not a bug in this article's code, it is a very real and very typical property of hand-made rules: a number that a human picked somewhat arbitrarily can tip a decision one way or the other.

Why hand-made features eventually hit a wall

The stop sign example above works, but only because the conditions are generous: a clean synthetic shape, strong uniform red, no shadows, no dirt, no snow, no graffiti, taken straight on. Real photographs rarely cooperate so completely, and every departure from the ideal tends to break a hand-made rule in its own specific way.

• A sign in shade or under orange streetlight shifts the hue values away from the fixed red range chosen in the code, so the mask misses pixels it should catch.
• A sign partly covered by a sticker, a branch or snow changes the contour's outline, so approxPolyDP may return 5 or 11 corners instead of 8.
• A sign photographed from an angle, rather than straight on, appears as a squashed, non-regular octagon, and the fixed epsilon that worked for a regular shape may no longer simplify it to exactly 8 points.
• A sign further from the camera occupies far fewer pixels, so the red fraction threshold tuned for a close-up photo may fail for a distant one.
• A faded, sun-bleached sign has paint closer to orange or pink than the red range in the code, again falling outside the fixed hue window.

Each of these problems has a fix, in principle: widen the hue range, add a rule for partial occlusion, scale the threshold with estimated distance. But each fix is itself a new hand-made decision, made by a human staring at a handful of failing photos and guessing at a patch. For a single, narrow, well-defined task like reading a known road sign shape, this kind of patching can get you a long way, which is exactly why the hands-on projects in parts 10 to 12 of this course, a document scanner, a coin counter and a motion detector, work well with nothing more than the tools from parts 1 to 8. Those tasks are narrow and the conditions are controllable.

The wall appears when the task stops being narrow. Try to write a hand-made rule that recognises a cat in any photograph: any breed, any pose, sitting, jumping, half hidden behind a sofa, lit by a window, lit by a lamp, photographed on a phone or a professional camera. Nobody has found a short formula built from edges, colours and corner counts that handles that variety reliably. The number of special cases a human would need to anticipate and hand-code grows far faster than any person can keep up with. That is the point at which a different approach becomes attractive: instead of a human guessing every rule in advance, let the rules be discovered from a large number of examples.

The core idea behind learned features

Go back to the Sobel kernel from part 5. It is a 3 by 3 grid of numbers, for example minus 1, 0, 1 in the top row, minus 2, 0, 2 in the middle row, and minus 1, 0, 1 in the bottom row for the vertical version. Those nine numbers were not guessed randomly, they were derived mathematically to approximate how brightness changes from left to right at each point, and sliding that grid across an image, multiplying and summing as it goes, is the convolution operation you already used with cv2.filter2D.

The core idea behind learned features is to keep exactly that same operation, a small grid of numbers sliding across the image and combining with it by multiplication and summation, but to stop deriving the numbers from calculus and instead find them by a different process. Start the grid with some arbitrary numbers. Show the computer an image along with the correct answer for some task, for instance a label saying cat or not cat. Let the computer compute its current best guess using the current numbers. Measure how wrong that guess was. Nudge every number in the grid very slightly in the direction that would have made the guess a little less wrong. Repeat this with another image, and another, across a large collection of labelled images, often many thousands or more. After enough repetitions the numbers settle into values that are genuinely useful for separating cats from non-cats, even though no person chose them and in general no person could easily explain in words why those particular nine, or ninety, or nine million numbers work.

How a machine actually learns a number: a tiny worked example

The mechanism above, predict, measure error, nudge, repeat, is called training, and the specific method of nudging numbers is called gradient descent. It is easiest to see with a single learnable number rather than millions. Suppose our entire model is one number, called a weight and written w, and it predicts an output by simple multiplication, y_hat equals w times x. Suppose the true relationship we want it to discover, hidden from the model, is y equals 2 times x, and we have three example pairs: x equals 2, y equals 4; x equals 3, y equals 6; x equals 5, y equals 10. We start the model off badly, with w set to 0.5, and show it the three examples one at a time, using a learning rate of 0.01 to control how big each nudge is.

1. Example one, x is 2 and y is 4. With w at 0.5 the prediction is 0.5 times 2, which is 1. The error, true value minus prediction, is 3. Using the squared error loss, the gradient works out to minus 2 times x times the error, which is minus 12. The weight update is w minus the learning rate times the gradient, giving 0.5 minus 0.01 times minus 12, which is 0.5 plus 0.12, equal to 0.62.
2. Example two, x is 3 and y is 6. With w now at 0.62 the prediction is 1.86. The error is 4.14. The gradient is minus 2 times 3 times 4.14, which is minus 24.84. The new weight is 0.62 plus 0.01 times 24.84, which is 0.62 plus 0.2484, equal to 0.8684.
3. Example three, x is 5 and y is 10. With w now at 0.8684 the prediction is 4.342. The error is 5.658. The gradient is minus 2 times 5 times 5.658, which is minus 56.58. The new weight is 0.8684 plus 0.01 times 56.58, which is 0.8684 plus 0.5658, equal to 1.4342.

After seeing each of the three examples exactly once, the weight moved from 0.5 to roughly 1.43, heading steadily towards the true value of 2 that generated the data. Seeing the same examples again, or many more examples like them, would push it closer still. Nothing in this process involved a human looking at the data and deciding the answer should be 2. The number emerged purely from repeatedly comparing predictions against known correct answers and nudging in the direction that reduced the error. A real convolutional network has not one number but millions, arranged across many filters and many layers, and it is shown millions of labelled images rather than three, with more careful optimisation methods than this simple version. But the mechanism driving every single one of those numbers during training is this same loop, scaled up enormously.

Convolutional neural networks: learned filters, stacked

A convolutional neural network, usually abbreviated CNN, is built from exactly the convolution operation you already know, repeated many times with learned numbers instead of hand-derived ones, and arranged in layers. The first layer applies many small learned kernels to the raw pixel grid, each one producing its own output image, called a feature map, much like the single edge map you produced with a Sobel filter in part 6, except now there are dozens of them. Because the numbers in these first-layer kernels were learned rather than designed, nobody decided in advance what each one should detect, but researchers who inspect trained networks often find that many early filters end up resembling simple edge detectors or colour detectors, not unlike a Sobel kernel, discovered independently by the training process rather than copied from a textbook.

A second layer of learned kernels then slides across that stack of feature maps, not the raw pixels, and combines them into new feature maps that tend to respond to slightly more complex patterns, such as corners, simple textures or short curved edges. Stacking further layers continues the pattern: edges combine into textures and parts, parts combine into recognisable chunks like an eye, a wheel or a doorway, and the final layers combine those chunks into a judgement about the whole image, such as 87 percent cat, 10 percent dog and 3 percent other. In between the convolution layers there are usually pooling layers, which shrink the feature maps by summarising small neighbourhoods, both to keep the amount of computation manageable and to make the network somewhat tolerant of a shape appearing a few pixels to the left or right of where it appeared during training.

Notice that this hierarchy, pixels to edges to shapes to whole-object decisions, is the exact progression this course has followed by hand across parts 2, 5, 6 and the stop sign example above. The difference is not the shape of the idea, it is who supplies the numbers at each stage and how many stages there are. The code below makes the shared mechanism concrete: a hand-designed Sobel kernel and a second kernel whose numbers are simply made up for illustration, standing in for numbers a trained network might have discovered, applied to the same simple image using the same filter2D function from part 5.

import cv2
import numpy as np

# A small synthetic grayscale image with a vertical edge
img = np.zeros((100, 100), dtype=np.uint8)
img[:, 50:] = 255  # right half bright, left half dark
img = img.astype(np.float32)

# A hand-made kernel: the classic vertical Sobel filter
sobel_kernel = np.array([
    [-1, 0, 1],
    [-2, 0, 2],
    [-1, 0, 1]
], dtype=np.float32)

# A kernel with different numbers, standing in for "whatever a network
# might discover by training on many images". These numbers are made up
# for illustration only, they are not the result of any real training.
example_kernel = np.array([
    [-0.3,  0.1,  0.9],
    [-0.8,  0.05, 0.7],
    [-0.4, -0.1,  0.6]
], dtype=np.float32)

sobel_response = cv2.filter2D(img, -1, sobel_kernel)
example_response = cv2.filter2D(img, -1, example_kernel)

print("Sobel response near the edge:", sobel_response[50, 47:53])
print("Example kernel response near the edge:", example_response[50, 47:53])

Run this and print the two rows of numbers. Both responses should spike somewhere around column 50, where the image jumps from dark to bright, because both kernels are sensitive to that brightness change. But the exact shape of the spike differs, because the nine numbers inside each kernel differ. That is the whole relationship between hand-made and learned features in one small piece of code: the same sliding, multiplying and summing operation, with the only difference being where the nine numbers came from, calculus and intuition in one case, a training loop like the one in the previous section, run at a far larger scale, in the other.

Where OpenCV fits once learning enters the picture

OpenCV itself is not usually used to train a convolutional network from scratch, that kind of training is typically done in a dedicated deep learning framework, commonly PyTorch or TensorFlow, often using specialised hardware to cope with the huge number of repetitions involved. What OpenCV does provide is a module called dnn, which can load a network that has already been trained elsewhere and run it on new images to get predictions, a step usually called inference rather than training. Functions such as cv2.dnn.readNetFromONNX or cv2.dnn.readNetFromTensorFlow load a saved model, cv2.dnn.blobFromImage prepares an image into the format the network expects, and calling forward on the loaded network returns its output. This course does not walk through that process, since it depends on having a specific pretrained model file and goes beyond what Python basics and the OpenCV functions covered so far can teach on their own, but it is worth knowing the door exists.

In practice, many real applications combine both worlds rather than choosing one. A system might use a cheap hand-made step, a colour mask or a threshold, to quickly find a small region of interest in a large image, and only run a slower, learned-feature model on that small crop, saving a great deal of computation. The skills from parts 1 to 8 of this course are not a stepping stone to be discarded once learned features appear, they remain directly useful as the fast, simple, interpretable half of many production pipelines.

Common mistakes and misconceptions

• Thinking learned means no feature engineering at all. Choosing how to resize, normalise and augment training images is still a human decision that strongly affects results, it has simply moved from designing individual kernels to designing the data pipeline around the network.
• Assuming hand-crafted features are obsolete. For a narrow, controlled task, such as the document scanner, coin counter and motion detector coming up in parts 10 to 12, a hand-made rule needs no training data, runs instantly on a laptop and is easy to debug line by line, all genuine advantages a learned model does not automatically have.
• Believing a bigger or deeper network always wins. Extra layers and extra data help only when the task and the available labelled examples support it. For a fixed, simple task with little data, a well-chosen hand-made rule can outperform an undertrained network.
• Expecting every learned filter to correspond to a clean, nameable concept. Early layers often do resemble edge or colour detectors, as mentioned above, but deeper filters frequently respond to messy combinations of patterns that have no simple one-word description.
• Confusing training with inference. Training, finding the numbers, needs labelled data, a loss function and many repetitions, and is usually the expensive step. Inference, using numbers that have already been found on a new image, is comparatively cheap and is what a function like cv2.dnn's forward call does.
• Treating the three-example gradient descent walkthrough above as a realistic training recipe. It was deliberately tiny to make the arithmetic followable by hand. Real training uses far more data, more careful optimisers and techniques to avoid memorising the training examples instead of learning a general pattern.

Summary and what's next

Across parts 2 to 8 you built a set of hand-made features: pixel values, colour ranges, blurred and gradient images, contour measurements, masks and frame differences, each one a formula a human derived from knowledge of optics, geometry or colour and then wrote down in code. This part named that pattern and placed it inside a bigger picture. Hand-made features are the product of a person deciding in advance what to measure. Learned features use exactly the same underlying operations, most notably the sliding, multiplying and summing of convolution you already know from part 5, but find the numbers inside those operations automatically, by repeatedly comparing predictions against known correct answers on a large set of examples and nudging the numbers to be a little less wrong each time.

Neither approach has made the other irrelevant. Hand-made features remain fast, require no training data and are straightforward to inspect and fix, which is exactly why the next three parts of this course, a document scanner, a coin counter and a webcam motion detector, are built entirely from the tools of parts 1 to 8 and need nothing beyond them. Learned features earn their cost, the data, the compute and the harder to inspect internals, on broader and more variable tasks where nobody has managed to write a short, reliable hand-made rule. Understanding both, and knowing which situation calls for which, is the real skill. Training neural networks yourself, choosing a framework and working with the OpenCV dnn module in more depth are each large enough topics to deserve their own learning path, but you now have the concepts needed to make sense of them when you get there. For now, part 10 puts everything from this path together to build a working document scanner using nothing but the hand-made tools you already understand inside and out.