Filters and Blurs: Cleaning Up an Image

Why bother cleaning up an image at all

Back in Part 2 you saw that a camera sensor does not record a perfect copy of the world. Every pixel value is the result of a small physical process: photons landing on a tiny patch of silicon, converted to a voltage, converted to a number. That process always carries a bit of randomness, especially in dim light or with a cheap sensor like the one in a laptop webcam. The result is noise: pixels that are slightly brighter or darker than they should be, scattered across the image in no particular pattern.

Noise is a nuisance for almost everything you will do later in this path. Part 6 will show you how to find edges by looking at sudden jumps in brightness between neighbouring pixels. If the image is noisy, every one of those random pixel variations looks like a tiny fake edge, and a clean outline gets buried in thousands of false ones. Part 7 will show you how to threshold an image, turning it into black and white based on a brightness cut-off. Noise pushes pixels above or below that cut-off at random, leaving speckles in what should be a clean mask. Cleaning the image up first, before asking any of these questions, makes everything downstream easier and more reliable.

The tool for this job is the filter, and the most common kind of filter is a blur. That might sound backwards: why would making an image blurrier make it more useful? The idea is that random noise is, by definition, uncorrelated from one pixel to the next, while the real structure in a photograph, a face, a coin, a page of text, is not. If you average a pixel together with its neighbours, the real structure survives because neighbouring pixels tend to agree with each other, while the random noise tends to cancel out because it does not. You trade a small amount of fine detail for a large reduction in randomness. Used carefully, that is a very good trade.

Pixels as neighbourhoods: the idea of a kernel

To understand any filter, you first need to understand the one idea they all share: looking at a small neighbourhood of pixels around each point and combining them with a set of weights. That set of weights is usually written as a small grid of numbers called a kernel, or sometimes a mask. A 3 by 3 kernel looks at the pixel itself and its eight immediate neighbours; a 5 by 5 kernel looks at a slightly wider patch, and so on.

Here is a concrete example. Suppose a tiny patch of a greyscale image, around the pixel we care about, has these brightness values from 0 (black) to 255 (white):

10  12 200
 9  11  14
11  10  13

The centre pixel reads 11, but its true surroundings are all around 10 to 14. The value 200 just above it is almost certainly a noise spike, a single bad reading. If we apply the simplest possible kernel, a 3 by 3 grid where every weight is 1/9, we multiply each of the nine values by 1/9 and add them up. That is just the average of the nine numbers: (10+12+200+9+11+14+11+10+13) / 9, which comes to 290 / 9, about 32. The centre pixel moves from 11 to roughly 32, pulled upward by that one bright outlier, but nowhere near as extreme as the original 200. Every pixel in the image gets this same treatment: look at its neighbourhood, apply the kernel, write the result as the new value for that pixel. This sliding process, repeated across the whole image, is what OpenCV's blur functions do under the hood.

Two details matter in practice, even though you rarely have to think about them directly. First, pixels near the border of the image do not have a full neighbourhood on every side; OpenCV handles this automatically by extending or reflecting the edge pixels outward, so you do not get a thin frame of garbage around your filtered image. Second, a kernel's weights do not have to be equal, and that is exactly what separates one kind of blur from another. Changing the weights, or changing how the output value is computed from the neighbourhood, gives you box blur, Gaussian blur, median blur and bilateral filtering, each with a different trade-off.

Setting up some test images

Rather than downloading a photo, we can build a small synthetic image with NumPy and OpenCV, which means every example in this article runs the same way on any machine. This assumes opencv-python 4.x and a recent NumPy (4.5 or later, and 1.20 or later, are both fine). We will draw a plain grey background with a white square and a black circle on it, giving us flat regions and a couple of sharp edges to test the filters against, then add some artificial noise on top.

import cv2
import numpy as np

# Build a simple synthetic image: a grey background with a white
# square and a black circle, so we have flat areas and sharp edges
img = np.full((300, 300, 3), 200, dtype=np.uint8)              # grey background
cv2.rectangle(img, (50, 50), (150, 150), (255, 255, 255), -1)   # white square
cv2.circle(img, (200, 200), 60, (0, 0, 0), -1)                  # black circle

# Add Gaussian noise: random values drawn from a normal distribution
rng = np.random.default_rng(seed=0)
noise = rng.normal(loc=0, scale=25, size=img.shape)
noisy = img.astype(np.int16) + noise.astype(np.int16)
noisy = np.clip(noisy, 0, 255).astype(np.uint8)

cv2.imwrite("clean.png", img)
cv2.imwrite("noisy.png", noisy)

Running this writes two files. Open clean.png and you will see crisp, flat shapes: a uniform grey field, a perfectly white square, a perfectly black circle. Open noisy.png and the same shapes are now covered in a fine grain, as if the photo had been taken on a cheap sensor in low light, which is exactly what we simulated. The scale of 25 we chose for the noise means most random offsets fall within about plus or minus 25 brightness levels of the true value, which is a fairly strong but realistic amount of sensor noise. We clip the result to the 0 to 255 range because pixel values cannot go negative or above 255, and we work in a 16-bit integer type temporarily so that adding noise does not silently wrap around (more on that trap later).

The simplest filter: the box blur

The box blur, also called the mean filter, uses the equal-weight kernel from our worked example above: every pixel in the neighbourhood counts the same, and the new value is just their average. OpenCV gives you this with a single call.

import cv2

noisy = cv2.imread("noisy.png")

blur3 = cv2.blur(noisy, (3, 3))
blur9 = cv2.blur(noisy, (9, 9))
blur21 = cv2.blur(noisy, (21, 21))

cv2.imwrite("box_3.png", blur3)
cv2.imwrite("box_9.png", blur9)
cv2.imwrite("box_21.png", blur21)

cv2.blur takes the source image and a kernel size as a (width, height) tuple. Looking at the three results side by side tells you most of what you need to know about kernel size. box_3.png, using a 3 by 3 kernel, looks only slightly smoother than the original; the grain is softened but still visible, because each output pixel only ever saw eight neighbours plus itself. box_9.png looks noticeably cleaner, the grain has mostly disappeared, but the edge of the white square and the black circle have both become a soft, blurry transition several pixels wide instead of a sharp line. box_21.png is very smooth, almost no trace of noise remains, but the shapes themselves have lost their crispness, and if you look closely you may notice a faint rectangular, blocky quality to the blur, a sign that every neighbour was weighted equally, including the near and far ones.

This is the fundamental trade-off of every blur: a bigger neighbourhood removes more noise but destroys more real detail. There is no kernel size that is simply correct; it depends on how noisy your image is and how much fine detail you need to keep. In practice, the box blur is rarely the final choice for serious work, because treating the nearest and the farthest pixel in the neighbourhood as equally important does not match how noise or real structure actually behaves, but it is the easiest filter to reason about and a good one to start with.

Gaussian blur: giving more weight to nearby pixels

A Gaussian blur fixes the box blur's biggest weakness by using a kernel whose weights are not all equal. Instead, the weight given to each neighbour follows a Gaussian, or bell-curve, distribution centred on the pixel itself: the centre pixel gets the highest weight, pixels one step away get a bit less, pixels two steps away get less still, and so on. This matches intuition much better: a pixel right next to you tells you more about your true value than a pixel five steps away, so it should count for more.

The spread of that bell curve is controlled by a parameter called sigma. A small sigma produces a tight, narrow bell curve, so only the very closest pixels get meaningful weight and the blur is subtle. A large sigma produces a wide, flat bell curve, so pixels further away still contribute noticeably, and the blur is much stronger. OpenCV's function takes both a kernel size and a sigma value, which can feel redundant at first, but the two serve slightly different roles: the kernel size sets a hard cutoff, how far out the neighbourhood is allowed to reach at all, while sigma sets how quickly the weight falls off within that neighbourhood.

import cv2

noisy = cv2.imread("noisy.png")

gauss_low = cv2.GaussianBlur(noisy, (9, 9), sigmaX=1)
gauss_high = cv2.GaussianBlur(noisy, (9, 9), sigmaX=5)

cv2.imwrite("gauss_sigma1.png", gauss_low)
cv2.imwrite("gauss_sigma5.png", gauss_high)

Both calls use the same 9 by 9 kernel size, but gauss_sigma1.png barely looks different from the original noisy image, because with sigma 1 almost all of the weight sits on the centre pixel and its immediate neighbours, so the wider parts of the 9 by 9 window contribute almost nothing. gauss_sigma5.png looks much smoother, closer to the strong box blur from before, but if you compare the edges of the shapes you will notice the transition looks softer and more natural, without the faint blockiness of the box blur, because the weighting fades gradually rather than cutting off sharply. If you pass a sigma of 0, OpenCV will pick a reasonable sigma automatically based on the kernel size, which is a convenient default when you are not trying to fine-tune anything.

Gaussian blur is the workhorse of classical image processing. It is the filter most often applied before edge detection, because it reduces noise without introducing the kind of blocky artifacts that can create false edges of their own, and because it is a reasonably good approximation of the blur a real lens already introduces. When you see a computer vision tutorial say something like "first blur the image slightly", it almost always means a Gaussian blur with a small kernel, such as 5 by 5, applied before the real work begins.

Median blur: good for speckle noise

Box and Gaussian blurs both compute some kind of weighted average of the neighbourhood, and averages have a well-known weakness: they are easily dragged around by extreme outliers. Go back to our worked example, the patch with a value of 200 sitting among a cluster of values around 10 to 14. The average of that patch was about 32, noticeably higher than the true local brightness, because a single wild number still pulls the average upward.

The median blur takes a different approach: instead of averaging the neighbourhood, it sorts all the values in the neighbourhood and picks the middle one, the median. In our 3 by 3 patch, sorting 9, 10, 10, 11, 11, 12, 13, 14, 200 gives a median of 11, which is exactly the kind of value we expected before the outlier appeared. The 200 is simply ignored, because no matter how extreme it is, it can only ever be one entry near the top of the sorted list; it takes more than half the neighbourhood being corrupted before the median itself gets dragged off course.

This makes median blur extremely effective against a particular kind of noise called salt-and-pepper noise, where individual pixels are randomly flipped to pure white or pure black, as if someone had sprinkled salt and pepper on the image, rather than every pixel being nudged a little as with Gaussian sensor noise. Let's generate that kind of noise and compare median blur with Gaussian blur on it.

import cv2
import numpy as np

img = cv2.imread("clean.png")

sp = img.copy()
rng = np.random.default_rng(seed=1)
mask = rng.random(img.shape[:2])
sp[mask < 0.02] = [0, 0, 0]          # pepper: 2% of pixels turned black
sp[mask > 0.98] = [255, 255, 255]    # salt: 2% of pixels turned white

cv2.imwrite("salt_pepper.png", sp)

median = cv2.medianBlur(sp, 5)
gauss = cv2.GaussianBlur(sp, (5, 5), 0)

cv2.imwrite("sp_median.png", median)
cv2.imwrite("sp_gauss.png", gauss)

salt_pepper.png shows the clean shapes peppered with scattered black and white dots, about 4 percent of pixels in total. Compare the two results: sp_median.png looks almost identical to the original clean image, the stray black and white dots have essentially vanished, because in any 5 by 5 neighbourhood (25 pixels), a handful of salt-and-pepper outliers is nowhere near enough to shift the median away from the true local value. sp_gauss.png, by contrast, still shows faint grey smudges where the dots used to be; the Gaussian blur smeared each stray black or white pixel into its surroundings rather than removing it, because an average, even a weighted one, always includes some contribution from every value in the neighbourhood.

cv2.medianBlur takes a single kernel size, which must be an odd number (3, 5, 7 and so on), since the median needs a well-defined middle value. Use median blur specifically when you know or suspect your noise looks like isolated bad pixels, such as dead or stuck pixels on a sensor, or compression artefacts; for the more common, gentler graininess of ordinary sensor noise, Gaussian blur usually looks more natural.

Bilateral filter: smoothing without losing edges

Every filter so far smooths flat regions and edges alike. Look back at the box and Gaussian results on the noisy image: the grey background became cleaner, but the boundary between the white square and the grey background also became blurrier, a wider, softer transition instead of a crisp line. Sometimes that is exactly what you do not want. If you are cleaning up noise on a face before detecting its features, or on a page of text before reading it, you want the flat regions smoothed but the real edges, the ones that carry meaningful information, kept sharp.

The bilateral filter solves this by changing what counts as a "neighbour" worth averaging. A Gaussian blur weights pixels only by how physically close they are. The bilateral filter weights pixels by two things at once: how physically close they are, exactly as before, and how similar their brightness or colour is to the centre pixel. A nearby pixel that is almost the same colour gets a high weight, same as in a normal Gaussian blur. A nearby pixel that happens to sit just across a strong edge, and therefore has a very different colour, gets a low weight even though it is spatially close, because the filter treats it as probably belonging to a different object. The practical effect is that the filter happily blends pixels within a flat, noisy region together, since they are all similar in colour, but it refuses to blend pixels across a sharp boundary, since their colours differ too much.

import cv2

noisy = cv2.imread("noisy.png")

gauss = cv2.GaussianBlur(noisy, (9, 9), sigmaX=3)
bilateral = cv2.bilateralFilter(noisy, d=9, sigmaColor=75, sigmaSpace=75)

cv2.imwrite("compare_gauss.png", gauss)
cv2.imwrite("compare_bilateral.png", bilateral)

cv2.bilateralFilter takes three main arguments. d is the diameter of the pixel neighbourhood considered, similar in spirit to a kernel size. sigmaColor controls how different two pixel values are allowed to be while still counting as similar; a larger value means the filter tolerates bigger colour differences and blends more aggressively, a smaller value means it only blends near-identical pixels. sigmaSpace plays the same role as sigma in a Gaussian blur, controlling how far away spatially a pixel can be and still be considered. Comparing the two output files, compare_gauss.png shows the familiar soft, blurred edge around the square and circle, while compare_bilateral.png shows a background that is just as smooth and noise-free, but with the edge of the square and the circle remaining close to its original sharpness.

This edge-preserving property makes the bilateral filter popular for cleaning up photos of faces, skin smoothing effects in photo apps, and any preprocessing step where you need to remove texture noise while keeping object boundaries intact for later steps like contour detection in Part 6. The cost is speed: a bilateral filter has to compare every pixel to every neighbour on two criteria instead of just one, which makes it noticeably slower than a box or Gaussian blur of the same neighbourhood size. On a single photo this rarely matters, but if you try to run it on every frame of a live webcam feed, as Part 8 will have you doing, you may find it struggles to keep up at larger kernel sizes, whereas Gaussian blur usually runs comfortably in real time.

Going the other way: sharpening with an unsharp mask

Everything so far has removed detail on purpose. Sometimes you want the opposite: an image that looks a little flat or soft, and you want to emphasise the edges and fine detail it does have. The classic technique for this, used long before digital cameras in darkroom photography, is called unsharp masking, and despite the name it is a sharpening technique.

The idea is to use a blur as a measuring tool rather than a final result. If you blur an image and then subtract that blurred version from the original, what is left over is roughly the fine detail and edges, the high-frequency content that the blur smoothed away. Adding a scaled-up copy of that leftover detail back onto the original image exaggerates exactly the fine detail and edges, making the image look sharper.

import cv2

img = cv2.imread("clean.png")

blurred = cv2.GaussianBlur(img, (0, 0), sigmaX=3)
amount = 1.5
sharpened = cv2.addWeighted(img, 1 + amount, blurred, -amount, 0)

cv2.imwrite("sharpened.png", sharpened)

Here, passing a kernel size of (0, 0) to cv2.GaussianBlur tells OpenCV to work out a sensible kernel size automatically from the sigma value we gave it. cv2.addWeighted computes a weighted combination of two images, alpha times the first plus beta times the second plus a constant: here that is (1 + amount) times the original minus amount times the blurred version, which algebraically equals original + amount times (original minus blurred), the unsharp mask formula. With amount set to 1.5, the edge detail is boosted by one and a half times its original strength. Because our clean.png has no noise, sharpening it mostly accentuates the already-sharp edges of the square and circle, giving them a faint bright-and-dark fringe, called a halo, right along the boundary; this is a normal and recognisable signature of unsharp masking, visible in almost every sharpened photograph if you zoom in far enough.

A practical warning: sharpening amplifies whatever detail is in the image, and that includes noise, since noise is also high-frequency content from the filter's point of view. Sharpening a noisy photo directly will make the noise more obvious, not less. If an image needs both denoising and sharpening, denoise first with one of the filters above, then sharpen the result, never the other way round.

A quick guide to choosing a filter

• Box blur (cv2.blur): the simplest and fastest option, treats every neighbour equally. Good for a quick, rough smoothing when quality does not matter much, or as a teaching example. Can look blocky at larger kernel sizes.
• Gaussian blur (cv2.GaussianBlur): the general-purpose default. Weights nearby pixels more than distant ones, giving a smoother, more natural-looking result than a box blur. The standard choice before edge detection.
• Median blur (cv2.medianBlur): the right tool when the noise looks like isolated bad pixels, stray black or white speckles, dead sensor pixels, or similar impulse noise. Poor choice for smooth gradients, since it can flatten subtle shading.
• Bilateral filter (cv2.bilateralFilter): use when you need to remove texture or noise from flat areas while keeping object boundaries sharp, for example before contour detection. Noticeably slower than the others, so use smaller images or smaller neighbourhoods for real-time video.
• Unsharp mask (GaussianBlur plus addWeighted): use to make an already clean image look crisper. Apply after denoising, never before, and expect a faint halo around strong edges if you push the amount too high.

Common mistakes

A few mistakes come up repeatedly when people first start applying filters, and they are worth calling out explicitly.

The first is choosing a kernel size without thinking about what it means for your specific image. A 21 by 21 kernel on a 300 by 300 pixel synthetic shape, as we used above, is already quite aggressive; the same kernel size on a 4000 by 3000 pixel photo from a phone camera would barely be noticeable, because the kernel only covers a tiny fraction of the image. Kernel size should be thought of relative to the size of the details you want to keep, not as an absolute number that works everywhere.

The second is forgetting that cv2.medianBlur, and the kernel size argument in most OpenCV filters, generally expects an odd number. An even kernel size has no single centre pixel, which causes either an error or undefined behaviour depending on the function, so stick to 3, 5, 7, 9 and so on.

The third, and the one that catches people out most often, is doing pixel arithmetic directly with NumPy instead of with OpenCV's functions, and running into silent overflow. A pixel value in a standard uint8 image can only range from 0 to 255. If you add two images, or add a noise array, directly with the plus operator on uint8 arrays, NumPy will wrap values around instead of capping them, which produces completely wrong colours rather than an error you would notice immediately.

import cv2
import numpy as np

a = np.array([200], dtype=np.uint8)
b = np.array([100], dtype=np.uint8)

print(a + b)          # NumPy wraps around: 300 - 256 = 44
print(cv2.add(a, b))  # OpenCV saturates at 255

Running this prints 44 from the plain NumPy addition, because 200 plus 100 is 300, which overflows the 0 to 255 range of a uint8 and wraps back around to 44, and prints 255 from cv2.add, which clips, or saturates, the result at the maximum valid value instead of wrapping. This is exactly why the noise-generation code earlier in this article converted the image to int16 before adding noise, then clipped and converted back to uint8 manually, rather than adding the noise straight onto the uint8 array. Whenever you combine two images or add a signed adjustment to pixel values by hand, either use OpenCV's arithmetic functions such as cv2.add, cv2.subtract and cv2.addWeighted, which saturate correctly, or convert to a wider integer type yourself, clip the result into the valid range, and convert back.

The fourth mistake is reaching for the bilateral filter out of habit for every job, including real-time video, and then being puzzled when a webcam feed starts lagging. It is the most expensive of the filters covered here; for anything running frame by frame, as you will do in Part 8, a Gaussian blur is almost always fast enough and a good first choice, with the bilateral filter reserved for still images or for video pipelines where you have already checked the frame rate holds up.

Summary and what's next

Every filter in this article works the same basic way: look at a small neighbourhood around each pixel, combine the values in that neighbourhood according to some rule, and write the result as the new pixel value. A box blur averages the neighbourhood with equal weights, which is simple but can look blocky. A Gaussian blur averages with weights that fall off smoothly with distance, giving a more natural result and making it the standard choice before further processing. A median blur replaces each pixel with the middle value of its neighbourhood rather than an average, which makes it very good at removing isolated speckle noise while leaving smooth gradients alone. A bilateral filter adds a second condition, similarity of colour as well as closeness of position, so it can smooth flat, noisy regions while keeping real edges sharp, at the cost of extra computation. Unsharp masking runs the idea in reverse, using a blur to measure fine detail and then adding that detail back, amplified, to make an image look crisper.

With a cleaned-up image in hand, the natural next question is: where exactly are the edges and shapes in it. Part 6 picks up directly from here, using the kind of smoothed image you now know how to produce as the input to edge detection, and showing how OpenCV turns those edges into contours, outlines you can measure, count and compare, the foundation for the coin-counting and document-scanning projects later in this path.