From edges to a simple yes-or-no question

In part 6 you learned to find edges by looking at how brightness changes from one pixel to the next, and then to trace those edges into contours and shapes. Edges answer the question "where does something change". This part answers a more direct question: "which pixels belong to the thing I care about, and which do not". That distinction sounds small, but it is the backbone of a huge number of computer vision tasks. If you want to count coins on a table, you first need to know which pixels are coin and which are table. If you want to scan a document with your phone, you first need to know which pixels are paper and which are shadow or background. If you want to track a red ball with a webcam, you first need to know which pixels are red.

The tool for answering that question is thresholding: picking a cutoff value and splitting pixels into two groups based on it. The result of thresholding is usually a mask, a black-and-white image where white means "this pixel matters" and black means "ignore this pixel". Once you have a mask, you can combine it with the original image, count the white pixels, find their outlines with the contour tools from part 6, or clean it up before using it further. This article covers how to build masks, how to make thresholding robust to messy lighting, how to do it by colour as well as by brightness, and how to clean up the inevitable noise.

Every example below uses opencv-python 4.x and NumPy, and builds its own small test images with cv2 drawing functions, so you can run everything on a laptop without downloading any sample files. If you do not have the libraries yet, install them with pip install opencv-python numpy.

What a threshold actually does

Recall from part 3 that a greyscale image is a 2D grid of numbers, usually stored as 8-bit integers from 0 (black) to 255 (white). A threshold is just a single cutoff number, say 127. Thresholding walks over every pixel and asks: is this value above the cutoff or not? Pixels above become one fixed value (often 255, pure white), and pixels at or below become another fixed value (often 0, pure black). That is the entire idea. The craft is in choosing the cutoff and in deciding what "above" and "below" should map to, which is exactly what the rest of this article covers.

Binary images and masks

The output of a threshold operation is called a binary image because each pixel only takes one of two values. When that binary image is used to select or hide parts of another image, people call it a mask. A mask has exactly the same width and height as the image it applies to, but only one channel, with values of 0 (excluded) or 255 (included). You will see both names used for the same kind of array; "binary image" emphasises what it looks like, "mask" emphasises what you do with it.

Global thresholding with cv2.threshold

OpenCV's basic thresholding function is cv2.threshold. It takes a single-channel image, a cutoff value, a maximum value to assign, and a thresholding type, and returns both the threshold actually used and the resulting image. Here is a small synthetic example: a dark background with a mid-grey ring and an even darker centre, built entirely with cv2.circle so there is nothing to download.

🐍Python
import cv2
import numpy as np

# Build a 300x300 grayscale canvas: black background, a grey ring, a darker centre
canvas = np.zeros((300, 300), dtype=np.uint8)
cv2.circle(canvas, (150, 150), 80, 200, -1)  # grey disc, value 200
cv2.circle(canvas, (150, 150), 40, 60, -1)   # darker disc on top, value 60

ret, mask = cv2.threshold(canvas, 127, 255, cv2.THRESH_BINARY)
print("Threshold used:", ret)
print("Unique values in mask:", np.unique(mask))
cv2.imwrite("mask_binary.png", mask)

The canvas has three regions: background at 0, a ring at 200, and a centre disc at 60. With a cutoff of 127, cv2.THRESH_BINARY keeps only pixels strictly greater than 127, setting them to 255, and sets everything else to 0. So the ring (200) survives as white, while both the background (0) and the centre (60) are flattened to black, even though they started at different brightness levels. If you open mask_binary.png you will see a white ring with a black hole in the middle, even though there was no real hole in the original shape, just a darker patch. This is worth sitting with for a moment: thresholding always throws information away on purpose. Anything that ends up on the same side of the cutoff looks identical afterwards, no matter how different it was before.

The five threshold types

cv2.threshold supports more than one way of mapping pixels once you know which side of the cutoff they fall on. They all use the same cutoff and maxval but behave differently:

  • THRESH_BINARY: pixels above the cutoff become maxval, others become 0.
  • THRESH_BINARY_INV: the exact opposite, pixels above the cutoff become 0, others become maxval.
  • THRESH_TRUNC: pixels above the cutoff are capped at the cutoff value, pixels at or below keep their original value unchanged.
  • THRESH_TOZERO: pixels above the cutoff keep their original value, pixels at or below are set to 0.
  • THRESH_TOZERO_INV: the opposite of TOZERO, pixels above the cutoff are set to 0, pixels at or below keep their original value.

Running all five on the same ring-and-disc canvas from before makes the differences concrete:

🐍Python
import cv2
import numpy as np

canvas = np.zeros((300, 300), dtype=np.uint8)
cv2.circle(canvas, (150, 150), 80, 200, -1)
cv2.circle(canvas, (150, 150), 40, 60, -1)

types = {
    "THRESH_BINARY": cv2.THRESH_BINARY,
    "THRESH_BINARY_INV": cv2.THRESH_BINARY_INV,
    "THRESH_TRUNC": cv2.THRESH_TRUNC,
    "THRESH_TOZERO": cv2.THRESH_TOZERO,
    "THRESH_TOZERO_INV": cv2.THRESH_TOZERO_INV,
}

for name, flag in types.items():
    _, result = cv2.threshold(canvas, 127, 255, flag)
    print(name, "-> unique values:", np.unique(result))

With a cutoff of 127 and the values 0 (background), 60 (centre) and 200 (ring) in the image, you should see: BINARY gives {0, 255} with the ring as 255; BINARY_INV gives {0, 255} but with the roles swapped, so the ring becomes 0 and everything else becomes 255; TRUNC gives {0, 60, 127}, since the ring is capped at 127 but the centre and background keep their original values; TOZERO gives {0, 200}, since the ring keeps its value and everything at or below the cutoff is zeroed; TOZERO_INV gives {0, 60}, the mirror image of TOZERO. TRUNC and TOZERO are less commonly used for building masks, since they keep some original brightness information rather than producing a clean two-value image, but they are useful when you want to flatten bright highlights (TRUNC) or suppress faint background noise while keeping strong signal intact (TOZERO).

Letting the image choose its own threshold: Otsu's method

Picking 127 by hand worked for the synthetic example because the values were chosen to make a point. Real images rarely come with a convenient cutoff baked in, and guessing one by eye is slow and brittle. Otsu's method, named after the method's author, looks at the image's histogram, the count of how many pixels have each brightness value, and automatically picks the cutoff that best separates the histogram into two groups. It works well when the histogram is bimodal, meaning it has two clear humps, for example a bright object against a darker background, with relatively few pixels in between. Otsu finds the valley between the humps.

🐍Python
import cv2
import numpy as np

# Build a noisy image with two brightness groups: a dim background and a brighter square
rng = np.random.default_rng(42)
image = rng.normal(60, 10, (200, 200))
image[50:150, 50:150] = rng.normal(180, 10, (100, 100))
image = np.clip(image, 0, 255).astype(np.uint8)

ret, otsu_mask = cv2.threshold(image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
print("Otsu chose threshold:", ret)
cv2.imwrite("otsu_mask.png", otsu_mask)

Notice the cutoff value passed to cv2.threshold is 0, which is ignored when you combine THRESH_BINARY with THRESH_OTSU; Otsu computes its own value and returns it as ret. Because the two groups here are centred around 60 and 180, Otsu should land somewhere roughly between them, but the exact printed number will depend on the random noise, so do not expect a fixed digit, just a value sitting in the gap between the two humps of the histogram. Otsu is convenient because it removes the guesswork, but it has a real limitation: it assumes one global cutoff works for the whole image. If lighting is uneven, one part of the background might be brighter than part of the actual object, and no single number, however cleverly chosen, can separate them correctly everywhere.

When the lighting is not even: adaptive thresholding

Think about photographing a sheet of paper on a desk with one lamp to the side. One edge of the page catches more light than the other. A single global threshold, Otsu or otherwise, has to pick one number for the entire image, so it will either miss faint ink on the darker side or mistake the darker background for ink. Adaptive thresholding fixes this by not using one number at all. Instead, for every pixel, it looks at a small neighbourhood around that pixel, computes a local average (or a local weighted average), and compares the pixel to that local value rather than to a global one. The cutoff effectively slides around the image to track local brightness.

🐍Python
import cv2
import numpy as np

h, w = 300, 300
# Simulate uneven lighting: a horizontal brightness gradient across the image
gradient = np.tile(np.linspace(30, 220, w), (h, 1)).astype(np.uint8)
image = gradient.copy()

# Three identical dark rectangles (same "ink"), placed at different brightness zones
cv2.rectangle(image, (30, 100), (90, 140), 10, -1)
cv2.rectangle(image, (150, 100), (210, 140), 10, -1)
cv2.rectangle(image, (240, 100), (280, 140), 10, -1)

# A single global cutoff
_, global_thresh = cv2.threshold(image, 100, 255, cv2.THRESH_BINARY_INV)

# A cutoff that adapts to the local neighbourhood
adaptive_thresh = cv2.adaptiveThreshold(
    image, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
    cv2.THRESH_BINARY_INV, blockSize=31, C=10
)

cv2.imwrite("global_thresh.png", global_thresh)
cv2.imwrite("adaptive_thresh.png", adaptive_thresh)

Open global_thresh.png and you should see the left third of the image turned almost entirely white, even though only the rectangle there was meant to be picked out, because the dim background on that side happened to fall below the single cutoff of 100 as well. Open adaptive_thresh.png and you should see only the three rectangles picked out cleanly, with the background staying black across the whole gradient. That happens because adaptiveThreshold never compares a pixel to a fixed number for the whole picture; it compares each pixel to the brightness of its own neighbourhood, defined by blockSize (the size of the square neighbourhood, which must be an odd number) and C (a constant subtracted from the local average to fine-tune sensitivity). ADAPTIVE_THRESH_GAUSSIAN_C weights nearby pixels more heavily than far ones when computing the local average; ADAPTIVE_THRESH_MEAN_C just takes a plain average. You will use adaptive thresholding again in the document scanner project later in this path, where uneven lighting across a photographed page is exactly the problem it solves.

Thresholding by colour: masks in HSV

Everything so far has thresholded on brightness in a single-channel image. Often what you actually want is to pick out pixels of a certain colour, for example a specific coloured object on a webcam feed. Part 3 covered HSV (hue, saturation, value) as a colour space where hue captures "what colour" largely independently of how light or dark it is. That makes HSV a much more forgiving space for colour-based thresholding than BGR, because a red object under dim light and the same red object under bright light will have very different BGR values but similar hue.

OpenCV represents hue for 8-bit images on a 0 to 179 scale rather than the more common 0 to 360 degrees, simply because a single byte cannot hold values up to 360. The function for colour-range thresholding is cv2.inRange, which takes a lower and upper bound for each channel and returns 255 where every channel of a pixel falls inside its bound, and 0 otherwise.

🐍Python
import cv2
import numpy as np

# A white canvas with three solid coloured squares (OpenCV uses BGR channel order)
canvas = np.full((200, 300, 3), 255, dtype=np.uint8)
cv2.rectangle(canvas, (10, 10), (90, 90), (255, 0, 0), -1)    # blue square
cv2.rectangle(canvas, (110, 10), (190, 90), (0, 255, 0), -1)  # green square
cv2.rectangle(canvas, (210, 10), (290, 90), (0, 0, 255), -1)  # red square

hsv = cv2.cvtColor(canvas, cv2.COLOR_BGR2HSV)

# Pure blue sits around hue 120 on OpenCV's 0-179 scale; give it some margin
lower_blue = np.array([100, 150, 50])
upper_blue = np.array([130, 255, 255])
blue_mask = cv2.inRange(hsv, lower_blue, upper_blue)

print("Blue pixels found:", cv2.countNonZero(blue_mask))

The square is drawn from (10, 10) to (90, 90), so you should see a count close to the area of that square, roughly 80 by 80 pixels, give or take a pixel or two at the edges depending on exactly how OpenCV rasterises the rectangle corners. The saturation and value lower bounds (150 and 50) matter as much as the hue bounds: they exclude the white background, which has very low saturation, and would otherwise slip through a hue-only test because white has no defined hue but can still fall inside a wide numeric range by coincidence of how the test is written. Always set sensible saturation and value floors when colour-thresholding, or washed-out or very dark pixels will leak into your mask.

One quirk worth knowing: red sits at both ends of the hue wheel, near 0 and near 179 on OpenCV's scale, because hue is circular and red is placed at the seam. If you threshold for red with a single range you will miss half of it. The usual fix is to build two masks, one for each end of the range, and combine them with cv2.bitwise_or, which is covered next.

What a mask lets you do

Once you have a mask, whether it came from a brightness threshold or a colour range, you can use it to isolate exactly the pixels you care about in the original image. The tool for this is a family of bitwise operations that work pixel by pixel, treating 0 as false and any nonzero value (typically 255) as true:

  • cv2.bitwise_and(img1, img2, mask=...) keeps a pixel only where both inputs (and the mask, if given) agree it should be kept.
  • cv2.bitwise_or(img1, img2) keeps a pixel if either input has it set.
  • cv2.bitwise_not(img) flips every pixel, turning the mask inside out.
  • cv2.bitwise_xor(img1, img2) keeps a pixel where exactly one of the two inputs has it set, useful for finding where two masks disagree.

A very common pattern is to pass the same image twice to bitwise_and along with a mask, which effectively keeps the original colours wherever the mask is white and blanks out everything else to black:

🐍Python
import cv2
import numpy as np

canvas = np.full((200, 300, 3), 255, dtype=np.uint8)
cv2.rectangle(canvas, (10, 10), (90, 90), (255, 0, 0), -1)
cv2.rectangle(canvas, (110, 10), (190, 90), (0, 255, 0), -1)
cv2.rectangle(canvas, (210, 10), (290, 90), (0, 0, 255), -1)

hsv = cv2.cvtColor(canvas, cv2.COLOR_BGR2HSV)
lower_blue = np.array([100, 150, 50])
upper_blue = np.array([130, 255, 255])
blue_mask = cv2.inRange(hsv, lower_blue, upper_blue)

result = cv2.bitwise_and(canvas, canvas, mask=blue_mask)
cv2.imwrite("blue_only.png", result)

Open blue_only.png and you should see the blue square exactly where it was, and everywhere else turned pure black, including the green and red squares and the white background. This is the pattern behind a lot of colour-tracking demos: threshold by colour to get a mask, then use the mask to either extract the object, draw a box around its contour using the tools from part 6, or simply count its pixels with cv2.countNonZero to estimate how much of the frame it covers.

Cleaning up a mask with morphology

Real masks from real images are rarely as tidy as the ones in these examples. Camera sensor noise, slightly uneven colour, and compression artefacts typically leave a mask peppered with small white specks in the background and small black holes inside the object. Counting contours on a mask like that will give you far too many small blobs, or will fragment one object into several pieces. The fix is a family of operations called morphology, which work directly on binary images using a small shape called a structuring element, or kernel, that slides across the image similarly to the filters from part 5, but with a simpler, non-weighted rule.

  • Erosion shrinks white regions: a pixel stays white only if the entire neighbourhood under the kernel is also white. This eats away small white specks and thins edges.
  • Dilation grows white regions: a pixel becomes white if any pixel in the neighbourhood is white. This fills in small black holes and thickens edges.
  • Opening is erosion followed by dilation. It removes small white specks from the background while roughly preserving the size of larger white regions, because the dilation grows them back afterwards.
  • Closing is dilation followed by erosion. It fills small black holes inside white regions while roughly preserving their outer boundary.

A useful way to remember the order: opening "opens up" small gaps by deleting tiny bright noise first; closing "closes up" small gaps by filling tiny dark noise first. Here is a worked example that adds artificial speckle noise to the blue mask from before and cleans it up:

🐍Python
import cv2
import numpy as np

canvas = np.full((200, 300, 3), 255, dtype=np.uint8)
cv2.rectangle(canvas, (10, 10), (90, 90), (255, 0, 0), -1)
hsv = cv2.cvtColor(canvas, cv2.COLOR_BGR2HSV)
lower_blue = np.array([100, 150, 50])
upper_blue = np.array([130, 255, 255])
blue_mask = cv2.inRange(hsv, lower_blue, upper_blue)

# Add random salt-and-pepper noise to simulate a messy real-world mask
rng = np.random.default_rng(0)
noise = rng.random(blue_mask.shape)
noisy_mask = blue_mask.copy()
noisy_mask[noise < 0.02] = 255   # random white specks in the background
noisy_mask[noise > 0.98] = 0     # random black holes inside the square

kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
opened = cv2.morphologyEx(noisy_mask, cv2.MORPH_OPEN, kernel)
cleaned = cv2.morphologyEx(opened, cv2.MORPH_CLOSE, kernel)

cv2.imwrite("noisy_mask.png", noisy_mask)
cv2.imwrite("cleaned_mask.png", cleaned)

Compare noisy_mask.png and cleaned_mask.png: the first should show scattered white dots in the background and scattered black dots inside the square, while the second should show a much cleaner square with most of that speckle gone. cv2.getStructuringElement builds the kernel shape; MORPH_ELLIPSE gives a roughly round kernel, which tends to look more natural than a plain square (MORPH_RECT) for most real objects. The kernel size is a trade-off: too small and it will not remove larger noise blobs or fill larger holes; too large and it will start eating away real detail, rounding off corners or merging nearby objects that should stay separate. A common approach is to start with a small kernel, such as 3 by 3 or 5 by 5, and increase it only if noise persists.

Putting it together: counting blobs in a cleaned mask

This is a good point to connect thresholding and morphology back to the contour tools from part 6. A typical small pipeline for something like counting objects is: blur slightly to reduce sensor noise, threshold (often with Otsu, since you may not know the right cutoff in advance), clean the mask with morphology, then run cv2.findContours on the cleaned mask and count the results. Here is a self-contained version using a synthetic image of coin-like circles on a noisy background:

🐍Python
import cv2
import numpy as np

# Build a noisy grey background with a few coin-like bright circles on it
canvas = np.full((300, 400), 50, dtype=np.uint8)
rng = np.random.default_rng(7)
noise = rng.normal(0, 8, canvas.shape)
canvas = np.clip(canvas.astype(np.int16) + noise, 0, 255).astype(np.uint8)

coins = [((70, 70), 35, 200), ((180, 90), 30, 190),
         ((300, 80), 40, 210), ((150, 220), 32, 195)]
for center, radius, value in coins:
    cv2.circle(canvas, center, radius, value, -1)

blurred = cv2.GaussianBlur(canvas, (5, 5), 0)
_, mask = cv2.threshold(blurred, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (7, 7))
clean_mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)

contours, _ = cv2.findContours(clean_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
print("Objects found:", len(contours))

With four well-separated circles and a modest amount of noise, you should see Objects found: 4 printed, but treat that as a reasonable expectation rather than a guarantee. If two circles end up close enough that the morphological opening does not fully separate them, or if the noise level and kernel size interact in an unlucky way, you might see three or five instead. That sensitivity is itself a useful lesson: thresholding and morphology have parameters (the cutoff, the kernel shape and size, the blur amount), and small changes in those parameters can change the final count. When you build the coin-counting project later in this path, expect to spend time nudging these parameters against a real photograph, not just trusting the first settings that work on a clean synthetic test.

Common mistakes

  • Forgetting that OpenCV reads and writes colour images in BGR order, not RGB, which matters when you hand-pick colour values for drawing test images or for bounds.
  • Thresholding for red in HSV with a single hue range and missing half of it, because red sits at both the 0 end and the 179 end of OpenCV's hue scale; combine two ranges with cv2.bitwise_or instead.
  • Passing a 3-channel colour image to cv2.threshold, which expects a single-channel image for the basic thresholding types; convert to grayscale first with cv2.cvtColor.
  • Mixing up THRESH_BINARY and THRESH_BINARY_INV and ending up with a mask where foreground and background are swapped, which is easy to miss until something downstream, like a count or a bounding box, looks obviously wrong.
  • Trusting Otsu's method on every image without checking the histogram; it assumes a roughly bimodal distribution of brightness and can pick a poor cutoff on cluttered scenes or images with more than two dominant tones.
  • Using a single global threshold on photographs with uneven lighting, shadows or vignetting, where cv2.adaptiveThreshold would track local brightness far more reliably.
  • Skipping morphological clean-up and feeding a raw, speckled mask straight into cv2.findContours, which inflates the object count with tiny noise blobs or fragments real objects into pieces.
  • Choosing a morphology kernel that is too large, which smooths away real detail or merges separate nearby objects into one, or too small, which leaves visible noise behind.
  • Assuming cv2.findContours always returns the same number of values; the return signature differs between major OpenCV versions, so check which version you are running if your code errors on unpacking.

Summary and what's next

Thresholding turns the question "what is in this image" into the much simpler question "is each pixel in or out", and the result is a mask: a single-channel array of 0s and 255s. You now have several ways to choose the cutoff: a fixed global value when you know the image well, Otsu's method when the histogram has two clear groups and you want the computer to choose for you, and adaptive thresholding when lighting varies across the frame. You have also seen how to threshold by colour using HSV ranges and cv2.inRange, how to apply a mask back onto an image with the bitwise operations, and how to clean up noisy masks with erosion, dilation, opening and closing before trusting a count or a contour.

Part 8 moves from single still images to video and webcam streams, where you will apply exactly these ideas, including thresholding and masks, to a sequence of frames arriving many times per second, and start thinking about what changes when the image in front of you is never quite the same from one moment to the next. The three hands-on projects later in this path, the document scanner, the coin counter and the webcam motion detector, all lean directly on the thresholding and masking techniques from this article, so it is worth running the code above yourself and trying different cutoffs, kernel sizes and colour ranges until the behaviour of each parameter feels familiar.