Semantic and Instance Segmentation

From Boxes to Pixels

Part 7 covered object detection: given an image, draw a box around each object and say what it is. Boxes are useful, but they are blunt instruments. A bounding box around a cyclist also includes a slice of road, a sliver of parked car, maybe a street sign in the corner. If you want to know exactly which pixels belong to the cyclist and which belong to the road behind them, a box cannot tell you. You need something that labels the image at the level of individual pixels.

That is what segmentation does. Instead of producing a short list of boxes and class names, a segmentation model produces an output map that is the same height and width as the input image (or very close to it), where every location holds a label. What that label means, and whether two objects of the same kind are told apart, is exactly what separates the different flavors of segmentation this article covers.

Segmentation sits on top of everything from the earlier parts of this path. It still uses convolution, padding, stride and pooling from part 2. It still benefits from the deep backbones of part 4 and from transfer learning as described in part 6. And as you will see, the best instance segmentation models are literally object detectors from part 7 with an extra piece bolted on. Nothing here is a clean break from what came before, it is an extension of it.

Three Flavors of "Segmentation"

In casual conversation people say "segmentation" to mean several different tasks, and mixing them up is a common source of confusion. Before touching any architecture, it is worth being precise about the vocabulary, because the choice of task determines which network design and which loss function make sense.

Semantic segmentation

Semantic segmentation assigns a class label to every pixel, with no notion of individual objects. Picture a photograph of a field with three sheep standing close together. A semantic segmentation model paints every pixel belonging to any sheep with the same color, say pink for "sheep". It cannot tell you that there are three sheep rather than one large sheep-shaped blob, because its output format has no room for that information. It only answers "what category is this pixel", never "which specific object does this pixel belong to".

A typical semantic segmentation output for a street scene would paint the sky one color, the road another, sidewalks another, buildings another, all the cars one single color regardless of how many cars are present, and so on. This is extremely useful for tasks like figuring out the drivable area of a road, where you do not care how many individual lane markings there are, you just want the boundary between road and not-road.

Instance segmentation

Instance segmentation goes further: it finds each individual object and produces a precise pixel mask for that specific object. Go back to the three sheep. An instance segmentation model would output three separate masks, each one outlining exactly one sheep, even where their bodies overlap in the photo. It combines detection (finding each object, as in part 7) with segmentation (outlining its exact shape).

Instance segmentation is usually restricted to "thing" classes, meaning countable objects with a clear boundary such as cars, people, sheep or bottles. It typically does not bother with "stuff" classes such as sky, grass or road, because it does not make sense to ask "which instance of sky is this". There is one sky.

Panoptic segmentation

Panoptic segmentation is the attempt to have it both ways. Every pixel gets exactly one label: stuff classes (sky, road, grass) get a plain semantic class, and thing classes (car, person, sheep) get both a class and an instance id, so pixel (120, 340) might be labeled "car, instance 2" while pixel (10, 10) is simply labeled "sky". This gives a single, complete description of the scene without forcing a choice between the two earlier tasks. You will see the term in papers and benchmark leaderboards; the underlying building blocks are the same ones covered in this article, combined.

A Worked Example You Can Compute by Hand

Before getting into architectures, it helps to have concrete numbers for the two measurements that come up again and again in segmentation: intersection over union (IoU) and the Dice coefficient. Both measure how well a predicted mask overlaps a true mask, and both are easiest to understand on a tiny grid you can count by eye.

Take a 4 by 4 grid of pixels. Suppose the true mask of an object looks like this, where 1 means "object" and 0 means "background", written row by row from top to bottom: row 0 is all zeros, row 1 is 0 1 1 0, row 2 is 0 1 1 0, and row 3 is all zeros. That is a 2 by 2 square of object pixels sitting in the middle, four pixels of object in total.

Now suppose a model predicts a mask shifted one column to the right: row 0 all zeros, row 1 is 0 0 1 1, row 2 is 0 0 1 1, row 3 all zeros. The predicted mask also has four object pixels, but it overlaps the true mask only partially.

Count the overlap by hand. The intersection is the set of pixels that are 1 in both masks: looking at row 1, column index 2 is 1 in both, and in row 2, column index 2 is also 1 in both; that gives an intersection of 2 pixels. The union is the set of pixels that are 1 in either mask: counting every pixel that is 1 in the true mask or the predicted mask gives 6 pixels (you can also get this with the shortcut: 4 true pixels plus 4 predicted pixels minus the 2 counted twice equals 6).

Intersection over union is simply intersection divided by union: 2 divided by 6, which is about 0.333. The Dice coefficient is defined slightly differently, as two times the intersection divided by the sum of the sizes of the two masks: 2 times 2, divided by (4 plus 4), which is 4 divided by 8, equal to 0.5. Notice Dice gives a higher number than IoU for the same overlap; that is a general property of the two metrics, not a coincidence of this example, and there is a direct formula linking them: Dice equals 2 times IoU divided by (1 plus IoU). Plugging in 0.333 gives 0.666 divided by 1.333, which is 0.5, matching the direct calculation.

Here is the same calculation done in code, using NumPy, to confirm the hand arithmetic and to show the pattern you will reuse whenever you need to score a predicted mask against ground truth.

import numpy as np

true_mask = np.array([
    [0, 0, 0, 0],
    [0, 1, 1, 0],
    [0, 1, 1, 0],
    [0, 0, 0, 0],
])

pred_mask = np.array([
    [0, 0, 0, 0],
    [0, 0, 1, 1],
    [0, 0, 1, 1],
    [0, 0, 0, 0],
])

intersection = np.logical_and(true_mask, pred_mask).sum()
union = np.logical_or(true_mask, pred_mask).sum()
iou = intersection / union
dice = 2 * intersection / (true_mask.sum() + pred_mask.sum())

print(f"Intersection: {intersection}")
print(f"Union: {union}")
print(f"IoU: {iou:.3f}")
print(f"Dice: {dice:.3f}")

Running this (tested with NumPy 1.x, any reasonably recent version works since it only uses basic array logic) prints an intersection of 2, a union of 6, an IoU of 0.333 and a Dice of 0.500, matching the hand calculation exactly. Keep this toy example in mind; every metric discussed later in this article is a variation on exactly this counting exercise, just applied class by class or object by object over a full-size image.

How Semantic Segmentation Networks Work

Why a plain classification CNN will not do

Recall from parts 2 and 3 how a standard image classification CNN is built: a stack of convolution and pooling layers progressively shrinks the spatial size of the feature maps while increasing the number of channels, and at the end a flatten step and one or more fully connected layers collapse everything down to a single vector of class scores. That design throws away spatial layout on purpose, because for classification you only need one answer for the whole image: "this is a dog".

Segmentation needs the opposite: an answer for every location in the image, at (ideally) the same resolution as the input. Flattening and fully connected layers are exactly the wrong tool, because they destroy the very spatial information you need to keep. The fix, developed in the mid-2010s, was to realize that a CNN can be made entirely out of convolutional operations, with no flattening step at all.

Fully Convolutional Networks (FCN)

A fully convolutional network takes a classification backbone such as VGG or ResNet (the kind described in part 4) and replaces its final fully connected layers with 1 by 1 convolutions. A 1 by 1 convolution acts like a fully connected layer applied independently at every spatial location, so it keeps the output as a small feature map instead of collapsing it into a single vector. The result, after the usual downsampling path, is a low-resolution map of class scores, for example 7 by 7 instead of the original 224 by 224.

That low-resolution map then needs to be scaled back up to the original image size, a step usually called upsampling. This can be done with a transposed convolution (sometimes called a deconvolution, though no true mathematical inverse of convolution is being computed) which is a learnable way of expanding a feature map, or with simple bilinear interpolation followed by a few more convolutions to clean up the result. Either way, the network stays convolutional end to end, which is why the approach is called "fully convolutional". This idea, introduced by Long, Shelhamer and Darrell around 2015, was one of the first demonstrations that an off-the-shelf classification backbone could be turned into a dense, per-pixel predictor with relatively modest changes.

The weak point of a plain FCN is that repeated downsampling throws away fine spatial detail, specifically the exact location of edges and boundaries. Upsampling a 7 by 7 map back to 224 by 224, however cleverly, cannot recover detail that was never there. The practical symptom is blobby, imprecise boundaries: a person's silhouette comes out roughly right but fuzzy around the hands and hair.

Encoder-decoder architectures with skip connections

The fix for blobby boundaries is to stop relying purely on the final, most-downsampled feature map and instead let the upsampling path reuse the higher-resolution feature maps computed earlier in the network, before they were shrunk away. This pattern is usually called an encoder-decoder architecture: the encoder is the familiar downsampling path (like a classifier backbone), and the decoder is a mirrored upsampling path that gradually restores resolution. Skip connections carry feature maps directly from a given resolution in the encoder across to the matching resolution in the decoder, usually by concatenating them, so the decoder can combine "what" information from deep, coarse layers with "where" information from shallow, fine layers.

The best known example of this shape is U-Net, introduced by Ronneberger, Fischer and Brox around 2015 for biomedical image segmentation, where training data is scarce and the exact boundary of a cell or a tumor matters a great deal. Drawn on paper, the architecture looks like the letter U: a contracting path goes down the left side, an expanding path goes up the right side, and horizontal arrows connect matching levels across the two sides. Although it was designed for medical images, the same encoder-decoder-with-skip-connections pattern turned out to be useful far beyond that original setting and shows up, in one variant or another, in a large share of modern segmentation models.

It is worth pausing on why the skip connections matter so much. Imagine segmenting a photo of a hand against a plain background. Deep in the encoder, after several rounds of pooling, the feature map knows "there is a hand somewhere around here" but has lost the precise pixel coordinates of each finger, because pooling (as explained in part 2) deliberately summarizes small neighborhoods into single values. A skip connection from an early encoder layer, where the resolution is still high, reintroduces the raw spatial detail of where the finger edges actually are, letting the decoder draw a sharp outline rather than a rough approximation.

Dilated (atrous) convolutions: resolution without shrinking

A different way to attack the same problem is to avoid shrinking the feature map so aggressively in the first place. A dilated convolution, also called an atrous convolution, is an ordinary convolution kernel with gaps inserted between its weights. A standard 3 by 3 kernel looks at nine adjacent pixels; a 3 by 3 kernel with a dilation of 2 looks at nine pixels spread over a 5 by 5 area, skipping every other position. This grows the receptive field (the area of the input that influences one output value, as discussed in part 2) without adding parameters and without reducing the spatial resolution of the feature map the way pooling or a strided convolution would.

This is the central idea behind the DeepLab family of models, developed by Chen and colleagues, which also introduced atrous spatial pyramid pooling, a technique that applies dilated convolutions with several different dilation rates in parallel to capture context at multiple scales (a small dilation sees fine local detail, a large dilation sees broad context) before combining the results. Some versions of DeepLab also applied a conditional random field as a post-processing step to sharpen boundaries further, though this extra step has become less common as other architectural improvements have taken over that job. The practical upshot of the dilated convolution approach is a final feature map that stays at a higher resolution throughout the network, needing less aggressive upsampling and generally producing sharper object boundaries than a naive encoder-decoder without this trick.

How Instance Segmentation Networks Work

Instance segmentation needs two things at once: the "where is each object" step from object detection (part 7, anchors and region proposals) and the "paint its exact shape" step from semantic segmentation (the previous section). Rather than invent something entirely new, the most widely used approach combines the two directly, by taking a two-stage object detector and adding a mask-predicting branch to it.

Mask R-CNN: adding a mask branch to a detector

Recall from part 7 how Faster R-CNN works as a two-stage detector: a region proposal network scans the image (using anchors, a set of candidate box shapes placed at every location) and proposes a few hundred candidate regions that might contain an object; a second stage then looks at each candidate region individually, classifies it into a specific class, and refines its box coordinates.

Mask R-CNN, introduced by He, Gkioxari, Dollar and Girshick around 2017, extends that second stage with a third, parallel branch. For every candidate region, alongside the existing class label output and box refinement output, the network also predicts a small binary mask, for example 28 by 28 pixels, indicating which pixels inside that region belong to the object versus the background. This mask head is itself a small fully convolutional network, run once for every candidate region, and it predicts one mask per class, so the class branch and the mask branch do not have to agree with each other during training; the mask for the predicted class is simply read off after the classification decision is made. Once predicted, the small mask is resized up to the size of the final box to produce the pixel mask in the original image.

So the overall pipeline is: propose regions, classify and refine boxes for each region, and paint a precise mask inside each kept box. Detection gives you the "which objects and roughly where", and the added mask branch gives you the "exactly which pixels".

RoIAlign: why pixel-precise alignment matters for masks

Part 7 may not have dwelt on a subtle detail of two-stage detectors: once a region proposal is made, its coordinates have to be mapped onto the feature map, which typically has a coarser resolution than the original image because of the downsampling in the backbone. The original Faster R-CNN approach, called RoIPool, handles this by rounding the proposal's boundaries to the nearest grid cell of the feature map. For a task like classifying "is this a cat" or nudging a box by a few pixels, that rounding error is harmless. For predicting a pixel-accurate mask, it is not: a misalignment of even one or two pixels at the feature map resolution can translate into a visibly wrong mask boundary once scaled back up to the full image.

Mask R-CNN fixes this with RoIAlign, which avoids rounding altogether. Instead of snapping a boundary that falls at, say, feature-map coordinate 3.7 down to 4, RoIAlign samples the feature values at the exact coordinate 3.7 using bilinear interpolation, blending the values of the neighboring grid cells 3 and 4 in proportion to the distance. This keeps the spatial correspondence between the mask prediction and the original image exact, and it was one of the key reasons Mask R-CNN produced noticeably sharper, more accurate masks than earlier attempts at the same idea.

Other approaches, briefly

Two-stage, Mask-R-CNN-style models are not the only way to do instance segmentation. Various one-stage approaches predict masks more directly, without a separate region-proposal stage, generally trading a little accuracy for speed. There are also instance segmentation heads built on top of the transformer-based detectors mentioned briefly at the end of part 7; vision transformers themselves are the subject of part 9, so this article will leave the details there and simply note that the mask-prediction idea (a branch that paints pixels for each detected object) carries over regardless of what kind of backbone or detector it is attached to.

Loss Functions for Segmentation

Pixel-wise cross-entropy

The most direct way to train a semantic segmentation network is to treat every pixel as its own tiny classification problem and use the same cross-entropy loss used for whole-image classification, just applied once per pixel and then averaged. If a pixel's true class is "road" and the network outputs a probability distribution over classes for that pixel, say 0.7 for road, 0.2 for sidewalk and 0.1 for everything else, the loss contribution for that one pixel is minus the logarithm of 0.7, about 0.36. Do this for every pixel in the image, average the results, and that is the pixel-wise cross-entropy loss for the whole image. It is simple, well understood, and a reasonable default starting point.

Class imbalance and weighted loss

Plain pixel-wise cross-entropy has a well-known weakness: in many real images, some classes cover far more pixels than others. In a typical street scene, sky and road pixels vastly outnumber pedestrian or traffic-sign pixels, sometimes by a factor of a hundred or more. A model can achieve a low average loss by getting the common classes right and more or less ignoring the rare ones, the same class-imbalance trap that shows up in plain image classification, just at the pixel level instead of the image level.

A common remedy is to weight the loss per class, giving rare classes a larger multiplier so that mistakes on them are penalized more heavily, often using weights inversely proportional to how frequently each class appears in the training data. Another remedy is to move away from cross-entropy altogether for the imbalanced case, which is where Dice loss comes in.

Dice loss and IoU loss

Recall the Dice coefficient from the worked example earlier: two times the intersection divided by the sum of the sizes of the two masks. Dice loss is simply defined as 1 minus the Dice coefficient, computed directly from the predicted (soft, probability-valued) mask and the true mask. Because Dice measures overlap as a ratio rather than counting pixels one by one, it is naturally less dominated by the size of the background class; a tiny tumor that covers 1 percent of an image still contributes meaningfully to the Dice score, whereas it would barely move a plain pixel-accuracy number. IoU loss works the same way, built directly from the IoU formula instead of Dice. In practice, many segmentation training setups use a combination, for example cross-entropy plus Dice loss added together, to get the stable gradients of cross-entropy together with the imbalance-robustness of Dice.

Evaluation Metrics

Pixel accuracy, and why it is misleading on its own

Pixel accuracy is the simplest possible metric: the fraction of pixels in the image where the predicted label matches the true label. It is tempting because it is easy to compute and easy to explain, but it inherits exactly the class-imbalance problem discussed above. If 95 percent of an image is road and sky, a model that predicts "road" everywhere and never once detects a pedestrian can still score above 90 percent pixel accuracy, despite being useless for the task that actually matters, spotting pedestrians. Pixel accuracy is worth reporting as a sanity check, but it should never be the only number used to judge a segmentation model.

Mean Intersection over Union (mIoU)

The standard fix is mean intersection over union, almost always abbreviated mIoU. Compute the IoU separately for each class, exactly as in the worked example earlier in this article (intersection of predicted and true pixels for that class, divided by their union), and then average the per-class IoU values across all classes, giving every class equal weight regardless of how many pixels it covers. A model that nails road and sky but completely misses pedestrians will have a very low IoU on the pedestrian class, which pulls the mean down and exposes the weakness that pixel accuracy hides. mIoU is the metric most commonly reported in semantic segmentation papers and benchmark leaderboards for exactly this reason.

Dice coefficient as a metric

Dice shows up again here, this time as an evaluation metric rather than a loss. It is especially common in medical imaging, where Dice scores between a model's predicted segmentation and an expert's manual annotation are a standard way to report how well an automated tool matches human judgment. Since Dice and IoU are related by the formula shown earlier (Dice equals 2 times IoU divided by 1 plus IoU), you can convert between the two if a paper reports one and you are more used to thinking in the other; just remember Dice numbers always read a little higher than IoU numbers for the same actual overlap, so do not compare a Dice score directly against an IoU score without converting first.

Average precision for instance segmentation (mask AP)

Instance segmentation is scored with the same average precision machinery introduced for object detection in part 7, just swapping the box IoU for a mask IoU. A predicted instance mask is counted as a correct match to a ground-truth instance if their mask IoU exceeds a chosen threshold, commonly 0.5, after which precision and recall are computed the same way they were for boxes, and the area under the resulting precision-recall curve gives the average precision for that class. Results are usually reported as mask AP to distinguish them clearly from the box AP of a plain detector, and many benchmarks report both box AP and mask AP for the same model, since a model can be good at locating objects roughly (high box AP) while still struggling to trace their exact outline (lower mask AP), or vice versa.

Common Mistakes and Pitfalls

• Reporting only pixel accuracy. As shown above, a model can score very high on pixel accuracy while being practically useless on rare but important classes. Always report at least mIoU alongside it, and look at per-class numbers, not just the overall average.
• Ignoring class imbalance during training. Without weighting or a Dice-style loss, a model trained with plain pixel-wise cross-entropy can end up effectively ignoring small, rare classes because they contribute so little to the total loss.
• Upsampling naively from a low-resolution output straight back to full image size. A single bilinear resize of a 7 by 7 class-score map to 224 by 224 throws away detail that cannot be recovered; prefer architectures with skip connections (U-Net style) or dilated convolutions (DeepLab style) that preserve spatial detail through the network.
• Using semantic segmentation for a task that actually needs instance-level counting. If the real question is "how many cells are in this image" or "are these two overlapping people one person or two", a semantic segmentation model will happily merge touching objects of the same class into a single blob, because it has no concept of separate instances; instance segmentation is the right tool there.
• Underestimating annotation cost and label noise. Pixel-level masks are far more expensive and time-consuming to create than bounding boxes or image-level class labels, and annotators often disagree near object boundaries, especially on fuzzy edges like hair or fur. Treat reported metrics with that uncertainty in mind rather than expecting a single ground truth to be perfectly objective.
• Training and testing at inconsistent resolutions or with inconsistent resizing and padding. A mismatch between how training images and evaluation images are resized can silently shift masks by a few pixels, which is invisible on the overall accuracy number but shows up as consistently misaligned masks.
• Expecting perfect separation of heavily overlapping instances. Even a well-trained Mask R-CNN style model will sometimes merge or confuse two objects of the same class that occlude each other strongly, since the mask branch still has to guess at pixels it cannot clearly see; this is a genuinely hard case, not just a sign of a badly trained model.

Where This Shows Up in Practice

Medical imaging is one of the most natural homes for segmentation. Outlining a tumor on an MRI scan, measuring the volume of an organ, or counting individual cells under a microscope are all pixel-level problems where a bounding box would not give a clinician the precision they need. Semantic segmentation handles organ outlines well; instance segmentation is needed when individual cells or lesions must be counted separately.

Autonomous driving relies heavily on semantic segmentation to identify the drivable road surface, lane markings, sidewalks and vegetation, and on instance segmentation (or detection combined with tracking) to follow individual pedestrians, cyclists and vehicles frame to frame, since knowing there are three distinct pedestrians rather than one large pedestrian-shaped region matters a great deal for predicting what each of them might do next.

Satellite and aerial imagery analysis uses semantic segmentation for land cover mapping, labeling every pixel as forest, water, urban or farmland, and instance segmentation for tasks like counting individual building footprints or tracking individual ships in a harbor.

Consumer photo and video tools use segmentation for background removal and portrait-mode blur, where a precise mask around a person separates foreground from background so that only the background can be blurred or replaced.

Agriculture and robotics applications use instance segmentation to count and locate individual plants, fruits or defective items on a production line, and to compute precise outlines of a target object so that a robotic arm knows where to grasp it.

Summary and What's Next

Segmentation answers a more detailed question than classification or detection: not just "what is in this image" or "roughly where is it", but "exactly which pixels belong to what". Semantic segmentation labels every pixel with a class and has no concept of separate objects of the same class. Instance segmentation finds each individual object and produces a precise mask for it, effectively combining detection with pixel-level painting. Panoptic segmentation combines both, giving every pixel either a stuff-class label or a thing-class-plus-instance label.

On the architecture side, semantic segmentation is typically built from fully convolutional networks, often arranged as an encoder-decoder with skip connections in the style of U-Net, or built around dilated convolutions in the style of DeepLab, both aimed at preserving fine spatial detail that plain downsampling would otherwise destroy. Instance segmentation is typically built by extending a two-stage detector like Faster R-CNN (part 7) with an extra mask-prediction branch, as in Mask R-CNN, with RoIAlign used to keep the mask pixel-accurate.

On the evaluation side, pixel accuracy is a weak metric on its own because of class imbalance; mean IoU is the standard metric for semantic segmentation; Dice is common in medical contexts and is related to IoU by a simple formula; and mask AP extends the average precision idea from detection to the instance segmentation case by matching masks instead of boxes.

Part 9 turns to vision transformers, an alternative to the convolutional backbones used throughout this path so far, and looks at how the attention mechanism that transformed natural language processing has been adapted to images, including its growing role as a backbone for detection and segmentation models like the ones described in this article.