What a CNN Looks At: Grad-CAM and Saliency

Why look inside the box at all

Across this path you have built convolutional networks, borrowed pretrained ones, fine-tuned them on new data, and even asked them to locate and segment objects. At every step the network has been a black box that takes pixels in and produces numbers out: a class score, a bounding box, a mask. The accuracy numbers can look excellent while the network has quietly learned to rely on the wrong evidence. This article is about a small toolkit, saliency maps and Grad-CAM, that lets you ask a trained CNN a very direct question: which pixels in this image made you say that.

This matters for three practical reasons. First, debugging: if a classifier gets an image wrong, a heatmap of its attention often tells you why in seconds, faster than staring at the architecture. Second, trust: before deploying a model in a setting where mistakes are costly, you want evidence that it is looking at the object and not at a watermark or a background artefact. Third, bias detection: models trained on imperfect datasets can pick up shortcuts that happen to correlate with the label during training but have nothing to do with the concept you actually care about.

The problem: high accuracy, wrong reasons

A frequently cited illustrative case comes from the interpretability literature, specifically the paper that introduced the LIME method (Ribeiro, Singh and Guestrin, 2016). The authors trained a classifier to distinguish photos of huskies from photos of wolves, and it performed well on their test images. When they inspected which pixels the model relied on, it turned out that many wolf photos in the training data happened to have snow in the background, while husky photos tended not to. The model had learned a snow detector, not a wolf detector. It would confidently call a husky photographed in snow a wolf, and a wolf photographed on grass a husky. The accuracy number alone never revealed this. Looking at the evidence did.

This is the general shape of the risk: a CNN optimises for the loss function you give it, and the loss function only knows about labels, not about what you intended the labels to mean. If a shortcut in the pixels is correlated with the label in your training set, the network is free to use it. Saliency maps and Grad-CAM are two of the simplest, most widely used tools for surfacing that kind of shortcut.

Vanilla gradient saliency maps

The oldest and simplest idea, introduced by Simonyan, Vedaldi and Zisserman in 2013, asks a question you already know how to compute: how does the predicted score change if I nudge this pixel. You have used gradients throughout this path to train networks, computing the gradient of a loss with respect to the network's weights and then updating the weights. Saliency maps reuse exactly the same backpropagation machinery, but point it at the input image instead of at the weights, and leave the weights untouched.

Concretely: run an image through a trained, frozen network. Take the score the network assigns to the predicted class, before or after the final softmax (before is usually preferred, since softmax scores can be flattened by competition from other classes). Backpropagate from that single score all the way to the input pixels. The result is a gradient tensor the same shape as the image, one number per pixel per colour channel, telling you the local sensitivity of the class score to that pixel. A large positive or negative gradient at a pixel means that a small change there would noticeably move the score; a value near zero means the network barely notices that pixel. Taking the absolute value and collapsing the three colour channels (typically by taking the maximum across channels) gives a single greyscale heatmap the size of the image, which you can overlay on the original photo.

The code below builds a synthetic image rather than downloading one, so it has no external data dependency beyond the pretrained weights, which torchvision downloads automatically the first time you run it (this requires internet access on that first run only; afterwards the weights are cached locally). It assumes torch 2.1, torchvision 0.16 and Pillow 10, though the logic has not changed across recent versions of these libraries.

# Tested with: torch==2.1.0, torchvision==0.16.0, Pillow==10.0.0, numpy==1.26.0
import torch
import torchvision.models as models
import torchvision.transforms as T
from PIL import Image, ImageDraw

# Build a simple synthetic image instead of downloading one:
# a red circle on a green background, 224x224 pixels.
img = Image.new("RGB", (224, 224), color=(34, 139, 34))
draw = ImageDraw.Draw(img)
draw.ellipse((60, 60, 160, 160), fill=(200, 30, 30))

# Preprocess the way ImageNet-pretrained models expect: resize,
# convert to a tensor in [0, 1], then normalise with ImageNet statistics.
preprocess = T.Compose([
    T.Resize((224, 224)),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(img).unsqueeze(0)  # shape: (1, 3, 224, 224)
input_tensor.requires_grad_()  # we need gradients with respect to the pixels

model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
model.eval()

output = model(input_tensor)            # shape: (1, 1000), raw class scores
predicted_class = output.argmax(dim=1).item()
score = output[0, predicted_class]

model.zero_grad()
score.backward()

saliency = input_tensor.grad.data.abs()      # (1, 3, 224, 224)
saliency, _ = torch.max(saliency, dim=1)     # (1, 224, 224): max over colour channels
saliency = saliency.squeeze().numpy()

print("Predicted ImageNet class index:", predicted_class)
print("Saliency map shape:", saliency.shape)
print("Saliency map value range:", saliency.min(), saliency.max())

Running this prints a predicted class index between 0 and 999 (the synthetic circle will not match any real ImageNet class convincingly, but the network will still produce its most confident guess), a saliency map shaped (224, 224), and a range of values typically spanning from 0 up to some small positive number, since gradients from a single backward pass are usually modest in magnitude. To turn this into something visual, you would normalise the array to [0, 1] and overlay it on the original image with partial transparency; in this toy example you should see the highest values concentrated around the edge of the red circle, since that boundary is where small pixel changes most affect how the network reads the shape.

Why vanilla saliency maps are noisy

If you try this on a real photograph you will usually find the result looks speckled, with bright and dark pixels scattered everywhere rather than cleanly outlining the object. There are two reasons for this. First, the ReLU activations used throughout most CNNs (see Part 3 for how these layers are assembled) create a piecewise-linear function with many kinks, and the gradient can change abruptly from one pixel to its neighbour even though the two pixels look almost identical to a human. Second, gradients measure sensitivity at a single point in a very high-dimensional input space; a network can be sensitive to a small, almost imperceptible shift in a pixel's value for reasons that have little to do with the pixel being part of the object of interest.

A simple fix, introduced by Smilkov and colleagues in 2017 under the name SmoothGrad, is to add a little random noise to the image, compute the saliency map, repeat this many times with different noise, and average the results. Averaging cancels out the noisy, idiosyncratic spikes while reinforcing the signal that is consistently present across many slightly perturbed copies of the image. The trade-off is cost: instead of one forward and backward pass, you need dozens, so SmoothGrad is slower but produces visibly cleaner maps. You can implement it by wrapping the saliency computation above in a loop that adds torch.randn_like(input_tensor) * sigma to the pixels before each pass, and averaging the resulting saliency arrays.

Grad-CAM: weighting feature maps by how much they matter

Vanilla saliency operates at the level of individual input pixels, which is exactly why it is noisy: a single pixel rarely carries enough context on its own to tell you about an object. Grad-CAM, introduced by Selvaraju and colleagues in 2017, works at a coarser and usually more informative level: the feature maps produced by one of the last convolutional layers, right before the network flattens everything down to a vector of class scores.

Recall from Part 3 and Part 4 that a convolutional network is a stack of layers where early layers detect simple patterns like edges and colour blobs over small receptive fields, and later layers combine these into increasingly abstract and spatially coarse representations, eventually covering large portions of the image. By the time you reach the last convolutional block of something like ResNet-50 (Part 4), the feature map might be only 7 by 7 spatial locations across, but each location has accumulated information from a large patch of the original 224 by 224 image, and each of the (for ResNet-50) 2048 channels at that stage has specialised, through training, to respond to some pattern useful for classification.

Grad-CAM's idea is: some of those 2048 channels are much more relevant to the predicted class than others. If we can work out how relevant each channel is, we can combine the channels in a weighted sum, keeping the ones that matter and discarding the ones that do not, and the result will be a 7 by 7 map showing where, spatially, the evidence for the predicted class is concentrated. The precise way Grad-CAM measures the relevance of a channel is itself a gradient: it backpropagates from the class score to the feature maps of the chosen layer (rather than all the way to the input pixels, which is what vanilla saliency does), and averages that gradient over the spatial positions of each channel. A channel whose average gradient is large and positive is one where increasing its activations would increase the class score, so it counts as evidence for that class; a channel with a negative average gradient works against the class.

Putting this together, Grad-CAM for a chosen convolutional layer and a chosen class works in four steps. First, run the image through the network and read off the score for the class you are interested in (usually the predicted class, but it can be any class, which is what makes Grad-CAM class-discriminative: you can ask it to show you the evidence for "cat" and separately the evidence for "dog" on the same image, even if the network predicted "cat"). Second, backpropagate from that score to the chosen layer's feature maps and average each channel's gradient over space to get one weight per channel. Third, form a weighted sum of the feature maps using those weights. Fourth, apply a ReLU to the result, keeping only the positive evidence for the class and discarding the negative evidence, and then normalise and upsample the resulting coarse map to the size of the original image so it can be overlaid as a heatmap.

It is worth noting that Grad-CAM is a generalisation of an earlier method called CAM (Class Activation Mapping, Zhou and colleagues, 2016), which produced similar heatmaps but only worked for networks built with a global average pooling layer directly feeding into the final classification layer, and required retraining the network if it did not already have that structure. Grad-CAM's use of gradients instead of the raw weights of the final layer means it works on essentially any CNN, including ones that already have several fully connected layers after the convolutions, with no retraining and no architecture changes.

A worked numeric example

To make the weighting and combining steps completely concrete, here is Grad-CAM worked by hand on a tiny toy case. Instead of 2048 channels of 7 by 7 feature maps, imagine the last convolutional layer has just 2 channels, each 3 by 3, call them channel A and channel B. Suppose the activations (the actual feature map values from the forward pass) are:

Channel A: 1 2 1 0 1 0 1 0 1 Channel B: 0 0 1 1 1 1 0 1 0

Now suppose backpropagation from the class score gives an average gradient of 0.5 for channel A and minus 0.2 for channel B. In Grad-CAM, this average gradient is exactly the weight for that channel: weight_A = 0.5, weight_B = -0.2. The positive weight for A means channel A's pattern supports the predicted class; the negative weight for B means channel B's pattern, wherever it fires, actually argues against the class, perhaps because it detects something like a background texture common to other classes.

The combined map before the ReLU is weight_A times channel A plus weight_B times channel B, computed position by position:

• Top-left (row 0, col 0): 0.5 x 1 + (-0.2) x 0 = 0.5
• Top-middle (row 0, col 1): 0.5 x 2 + (-0.2) x 0 = 1.0
• Top-right (row 0, col 2): 0.5 x 1 + (-0.2) x 1 = 0.3
• Middle-left (row 1, col 0): 0.5 x 0 + (-0.2) x 1 = -0.2
• Centre (row 1, col 1): 0.5 x 1 + (-0.2) x 1 = 0.3
• Middle-right (row 1, col 2): 0.5 x 0 + (-0.2) x 1 = -0.2
• Bottom-left (row 2, col 0): 0.5 x 1 + (-0.2) x 0 = 0.5
• Bottom-middle (row 2, col 1): 0.5 x 0 + (-0.2) x 1 = -0.2
• Bottom-right (row 2, col 2): 0.5 x 1 + (-0.2) x 0 = 0.5

Applying ReLU zeroes out every negative entry, leaving: 0.5 1.0 0.3 0 0.3 0 0.5 0 0.5 Dividing by the maximum value, 1.0, gives the final normalised Grad-CAM map, with the top-middle position standing out as the single strongest piece of evidence, the corners next, and the three locations that had a negative combined value contributing nothing. In a real network this 3 by 3 (or 7 by 7, or whatever the layer's spatial size is) map would then be upsampled with interpolation to the original image resolution and overlaid as a heatmap, so that the strongest region appears as a bright patch over the corresponding part of the photograph.

Implementing Grad-CAM with hooks

The practical difficulty in implementing Grad-CAM is capturing both the forward activations and the backward gradients of a specific internal layer, since by default PyTorch only keeps gradients for leaf tensors like model parameters and inputs that explicitly request them, not for intermediate activations. The standard solution is to register hooks: small callback functions that PyTorch calls automatically during the forward and backward pass, letting you save whatever you need. The code below extends the earlier example, attaching hooks to the last block of ResNet-50's final convolutional stage, layer4.

# Tested with: torch==2.1.0, torchvision==0.16.0, Pillow==10.0.0, numpy==1.26.0
import torch
import torch.nn.functional as F
import torchvision.models as models
import torchvision.transforms as T
from PIL import Image, ImageDraw

img = Image.new("RGB", (224, 224), color=(34, 139, 34))
draw = ImageDraw.Draw(img)
draw.ellipse((60, 60, 160, 160), fill=(200, 30, 30))

preprocess = T.Compose([
    T.Resize((224, 224)),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(img).unsqueeze(0)

model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
model.eval()

activations = {}
gradients = {}

def save_activation(module, module_input, module_output):
    activations["value"] = module_output

def save_gradient(module, grad_input, grad_output):
    gradients["value"] = grad_output[0]

# layer4 is the last residual stage of ResNet-50; its output feature
# maps are the ones Grad-CAM combines.
target_layer = model.layer4[-1]
target_layer.register_forward_hook(save_activation)
target_layer.register_full_backward_hook(save_gradient)

output = model(input_tensor)
predicted_class = output.argmax(dim=1).item()
score = output[0, predicted_class]

model.zero_grad()
score.backward()

act = activations["value"][0]           # shape: (2048, 7, 7) for ResNet-50
grad = gradients["value"][0]            # shape: (2048, 7, 7)

weights = grad.mean(dim=(1, 2))         # one weight per channel, shape: (2048,)
cam = torch.zeros(act.shape[1:], dtype=torch.float32)
for c, w in enumerate(weights):
    cam += w * act[c]

cam = F.relu(cam)
cam = cam / (cam.max() + 1e-8)           # normalise to [0, 1]

# Upsample the coarse 7x7 map to the original 224x224 image size.
cam_tensor = cam.unsqueeze(0).unsqueeze(0)      # shape: (1, 1, 7, 7)
cam_upsampled = F.interpolate(cam_tensor, size=(224, 224),
                               mode="bilinear", align_corners=False)
cam_upsampled = cam_upsampled.squeeze().detach().numpy()

print("Predicted class index:", predicted_class)
print("Raw Grad-CAM shape:", cam.shape)
print("Upsampled Grad-CAM shape:", cam_upsampled.shape)
print("Upsampled Grad-CAM range:", cam_upsampled.min(), cam_upsampled.max())

Running this prints a predicted class index, the raw 7 by 7 shape of the Grad-CAM map before upsampling, the upsampled 224 by 224 shape, and a value range from 0 to 1 after normalisation. To actually see the result you would convert cam_upsampled to an 8-bit colour map (for instance with matplotlib's colormap functions) and blend it with the original image at around 40 percent opacity. On the synthetic circle image, you should see the warm colours of the heatmap concentrated over the red ellipse and its immediate surroundings, since that is the region the last convolutional layer's most class-relevant channels responded to, rather than scattered across the uniform green background.

A detail worth remembering: register_full_backward_hook is the modern, correct hook for capturing gradients flowing into a module in current PyTorch versions; an older function called register_backward_hook exists for backward compatibility but has documented inconsistencies with modules that have multiple inputs or outputs, so prefer the full version. If you are using a very different PyTorch release, check the current documentation for the exact hook API, since this is one of the areas that has changed between versions.

Saliency maps versus Grad-CAM: which one to reach for

The two methods sit at different points on a resolution versus noise trade-off, and it helps to think of them as complementary rather than competing. Vanilla gradient saliency operates at the resolution of individual pixels, so in principle it can highlight fine detail, like a thin edge or a small distinguishing mark, but because it is computed from a single, local gradient it tends to be visually noisy and can be sensitive to irrelevant high-frequency details in the image. Grad-CAM operates at the resolution of the last convolutional layer's feature maps, typically a small grid like 7 by 7 for a 224 by 224 input, so the resulting heatmap after upsampling is smooth and easy to read, clearly indicating which region of the image mattered, but it cannot tell you about structure finer than that grid; it will happily highlight an entire object-sized patch without distinguishing which exact pixels within that patch were decisive.

In practice, Grad-CAM is usually the better first tool for sanity-checking whether a classifier is looking at the right object, precisely because its coarse, smooth heatmaps are much easier for a human to interpret at a glance: did the hot region land on the dog, or on the lawn next to it. Saliency maps and their smoothed variants are more useful when you need pixel-level detail, for example to check whether a model is relying on a thin watermark or a specific small region like an eye or a logo, something a 7 by 7 grid would blur past.

A few other methods worth knowing by name

The interpretability literature has produced many variations on these two ideas; you do not need to implement all of them, but recognising the names will help you read papers and library documentation.

• Guided backpropagation modifies how gradients flow backward through ReLU layers, keeping only positive gradients at each ReLU, which tends to produce sharper, less noisy pixel-level maps than vanilla saliency, though it answers a slightly different question about the network's behaviour.
• Guided Grad-CAM combines the two ideas directly: it multiplies the smooth, class-discriminative Grad-CAM heatmap (upsampled to pixel resolution) with the sharp, high-resolution guided backpropagation map, aiming to get the best of both: fine detail that is also tied to the correct class.
• Grad-CAM++ (Chattopadhay and colleagues, 2018) refines the weight computation in Grad-CAM to handle images with multiple instances of the same class more gracefully, using a more careful weighting of the gradients rather than a plain spatial average.
• Integrated Gradients (Sundararajan and colleagues, 2017) addresses a known weakness of plain gradients, namely that a network's output can saturate (stop changing) even though a feature is clearly present, which makes the gradient at that point misleadingly close to zero. It instead accumulates gradients along a straight path from a baseline image (often an all-black or all-grey image) to the actual input, giving an attribution that satisfies some useful mathematical properties that plain gradie
• Occlusion sensitivity (Zeiler and Fergus, 2014) takes a conceptually different and much simpler approach: slide a grey or black patch over the image, recording how much the class score drops each time a given region is covered. Regions whose occlusion causes a big drop in score must have been important. This method needs no gradients at all, just repeated forward passes, which makes it slow but very easy to trust, since it is directly measuring the effect of removing information rather than infe

Common mistakes and how to avoid them

Picking the wrong layer for Grad-CAM

Grad-CAM can technically be applied to any convolutional layer, but the quality of the result depends heavily on which one you choose. Early layers have high spatial resolution but their feature maps correspond to simple, low-level patterns like edges and colours, which are not yet specialised for any particular class, so the resulting heatmap tends to look generically edge-like rather than object-specific. Very late layers, right before the final classification layer, are the most class-specific and are the standard choice (layer4 in ResNet-50, for instance), but if a network has an unusual architecture it is worth checking that the chosen layer still has meaningful spatial extent; a layer that has already been pooled down to a 1 by 1 size has nothing left to localise.

Treating a heatmap as proof of causation

A heatmap shows correlation between regions of the input and the gradient of the output, not a guaranteed causal story about why the network made its decision. It is tempting to look at a Grad-CAM map that lands squarely on a dog's face and conclude the network has learned a robust, generalisable notion of "dog", but the heatmap alone cannot rule out the possibility that the network is also using some other cue elsewhere in the image that happens to correlate with the dog's position, or that it would produce a similarly convincing-looking heatmap on a very different, wrong prediction. Treat these visualisations as a source of hypotheses to test further (for example, by occluding the highlighted region and checking that the prediction actually changes), not as a final verdict.

Confirmation bias when reading heatmaps

Because these heatmaps are visual and often land somewhere plausible, it is easy to see what you expect to see. If you already believe the model should be looking at the object, a heatmap that is roughly centred on the image, which is where objects in many photo datasets happen to be, can look like confirmation even if the network is actually responding to something else that also happens to be near the centre. Good practice is to test on a handful of deliberately chosen images where you know the answer in advance, for instance an image where the object has been moved to an unusual position or the background has been deliberately swapped, and check that the heatmap moves with the object rather than staying fixed.

Skipping sanity checks on the method itself

A widely cited paper by Adebayo and colleagues in 2018, called "Sanity Checks for Saliency Maps", tested a range of saliency methods by randomising parts of the model, for instance reinitialising the weights of some layers to random values, or randomising the training labels, and checking whether the resulting heatmaps changed. A saliency method that is actually reflecting something the model learned should produce noticeably different, and generally worse, heatmaps once the model has been scrambled in this way. The paper found that some popular methods produced heatmaps that barely changed even after this kind of randomisation, meaning those particular heatmaps were behaving more like edge detectors applied to the raw image than like genuine explanations of the trained model's behaviour. The practical lesson is not that saliency and Grad-CAM are useless, but that it is worth being specifically cautious with any method, checking its known limitations, and not assuming a visually plausible heatmap is automatically a faithful one. Vanilla gradients and Grad-CAM both behave reasonably under these particular sanity checks in the original study's findings, which is part of why they remain popular default choices, but new variants should still be checked rather than trusted purely because they look good on a few example images.

Forgetting that normalisation hides absolute confidence

Both saliency maps and Grad-CAM are typically normalised to a fixed range like 0 to 1 before display, so that the brightest pixel in the heatmap always appears maximally bright regardless of how confident the underlying prediction actually was. This means a heatmap for a prediction the network made with 99 percent confidence can look visually identical to a heatmap for a prediction made with 30 percent confidence spread almost evenly across several classes. Always check the predicted class and its score alongside the heatmap, rather than reading confidence into the visualisation itself.

Where this fits into real projects

In practice, Grad-CAM and saliency maps earn their keep in a handful of recurring situations. When fine-tuning a pretrained network on your own dataset, as you will do in Part 12, running Grad-CAM over a sample of validation images, especially the ones the model gets wrong, is one of the fastest ways to spot whether the model has picked up a shortcut specific to your data collection process, such as a particular camera's colour cast, a consistent watermark, or a background that correlates with one class because of how the photos happened to be taken. When a stakeholder asks why a deployed model made a particular decision, a heatmap is a far more persuasive and genuinely useful answer than a bare confidence score, though it should be presented honestly as a sensitivity map rather than as an explanation of the model's internal reasoning in any deeper sense. In domains with real consequences, such as medical imaging, these tools are used as one additional check among several, not as the sole basis for trusting a model, precisely because of the sanity check limitations described above; a heatmap that looks reasonable is encouraging but is not equivalent to a clinical validation study.

It is also worth remembering that both methods were designed primarily around CNN architectures, where the idea of a spatial grid of feature map channels maps naturally onto the Grad-CAM weighting scheme. When you reach Part 9's territory of vision transformers in later study, you will find the attention mechanisms there offer a different, complementary way of asking what the model is looking at, since transformers naturally produce attention weights over patches as part of their computation, which is a separate topic from the gradient-based methods covered here.

Summary and what is next

A trained CNN can be probed, not just evaluated by accuracy. Vanilla gradient saliency backpropagates from the predicted class score all the way to the input pixels, producing a fine-grained but often noisy map of pixel sensitivity; SmoothGrad cleans this up by averaging over many noisy copies of the input. Grad-CAM instead backpropagates to the feature maps of a chosen convolutional layer, usually the last one, averages the gradient of each channel to get a weight reflecting how much that channel supports the predicted class, combines the channels with those weights, and keeps only the positive evidence, producing a coarse but smooth and easy-to-read heatmap. Neither method proves causation, both can be read with more confidence than they deserve if you are not careful, and both should be treated as a starting point for further investigation rather than a final explanation. Used well, alongside basic sanity checks like testing whether the heatmap follows the object when you move it, they are an efficient way to catch models that are right for the wrong reasons before those reasons cause a problem in deployment.

The next two parts of this path put these ideas into practice directly. Part 12 walks through fine-tuning a ResNet on your own photos end to end, a natural place to apply Grad-CAM to the model you build. Part 13 moves to training a YOLO detector on a custom dataset, where similar questions about what the model is really keying on become relevant in a bounding-box setting. Part 14 then combines fine-tuning with deployment in a complete plant disease classifier project, where a quick Grad-CAM check before shipping the API is exactly the kind of habit this article has been arguing for.