Read it, then run it
This page is the read-only chapter intro. To actually execute the cells (load the dataset, train a model, render GradCAM heatmaps on your own images), open the Colab notebook in a free T4 runtime. The curriculum is designed to be executed, not read.
Chapter 7: Explainability with GradCAM¶
Learning objectives: - Generate visual explanations of model predictions using GradCAM - Interpret heatmaps for correctly and incorrectly classified images - Understand the limitations of explainability methods - Apply GradCAM to both CNN and ViT architectures
Clinical context: When you receive an AI-assisted diagnosis, your first question isn't "What is the AUC?" - it's "Show me where you see the finding." GradCAM answers this by highlighting the image regions that most influenced the model's prediction.
Why Explainability Matters¶
Imagine two scenarios:
Scenario A: An AI system says "This chest X-ray is 92% likely to show pneumonia." You have no way to see why.
Scenario B: The same AI says "92% pneumonia" and overlays a heatmap showing it focused on the right lower lobe consolidation - exactly where you see the opacity.
Which would you trust more? Which would you feel comfortable acting on?
Explainability tools like GradCAM don't replace quantitative metrics - they complement them. A model with high AUC and clinically sensible heatmaps is far more trustworthy than one with high AUC but heatmaps that highlight irrelevant regions (like the image border or patient labels burned into the scan).
How GradCAM Works (5 Steps, No Math)¶
GradCAM (Gradient-weighted Class Activation Mapping) produces a heatmap by asking: "Which parts of the last feature maps were most important for predicting this class?"
| Step | What happens | Plain language |
|---|---|---|
| 1 | Forward pass | Run the image through the model to get a prediction |
| 2 | Compute gradients | Ask: "How much does the predicted class score change if I tweak each feature map?" |
| 3 | Weight feature maps | Important feature maps get high weights; irrelevant ones get low weights |
| 4 | Combine into heatmap | Sum the weighted feature maps into a single 2D map |
| 5 | Overlay on image | Resize the heatmap to match the input image and overlay it |
The result: a heatmap where warm colors (red/yellow) = high importance, cool colors (blue) = low importance.
Interpreting Correct Predictions¶
Look at each pair of images above and ask yourself:
- Is the model focusing on the clinically relevant region?
- For CXR: Is it looking at the lungs and mediastinum?
- For HeadCT: Is it focused on the brain parenchyma?
-
For Hand: Is it looking at the bones, not the background?
-
Correct prediction + correct region = higher confidence that the model has learned meaningful features, not spurious shortcuts (like text labels or scanner artifacts).
-
Correct prediction + wrong region = red flag. The model may be relying on confounders - for example, all CXR images might come from the same scanner with a distinctive border, and the model might be recognizing the border, not the anatomy.
Diagnosing Model Failures¶
For each misclassified image, ask:
-
Is the model looking at the wrong region? → It may be distracted by artifacts, borders, or irrelevant structures. This suggests the training data has confounders.
-
Is the model looking at the right region but misinterpreting? → The features in that region may genuinely look similar across classes (e.g., some AbdomenCT and ChestCT slices can look alike at the boundary).
-
Would a human also struggle with this image? → Some images are ambiguous even for experts. These "borderline" cases are expected errors, not model failures.
Key insight: GradCAM on misclassifications is more informative than GradCAM on correct predictions. It reveals systematic failure modes that metrics alone cannot capture.
Limitations of Explainability Methods¶
GradCAM and similar tools are powerful but imperfect. Before using them in clinical decision-making, understand these limitations:
1. Post-hoc, not causal¶
GradCAM shows correlation, not causation. The highlighted region is where the model's prediction was most sensitive - but this doesn't prove the model "understood" the anatomy or pathology in that region.
2. Different methods disagree¶
GradCAM, Integrated Gradients, SHAP, and saliency maps can produce different heatmaps for the same image and model. There is no ground truth for "the correct explanation."
3. Convincing but wrong¶
A model can produce a beautiful, clinically plausible heatmap while being right for wrong reasons. For example, a model might focus on chest tubes (which correlate with ICU patients, who correlate with certain diagnoses) rather than the actual pathology.
4. Not a substitute for validation¶
Explainability is one piece of evidence alongside quantitative metrics, external validation, and clinical judgment. Never deploy a model based on heatmaps alone.
In clinical practice: Treat GradCAM as hypothesis-generating, not definitive proof. It helps you ask better questions about the model's behavior - which is exactly what a thoughtful radiologist does with any new diagnostic tool.