Read it, then run it
This page is the read-only chapter intro. To actually execute the cells (load the dataset, train a model, render GradCAM heatmaps on your own images), open the Colab notebook in a free T4 runtime. The curriculum is designed to be executed, not read.
Chapter 3: CNN Architectures & Transfer Learning¶
Learning objectives: - Understand how convolutional neural networks (CNNs) extract features from images - Learn what transfer learning is and why it's essential for medical imaging - Initialize and compare three pretrained CNN architectures
Clinical context: Just as different imaging modalities (CT, MRI, X-ray) capture different types of information, different CNN architectures process images in different ways. Choosing the right architecture is like choosing the right imaging protocol for a clinical question.
How CNNs "Read" Images¶
A Convolutional Neural Network processes images through a hierarchy of layers, much like how a radiologist reads an image at multiple levels:
1. Convolutional layers - A small filter (typically 3×3 pixels) slides across the image, computing a "response" at each position. This produces feature maps that highlight specific patterns: - Early layers detect simple features: edges, corners, brightness gradients - Middle layers combine edges into textures and shapes - Deep layers recognize complex structures: organs, tumors, anatomical landmarks
2. Pooling layers - Reduce the spatial dimensions (e.g., 64×64 → 32×32) while keeping the most important information. This is like stepping back from a microscope slide to see the overall pattern.
3. Fully connected layers - Take the extracted features and make a classification decision, analogous to the radiologist's final interpretation.
The key insight: The CNN learns these features automatically from data - nobody programs it to look for edges or organ boundaries. The training process discovers which patterns best predict the correct diagnosis.
Transfer Learning: Standing on the Shoulders of Giants¶
Training a deep neural network from scratch requires millions of images and days of GPU time. Most clinical datasets have only hundreds or thousands of labeled images - far too few to train a good model from zero.
Transfer learning solves this by starting with a model that has already learned useful features on a large dataset (ImageNet: 14 million natural images, 1000 categories). Even though these are photos of dogs, cars, and furniture - not X-rays - the early and middle layers learn universal visual features (edges, textures, shapes) that transfer remarkably well to medical images.
Clinical analogy: A radiologist trained in general radiology can specialize in neuroradiology much faster than someone who has never looked at an image. The foundational visual skills (anatomy recognition, density assessment, pattern matching) transfer across subspecialties. Transfer learning works the same way.
The evolution of transfer learning:¶
| Era | Approach | Data Needed |
|---|---|---|
| 2015–2019 | Train from scratch | 100,000+ images |
| 2019–2022 | ImageNet pretrained → fine-tune | 1,000–10,000 images |
| 2022–present | Medical foundation models → fine-tune | 100–1,000 images |
We'll explore foundation models in Chapter 8. For now, ImageNet pretraining is our starting point.
Practical Guidance: When to Use Transfer Learning¶
Rule of thumb for clinical datasets:
| Dataset Size | Recommended Approach |
|---|---|
| < 1,000 images | Pretrained model, freeze most layers, train only the classifier head |
| 1,000–10,000 images | Pretrained model, fine-tune all layers with a low learning rate |
| 10,000–100,000 images | Pretrained or train from scratch - experiment with both |
| > 100,000 images | Training from scratch becomes viable |
For most clinical research projects, you'll have under 10,000 labeled images - making pretrained models your default starting point. The three architectures we initialized above (DenseNet121, ResNet50, EfficientNet-B0) all start with ImageNet-learned features.
Key takeaway: Always start with
pretrained=True. Training from scratch on small clinical datasets almost always leads to overfitting and poor generalization.