Read it, then run it
This page is the read-only chapter intro. To actually execute the cells (load the dataset, train a model, render GradCAM heatmaps on your own images), open the Colab notebook in a free T4 runtime. The curriculum is designed to be executed, not read.
Chapter 8: Foundation Models & Fine-Tuning¶
Learning objectives: - Understand the paradigm shift from training-from-scratch to foundation model fine-tuning - Learn the freeze → linear probe → selective unfreeze → fine-tune workflow - Fine-tune a foundation model on MedNIST and compare to CNNs and ViTs
Clinical context: Foundation models are like hiring a radiologist who has already read millions of cases across dozens of institutions. Instead of training from scratch, you only need to teach them your institution's specific protocols and case mix. This dramatically reduces the data and compute you need.
The Foundation Model Revolution¶
The history of deep learning in medical imaging follows three eras:
| Era | Approach | Data Required | Example |
|---|---|---|---|
| 2015–2019 | Train CNNs from scratch | 100,000+ labeled images | CheXNet (2017) |
| 2019–2022 | ImageNet pretrained → fine-tune | 1,000–10,000 labeled images | Chapters 3 & 5 of this tutorial |
| 2022–present | Foundation model → fine-tune | 100–1,000 labeled images | This chapter |
What makes foundation models different?¶
Foundation models are pretrained on millions of images using self-supervised learning - they learn visual representations without human labels. Key examples in medical imaging:
- DINOv2 (Meta): Self-supervised ViT trained on 142 million diverse images
- BiomedCLIP (Microsoft): Vision-language model trained on 15 million biomedical image-text pairs
- RAD-DINO (Microsoft): DINOv2 fine-tuned specifically on radiology images
- CheXzero (Stanford): CLIP-style model for chest X-ray interpretation
Clinical analogy: Hiring a radiologist who has already read millions of cases and just needs to learn your institution's specific protocols - versus training a medical student from year one.
The Fine-Tuning Workflow¶
We'll follow the standard foundation model workflow:
- Load a pretrained foundation model (DINOv2-Small via timm)
- Freeze all backbone parameters - only train a new classification head (linear probing)
- Evaluate - see how well frozen features work
- Selectively unfreeze the last 2 transformer blocks
- Fine-tune with a lower learning rate
- Compare to all previous models
Why DINOv2? It downloads reliably on Colab without authentication tokens, is small enough for a T4 GPU, and uses a ViT backbone. While not medical-specific, the same fine-tuning workflow applies identically to medical foundation models like RAD-DINO or BiomedCLIP when you have access to them.
If Your Colab Session Timed Out: Load From HF Hub¶
The fine-tune above takes ~5-10 minutes on a T4. If your session disconnected, you can pick up exactly where the cells above leave off - the same recipe is published as a checkpoint at t22000t/dinov2-small-mednist. The model card includes the held-out validation indices (val_indices.json, sha256-pinned), so the val AUC you see here matches the published one on the same examples.
Run the cell below to skip the linear-probe + fine-tune steps and load the pre-trained weights directly into the foundation_model you constructed earlier in this chapter.
Multi-Architecture Comparison: Where Each Model Looks¶
Different architectures attend to different image regions, even when they all reach the same prediction. This cell runs the same 4 misclassified validation images through three models and renders their explainability outputs side-by-side:
| Model | Explainability method |
|---|---|
| DenseNet121 (Ch 3) | GradCAM on model.features.denseblock4 |
| ViT-Small (Ch 4) | Attention rollout from [CLS] to patch tokens |
| DINOv2-Small (Ch 8) | Attention rollout from [CLS] to patch tokens |
This is the visual payoff for Chapters 4 + 7 + 8 combined. Look at where each architecture disagrees with the others on the same input - that disagreement is the most pedagogically useful signal.
GPU required. This cell is tagged
gpu-required; the nightly CI smoke test skips it.
Key Takeaway: The Foundation Model Workflow¶
The freeze → linear probe → selective unfreeze → fine-tune workflow is what you'll most likely use in real clinical AI research:
- Start frozen - test if the pretrained features already work for your task (linear probing)
- If not good enough - unfreeze the last few layers and fine-tune with a low learning rate
- If still not enough - unfreeze more layers, but risk overfitting on small datasets
This workflow: - Requires the least labeled data (good for clinical datasets) - Trains faster (fewer trainable parameters) - Often achieves the best results (leverages massive pretraining)
For your own research: When medical foundation models (RAD-DINO, BiomedCLIP) are available for your imaging modality, prefer them over ImageNet-pretrained models. The domain-specific pretraining gives an additional boost - like the difference between a general radiologist and a subspecialist.