Read it, then run it
This page is the read-only chapter intro. To actually execute the cells (load the dataset, train a model, render GradCAM heatmaps on your own images), open the Colab notebook in a free T4 runtime. The curriculum is designed to be executed, not read.
Chapter 4: Vision Transformers¶
Learning objectives: - Understand how Vision Transformers (ViTs) work - without any math - Learn how ViTs differ fundamentally from CNNs - Know when each architecture is appropriate for medical imaging
Clinical context: CNNs process images the way you read a report - scanning systematically from start to finish. Vision Transformers work more like an expert radiologist's eye - immediately relating distant findings across the entire image. Both approaches have value depending on the clinical task.
How Vision Transformers Work (Intuition, No Math)¶
The key idea: images as sequences of patches¶
A Vision Transformer doesn't look at an image pixel by pixel or with sliding filters. Instead, it:
- Divides the image into patches - like cutting a photograph into a grid of tiles (e.g., 16×16 pixel patches). Each patch becomes the ViT's equivalent of a "word" in a sentence.
- Converts each patch into a vector - a list of numbers that represents that patch's content. This is called patch embedding.
- Lets every patch attend to every other patch - this is self-attention, the core innovation. The model learns which patches are related, regardless of how far apart they are in the image.
Self-attention: the expert eye¶
A CNN scans an image systematically with small local filters - it has strong "spatial bias" built in. It naturally captures nearby relationships but struggles with long-range ones.
A ViT is like your expert eye - when reading a chest X-ray, you can immediately relate a finding in the right lower lobe to the cardiac silhouette, even though they're on opposite sides of the image. Self-attention lets the model make these distant connections from the very first layer, without having to stack dozens of convolutional layers.
Positional encoding: where am I?¶
Unlike CNNs, transformers have no built-in sense of spatial position - they process patches as an unordered set. To fix this, we add positional encoding: a unique signal added to each patch that tells the model "this patch came from the top-left" or "this patch came from the bottom-right." Without it, the model couldn't distinguish a rotated image from the original.
The trade-off¶
ViTs are powerful but hungry - they need more data and more compute than CNNs. With ImageNet pretraining, a ViT can match or beat CNNs on many medical imaging tasks. Without pretraining, CNNs typically win on small clinical datasets because their built-in spatial bias gives them a head start.
The Practical Cost of Vision Transformers¶
Notice that we had to make two significant changes for the ViT:
-
Image size: 64×64 → 224×224 - the ViT's patch-based design requires a fixed input size. Resizing from 64×64 to 224×224 means each image is ~12× larger in memory.
-
Batch size: 300 → 64 - because each image is larger, fewer fit in GPU memory at once. This means more batches per epoch and slower training.
This is a real practical consideration when choosing architectures:
| CNN (64×64 input) | ViT (224×224 input) | |
|---|---|---|
| Batch size | 300 | 64 |
| Memory per image | ~16 KB | ~200 KB |
| Batches per epoch | ~157 | ~737 |
| Training time | Faster | ~4–5× slower |
Clinical implication: If you need results quickly during a research project or have limited GPU access, a CNN may be the practical choice. If accuracy is paramount and you have the compute budget, a ViT may be worth the extra cost.
CNN vs. Vision Transformer: Side-by-Side Comparison¶
| Property | CNN (e.g., DenseNet121) | ViT (e.g., ViT-Small) |
|---|---|---|
| Image processing | Sliding filters scan locally | Self-attention relates all patches globally |
| Inductive bias | Strong spatial bias (locality, translation equivariance) | Minimal - learns spatial relationships from data |
| Data efficiency | Good with small datasets (bias helps generalize) | Needs more data or strong pretraining |
| Input flexibility | Works with various image sizes | Fixed patch grid (typically 224×224) |
| Computational cost | Lower - efficient convolutions | Higher - attention scales quadratically with patches |
| Best suited for | Small clinical datasets, real-time applications | Large datasets, tasks where global context matters |
Which should you use?¶
Start with a CNN when: - Your dataset has < 5,000 labeled images - You need fast training iterations - The diagnostic features are local (e.g., a small lesion)
Consider a ViT when: - You have a larger dataset or strong pretrained weights - Global context matters (e.g., relating findings across the whole image) - You're working with a foundation model that uses a ViT backbone (Chapter 8)
In practice, many state-of-the-art medical imaging systems use hybrid approaches or ensembles of both. There is no single "best" architecture - the right choice depends on your data, compute, and clinical question.
What's Next?¶
We've now initialized four models - three CNNs and one Vision Transformer - but we haven't trained any of them yet. This was intentional: separating model setup from training lets us compare architectures fairly.
In Chapter 5, we'll build a generic training function that works with any of these models, train them all, and compare their learning curves and validation performance.
Key takeaway: ViTs represent a fundamentally different philosophy for processing images. CNNs assume locality matters and build in spatial biases. ViTs make fewer assumptions and learn spatial relationships from data - more flexible, but more data-hungry. Both have their place in clinical AI, and understanding both makes you a more effective researcher.