Visual Intelligence Seminar: Do Concepts Help? Evaluating Concept-Based Robustness in Medical Image Analysis
Presented by Valentina Corbetta, PhD Research Fellow at The Netherlands Cancer Institute and Maastricht University
Deep learning models for medical image analysis routinely achieve strong performance in controlled settings, yet frequently fail when deployed under real-world distribution shifts. A central reason is their tendency to exploit spurious correlations — visual patterns that are statistically predictive during training but clinically meaningless.
This talk presents work that tackles this problem by anchoring model representations in clinical concepts.We introduce LCRReg, an in-hoc regularisation method that incorporates Latent Concept Representations directly into the training objective, steering intermediate activations toward clinically meaningful subspaces without requiring dense concept annotations across the full training set.
We then extend this investigation to domain-specific medical vision–language models (VLMs), asking whether contrastive pre-training on clinical image–report pairs already confers robustness to spurious features, and whether explicit concept supervision adds further benefit.
Through controlled experiments across two modalities — fundus photography and mammography — we compare architectures spanning a spectrum of concept integration depth. The results reveal a nuanced picture: contrastive pre-training provides partial robustness, but models remain sensitive to strong spurious associations, and deeper concept integration improves counterfactual robustness at the cost of baseline accuracy.
Together, these results highlight both the promise and the current limitations of concept-based approaches for building more reliable medical AI.