Stanford University / University of Pennsylvania / UCSF / Georgetown (Shad, Zakka, Hiesinger et al.)
Foundation vision-language model for cardiac MRI that learns pathophysiological visual representations directly from the natural-language radiology reports accompanying each scan, rather than from hand-labeled targets. A Multi-scale Vision Transformer (MViT, Kinetics-400-initialized) video encoder for cine CMR sequences is contrastively pretrained (InfoNCE) against a PubMed-pretrained BERT text encoder over 19,041 multi-institutional CMR studies. The frozen vision encoder transfers with strong performance to left-ventricular ejection-fraction regression (MAE 3.34% on a UK Biobank hold-out of ~4,259-45,623 participants) and detecting HFrEF (LVEF<40%, AUC 0.880), and the paper reports emergent zero-/few-shot performance across 39 cardiac and non-cardiac conditions including cardiac amyloidosis and hypertrophic cardiomyopathy. Code and pretrained MViT encoder weights are both released (Hugging Face, CC BY-NC 4.0).
Architecture
Hybrid
Multi-scale Vision Transformer (MViT) video encoder, Kinetics-400 pretrained, contrastively aligned (InfoNCE) against a PubMed-pretrained BERT text encoder over paired CMR cine sequences and radiology reports
Framework
PyTorch
Added to catalog
2026-08-10
CC BY-NC 4.0
License for model weights only. Associated code may be licensed seperately, check code source for specific terms.
Stanford / UPenn / UCSF / Georgetown Cardiac MRI Cohort (CMR-Transformer)
19,041 cardiac MRI scans with accompanying radiology reports pooled from four large US academic medical centers, used to pretrain a contrastive vision-language CMR encoder.
UK Biobank cardiac MRI imaging substudy; CineMA pretrained on 74,916 cine CMR studies, ukbb_cardiac (Bai et al. 2018) trained on ~4,875 subjects / 93,500 annotated images from an earlier release.
Contrastively-learned embedding of cine cardiac MRI sequences, transferable to downstream regression/classification tasks
Left ventricular ejection fraction (LVEF) regression from cine CMR
Binary classification of heart failure with reduced ejection fraction (HFrEF, LVEF<40%) from cine CMR