Predicting breast cancer types on and beyond molecular level in a multi-modal fashion

Tianyu Zhang, Tao Tan, Luyi Han, Linda Appelman, Jeroen Veltman, Ronni Wessels, Katya M. Duvivier, Claudette Loo, Yuan Gao, Xin Wang, Hugo M. Horlings, Regina G. H. Beets-Tan, Ritse M. Mann

Research output: Contribution to journalArticleAcademicpeer-review

12 Citations (Scopus)


Accurately determining the molecular subtypes of breast cancer is important for the prognosis of breast cancer patients and can guide treatment selection. In this study, we develop a deep learning-based model for predicting the molecular subtypes of breast cancer directly from the diagnostic mammography and ultrasound images. Multi-modal deep learning with intra- and inter-modality attention modules (MDL-IIA) is proposed to extract important relations between mammography and ultrasound for this task. MDL-IIA leads to the best diagnostic performance compared to other cohort models in predicting 4-category molecular subtypes with Matthews correlation coefficient (MCC) of 0.837 (95% confidence interval [CI]: 0.803, 0.870). The MDL-IIA model can also discriminate between Luminal and Non-Luminal disease with an area under the receiver operating characteristic curve of 0.929 (95% CI: 0.903, 0.951). These results significantly outperform clinicians’ predictions based on radiographic imaging. Beyond molecular-level test, based on gene-level ground truth, our method can bypass the inherent uncertainty from immunohistochemistry test. This work thus provides a noninvasive method to predict the molecular subtypes of breast cancer, potentially guiding treatment selection for breast cancer patients and providing decision support for clinicians.
Original languageEnglish
Article number16
JournalNPJ Breast Cancer
Issue number1
Publication statusPublished - 1 Dec 2023

Cite this