We are developing multimodal AI systems to better predict and detect breast cancer.
Breast cancer remains the most prevalent cancer among women worldwide and a leading cause of cancer-related mortality. Early detection through imaging-based screening has dramatically improved outcomes, yet the process remains limited by human variability, false positives, and the challenge of identifying subtle, high-risk features across modalities such as mammography and ultrasound. Our research aims to address these limitations through the development of multimodal artificial intelligence systems that enhance both cancer detection and risk prediction.
To accommodate the unique challenges of high-resolution breast imaging, we have designed hierarchical and weakly supervised AI models capable of learning from image-level labels while producing lesion-level interpretability maps. These systems matched radiologist performance in detecting malignant findings on mammography and ultrasound while enabling a significant reduction in false positives during reader studies. Beyond supervised training, we advanced self-supervised and multiple-instance learning (MIL) methods to improve instance-level representations when only coarse labels are available, a common scenario in clinical datasets.
Our models integrate multi-view and multimodality information, combining mammography, ultrasound, and density features to enhance predictive accuracy and generalizability across millions of clinical images. This line of research not only advances technical understanding of AI architectures for medical imaging but also provides clinically actionable tools that improve diagnostic consistency, reduce unnecessary biopsies, and enable data-driven breast cancer risk profiling—paving the way to precision screening and personalized prevention strategies.
Figure 1. (a) Ultrasound (US) images were pre-processed to extract the breast laterality (i.e., left or right breast) and to include only the part of the image which shows the breast (cropping out the image periphery which typically contains textual metadata about the patient and US acquisition technique). (b) For each breast, we assigned a cancer label using the recorded pathology reports for the respective patient within a time interval ranging from 30 days before to 120 days after the US examination. We applied additional filtering on the internal test set to ensure that cancers in positive exams are visible in the US images and negative exams have at least one cancer-negative follow-up. (c) The AI system processes all US images acquired from one breast to compute probabilistic predictions for the presence of malignant lesions. The AI system also generates saliency maps that indicate the informative regions in each image. (d) We evaluated the system on an internal test set (AUROC: 0.976, 95% CI: 0.972, 0.980, n = 79,156 breasts) and an external test set (AUROC: 0.927, 95% CI: 0.907, 0.959, n = 780 images). e In a reader study consisting of 663 exams (n = 1024 breasts), we showed that the AI system can improve the specificity and positive predictive value (PPV) for 10 attending radiologists while maintaining the same level of sensitivity and negative predictive value (NPV).
Figure 2. Overall architecture of the Globally-Aware Multiple Instance Classifier (GMIC). The model first employs a computationally efficient global module to generate a saliency map that captures global context and highlights regions of interest (ROIs) on the mammogram that may correspond to breast cancers. A local module then processes only these ROIs to extract fine-grained spatial details. Finally, a fusion module integrates both global and local information to produce the final cancer diagnosis. The entire model is trained end-to-end using only image-level binary labels indicating the presence of breast cancer, yet it can accurately localize suspicious regions.
Project Lead
We acknowledge support from the following grants: NIH 1R01EB036530, Milstein Pilot Project Award 2025, Manhasset Women’s Coalition Against Breast Cancer Research Fund 2024, Shifrin-Myer Breast Cancer Discover Award 2024.
Researchers at the Center for Biomedical Imaging at NYU Langone Health develop transformative imaging technologies to advance basic science and address unsolved clinical problems.
660 First Avenue
4th Floor
New York, NY 10016
© Center for Biomedical Imaging (CBI). All rights reserved.
This content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, NYU Langone Health, or New York University.
