We are building a multimodal artificial intelligence system to provide early estimate of pancreatic cancer risk, enable early detection, facilitate cost-effective surveillance, and support timely intervention.
Pancreatic cancer is among the deadliest malignancies, largely because it is typically diagnosed at an advanced stage when curative treatments are no longer possible. In this project, we are building an AI system that integrates multimodal and longitudinal data to predict a patient’s near- and long-term risk of pancreatic cancer, guide personalized surveillance, and inform management strategies.
To enable such a system, we are constructing a comprehensive longitudinal database that combines imaging (CT and MRI), demographics, family and social history, comorbidities, and laboratory results. The AI model will learn to dynamically estimate an indifvidual’s riwk of developing pancreatic cancer by analyzing trajectories of subtle tissue and ductal changes on imaging alongside evolving clinical variables.
A personalized system for early prediction of pancreatic cancer risk stands to support clinical care in several ways: by optimizing the frequency and modality of imaging surveillance, identifying when invasive diagnostic workups are warranted, and guiding surgical decisions in borderline cases. Furthermore, the system we envision has the potential to reveal imaging biomarkers that precede the onset of cancer by years and to offer new insights into the biological mechanisms of malignant transformation.
Figure 1. We developed and evaluated an LLM-based system for the automatic extraction and risk categorization of pancreatic cystic lesions (PCLs) from radiology reports. The system processes free-text reports to extract clinically relevant PCL features, which are then mapped to risk categories based on established guidelines. We evaluated both proprietary (GPT-4o) and open-source (LLaMA, DeepSeek) models, including fine-tuned variants, across five dimensions: PCL feature extraction accuracy, PCL risk categorization performance, error and hallucination analyses, radiologist agreement assessment, and cost-efficiency. Fine-tuned models achieved feature extraction accuracy comparable to GPT-4o (LLAMA-FT: 97% [95% CI: 97–98%], LLAMA-FT-CoT: 97% [97–98%], DeepSeek-FT-CoT: 98% [97–98%], GPT-CoT: 97% [97–98%]). Risk categorization F1 scores were similarly high (LLAMA-FT: 0.95 [0.91–0.98], LLAMA-FT-CoT: 0.93 [0.89–0.97], DeepSeek-FT-CoT: 0.94 [0.90–0.98], GPT-CoT: 0.97 [0.93–0.99]). The radiologist agreement assessment showed strong agreement between models and expert radiologists (Fleiss’ kappa: radiologists alone = 0.888; radiologists + DeepSeek-FT-CoT = 0.893; radiologists + GPT-CoT = 0.897), indicating LLM-derived outputs are on par with expert interpretation.
Figure 2. Workflow of the AI system. Data Collection: CT studies with multiple series were used in this study. Severity labels were assigned based on the Revised Atlanta Classification (RAC). Image Preprocessing: During training, MedSAM, an automated deep learning segmentation model, was used to generate pancreas masks and identify relevant slices. These were combined with randomly sampled non-pancreas slices to create preprocessed input series. Segmentation is not required at inference time. Model Development: A transformer-based backbone (pretrained via self-supervised learning) extracted slice-level features. Slice scores derived from saliency maps were aggregated using attention (ATTN) to generate severity predictions, optimized with binary cross entropy (BCE), Dice, and L1 losses. Model Evaluation: Study-level predictions were generated by using the maximum series-level score. Retrospective Triage Simulation was performed on both internal and external test sets.
Project Lead
We acknowledge support from the following grants: Society of Abdominal Radiology Research Award 2025, RSNA 2024 Research / Education Grant, and Hirshberg Foundation Seed Grant 2024.
Researchers at the Center for Biomedical Imaging at NYU Langone Health develop transformative imaging technologies to advance basic science and address unsolved clinical problems.
660 First Avenue
4th Floor
New York, NY 10016
© Center for Biomedical Imaging (CBI). All rights reserved.
This content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, NYU Langone Health, or New York University.
