Explainable Multimodal Vision–Language Foundation Models for Automated Rib Fracture Detection, Anatomical Reasoning, and Clinically Faithful Radiology Report Generation
DOI:
https://doi.org/10.68339/jaims.v1i1.6Keywords:
RibFrac dataset, Rib fracture detection, Multimodal vision-language models, explainable artificial intelligence (XAI), Radiology report generationAbstract
Rib fractures are frequent thoracic injuries that need to be promptly diagnosed to avoid complications. Multi-modal vision–language foundation models (VLMs) have recently demonstrated success in medical image understanding, but their performance in detecting rib fractures, performing anatomical reasoning, and generating radiology reports has not been thoroughly assessed. Systematic comparison of four state-of-the-art models: ChatGPT-5 Vision, Gemini 2.5 Pro, Claude Sonnet 4.5 Vision, and Mistral Multimodal, on an objective test set of 120 thoracic computed tomography (CT) examinations from the RibFrac dataset. A standard zero shot prompting approach was used to evaluate each model, and the predictions were compared to reference annotations. The effectiveness of the model was assessed by measuring diagnostic accuracy, sensitivity, specificity, precision, F1 score, localization accuracy, report quality and explainability with feature analysis using SHAP. The experimental results show that ChatGPT-5 had the best diagnostic performance with 95%, followed by Gemini 2.5 Pro with 93%, Claude Sonnet 4.5 with 91%, and Mistral with 88%. The fracture visibility, rib location, laterality, and displacement were the most influential factors identified by SHAP analysis that influenced model predictions. The results suggest that AI multimodal foundation models hold promise for enhancing rib fracture evaluation, radiology reporting, and the need for explainability in clinical decision support for trustworthy AI applications.
References
[1] N. Wu, J. Zhang, W. Zhao, J. Wang, C. Yu, Z. Xiu, D. Dai, Z. Xu, et al., “OrthoInsight: Rib fracture diagnosis and report generation based on multi-modal large models,” arXiv preprint arXiv:2507.13993, 2025, doi: 10.48550/arXiv.2507.13993.
[2] R. C. Lee, R. Hadidchi, M. Coard, Y. Rubinov, T. T. Alamuri, A. Liaw, R. Chandrupatla, and T. Q. Duong, “Use of large language models on radiology reports: A scoping review,” Journal of the American College of Radiology, 2025, doi: 10.1016/j.jacr.2025.10.005.
[3] Y. Artsi, E. Klang, J. Collins, B. Glicksberg, P. Korfiatis, G. Nadkarni, and V. Sorin, “Large language models in radiology reporting: A systematic review of performance, limitations, and clinical implications,” medRxiv, 2025, doi: 10.1101/2025.03.18.25324193.
[4] B. Khosravi, T. Dapamede, F. Li, Z. M. Chisango, A. Bikmal, S. Garg, B. Owosela, A. Khosravi, et al., “Role of model size and prompting strategies in extracting labels from free-text radiology reports with open-source large language models,” Journal of Imaging Informatics in Medicine, 2025, doi: 10.1007/s10278-025-01505-7.
[5] M. Mergen, D. Spitzl, C. Ketzer, M. Strenzke, A. Marka, M. R. Makowski, K. Bressem, L. C. Adams, et al., “Leveraging large language models for accurate AO fracture classification from CT text reports,” Journal of Imaging Informatics in Medicine, 2025, doi: 10.1007/s10278-025-01603-6.
[6] J. Liu, B. Koopman, N. J. Brown, K. Chu, and A. Nguyen, “Generating synthetic clinical text with local large language models to identify misdiagnosed limb fractures in radiology reports,” Artificial Intelligence in Medicine, vol. 158, Art. no. 103027, 2024, doi: 10.1016/j.artmed.2024.103027.
[7] C. K. C. Ng, Z. Sun, and I. K. H. Te, “Performance of large language models for radiology report impression generation: A systematic review,” Technologies, vol. 14, no. 2, Art. no. 99, 2026, doi: 10.3390/technologies14020099.
[8] S. Lee, J. Youn, H. Kim, M. Kim, and S. Yoon, “CXR-LLaVA: A multimodal large language model for interpreting chest X-ray images,” European Radiology, vol. 35, 2025, doi: 10.1007/s00330-024-11339-6.
[9] S. Bannur, K. Bouzid, D. C. Castro, A. Schwaighofer, S. Bond-Taylor, M. Ilse, F. Pérez-García, V. Salvatelli, et al., “MAIRA-2: Grounded radiology report generation,” arXiv preprint arXiv:2406.04449, 2024, doi: 10.48550/arXiv.2406.04449.
[10] D. Reichenpfader, H. Müller, and K. Denecke, “A scoping review of large language model based approaches for information extraction from radiology reports,” npj Digital Medicine, vol. 7, no. 1, Art. no. 224, 2024, doi: 10.1038/s41746-024-01219-0.
[11] L. Zhang, M. Liu, L. Wang, Y. Zhang, X. Xu, Z. Pan, Y. Feng, J. Zhao, et al., “Constructing a large language model to generate impressions from findings in radiology reports,” Radiology, vol. 313, no. 3, Art. no. e240885, 2024, doi: 10.1148/radiol.240885.
[12] B. Altalla, A. Ahmad, L. Bitar, M. O. Al-Bssol, A. A. Omari, and I. Sultan, “Radiology report annotation using generative large language models: Comparative analysis,” International Journal of Biomedical Imaging, vol. 2025, Art. no. 5019035, 2025, doi: 10.1155/ijbi/5019035.
[13] F. Busch, L. Hoffmann, D. Pinto dos Santos, M. R. Makowski, L. Saba, P. Prucker, M. Hadamitzky, N. Navab, et al., “Large language models for structured reporting in radiology: Past, present, and future,” European Radiology, vol. 35, 2025, doi: 10.1007/s00330-024-11107-6.
[14] T. Liu, J. Wang, Y. Hu, M. Li, J. Yi, X. Chang, J. Gao, and B. Yin, “HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation,” arXiv preprint arXiv:2412.11070, 2024, doi: 10.48550/arXiv.2412.11070.
[15] J. Yang, L. Jin, B. Ni, and M. Li, “RibFrac Dataset: A Benchmark for Rib Fracture Detection, Segmentation and Classification (Training Set Part 2),” Zenodo, Jun. 15, 2020, doi: 10.5281/zenodo.3893498.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Fnu Muhibullah, Amna Ehsan, Tayyaba Kauser (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
