AI Radiology Tools Show Minimal Time Savings
Brief notes from three papers that did real-world studies on the impact of introducing an AI assistant into radiology workflows.
Overall, AI tools seem to have a very modest impact on radiology report writing times. These 3 studies show that AI tools either increase writing times by 5s per report (+36%), decrease it by 12s per report (-50%), or decrease it by 1s per report (-10%).
1. Kim, Eun Young, et al. “Concordance rate of radiologists and a commercialized deep-learning solution for chest X-ray: Real-world experience with a multicenter health screening cohort.” PLoS One 17.2 (2022): e0264383.
Link: https://pubmed.ncbi.nlm.nih.gov/35202417/
Setup: Retrospective evaluation of concordance (i.e. agreement) between radiologists and the Lunit INSIGHT CXR commercial AI tool on thoracic abnormality identification in chest radiography (CXR) images. Lunit marked image with a heatmap of abnormalities, and classified anything above 15% as abnormal. A total of 3k patients were involved.
Results:
- Concordance rate was 86.8%
- AI tool slightly increases median time to write report from 14s => 19s
Takeaways:
- AI tool is accurate, but increases time taken to write report
2. Sung, Jinkyeong, et al. “Added value of deep learning–based detection system for multiple major findings on chest radiographs: a randomized crossover study.” Radiology 299.2 (2021): 450-459.
Link: https://pubs.rsna.org/doi/full/10.1148/radiol.2021202818
Setup: Retrospective evaluation of VUNO Med-Chest X-Ray commercial AI tool’s AUROC. A total of 228 patients were involved.
Results:
- Human-only AUROC = 0.93
- Human + AI AUROC = 0.98
- AI-only AUROC = 0.98
- Mean time to write report decreased from 24s => 12s
- Under other metrics (JAFROC), AI-only outperformed both Human-only and Human+AI:
Takeaways:
- AI tool alone is most accurate
- AI decreases time taken to write report
3. Shin, Hyun Joo, et al. “The impact of artificial intelligence on the reading times of radiologists for chest radiographs.” NPJ Digital Medicine 6.1 (2023): 82.
Link: https://www.nature.com/articles/s41746-023-00829-4
Setup: Measure reading times of radiologists with and without commercial AI tool. A total of 18,680 CXRs were involved.
Results:
- Mean reading time decreased from 14.8s => 13.3s
- If no abnormality detected by AI, decreased from 13.1s => 10.8s
- If abnoramlity detected by AI, same time of 18.4s => 18.6s
Takeaways:
- AI tool slightly decreases reading time