Commercially available chest X-ray AI products fail to improve diagnostic accuracy, real world study shows
A new analysis on the performance of several commercially available artificial intelligence tools is challenging assertions that the technology is the answer to many of radiology’s workflow-related issues.
Published in Academic Radiology, the study investigates the utility of four commercially available AI tools designed to interpret chest X-rays—the most common imaging exam in radiologists' work lists. Experts determined that although the tools are beneficial for workflows, they do little to improve detection accuracy.
Imaging AI's potential has been hyped for years, backed by a multitude of studies highlighting efficiency gains, predictive potential and increased detection rates. While promising, these studies are just one small piece AI’s larger place in radiology, and there is not yet enough prospective data to determine how the technology will truly affect organizations, authors of the new paper cautioned.
“Most published studies are retrospective and rely on curated datasets, often excluding low-quality imaging data, and typically focus on individual AI algorithms rather than comparative analyses. Prospective evaluations in real-world settings remain scarce,” Felix Busch, MD, with the Institute for Diagnostic and Interventional Radiology, TUM School of Medicine and Health in Munich, Germany, and colleagues noted. “Moreover, human-AI interaction phenomena such as automation bias and reader dependence are not yet well characterized in chest radiography, despite their potential to influence both diagnostic safety and efficiency.”
For the analysis, researchers evaluated the performance of five readers with one to six years of experience who interpreted 1,861 chest radiographs from 1,200 consecutive patients. Each reader reviewed cases independently without AI and then again using four different commercial algorithms, with a 14-day washout period between sessions. The study assessed detection of pulmonary infiltrates, pleural effusions, mediastinal masses, pneumothorax and pulmonary nodules, and also took note of both interpretation times and detection accuracy.
Overall, the tools did not improve detection accuracy. For pleural effusions and pulmonary nodules, accuracy actually decreased in several reader-tool combinations; this was primarily due to an increase in false positive findings.
However, the tools provided several workflow benefits. Three of the five readers reported faster interpretation times, with median reductions ranging from 6 to 17 seconds per case. Four readers also reported greater diagnostic confidence when using AI assistance. In selected reader-tool combinations, AI reduced the need for consultation with a senior radiologist and, in one instance, eliminated the need for CT escalation.
These findings suggest that efficiency gains from AI do not necessarily translate into better diagnostic performance, the team noted. Researchers also cautioned that increased confidence in AI-assisted interpretations could contribute to automation bias, particularly when an algorithm produces false positive results.
“These results suggest that while current commercial AI algorithms can streamline workflow efficiency and bolster reader confidence, they do not necessarily improve diagnostic accuracy and may introduce false positives, particularly for low-prevalence findings,” the group warned. “A ‘one-size-fits-all' approach to clinical AI is therefore inappropriate, and deployment must be carefully tailored to local workflows and user needs.”
Read more here.
In addition to her background in journalism, Hannah also has patient-facing experience in clinical settings, having spent more than 12 years working as a registered rad tech. She began covering the medical imaging industry for Innovate Healthcare in 2021.