Radiology AI excels in certain real-world healthcare settings more so than others
A commercial radiology artificial intelligence algorithm excels in certain real-world healthcare settings more so than others, according to new Neiman Health Policy research published Monday.
Intracranial aneurysms are a common clinical concern, seen in about 3% to 4% of the general population. Early and accurate identification of ballooning blood vessels in the brain is crucial before they leak or burst, raising the risk of death or significant injury.
However, identification on CT scans can be challenging even for experienced radiologists, given factors such as reader fatigue or smaller, harder to spot aneurysms. Deep learning has emerged as a promising potential aid to physicians, though concrete evidence is lacking, experts write in the Journal of the American College of Radiology.
A new prospective study aims to close this information gap. In one eye-opening finding, AI shined in emergency and inpatient scenarios, while its benefits were more modest in outpatient settings, generating more false positives than correct ones.
“A likely explanation is that higher-acuity inpatient and emergency settings involve more clinically complex examinations, creating additional opportunities for AI to provide value by serving as a complementary detection tool alongside radiologist interpretation,” study co-author Matthew Barish, MD, Northwell Health’s vice chair of radiology informatics, said in a statement Sept. 15.
The study utilized nearly 4,000 consecutive CT angiography scans gathered at the New Hyde Park, New York-based hospital system in late 2023. Images were processed using an FDA-cleared deep learning algorithm from vendor Aidoc, while natural language processing helped to sort through human physicians’ findings. Radiologists were “blinded” to the AI results, with about 5% (or 195) of the scans positive for intracranial aneurysm.
Physicians and the algorithm agreed in over 96% of cases, researchers found. Artificial intelligence was able to identify 55 true-positive instances—where patients had experienced an aneurysm—that human readers missed. This corresponded to a 39% relative increase in detection compared to radiologist-only performance. Though AI was more sensitive than rads alone (at 85% vs. 72%), finding more aneurysms that were truly present, human readers were more likely to be correct when they pinpointed a positive case (93% vs. 78%).
Both AI and radiologists were similarly strong at ruling out aneurysms when none were present, avoiding false alarms, the authors noted. Barish and colleagues conducted the study in “shadow mode,” with AI processing scans in parallel to providers, so as not to influence their decisions. When any discrepancies arose, independent expert neuroradiologists reviewed the images to establish a correct finding. Radiologists were able to identify 30 true-positive aneurysms that were missed by AI, and 46 of the AI-0nly findings proved to be false positives.
Artificial intelligence’s performance varied significantly across healthcare settings, the authors noted. On the inpatient side, the algorithm identified 18 additional aneurysms while generating only 7 false-positive alerts. Performance also was favorable in the ED, but benefits were more modest in outpatient care. In the latter, AI contributed only four additional detections and generated more false positives than true ones.
“The findings demonstrate why healthcare organizations should evaluate AI based on how it improves physician performance and patient care in real-world use, not solely on results achieved in its original testing environment,” study co-author Elizabeth Rula, PhD, executive director of the American College of Radiology-backed policy institute, said in the announcement. “This study shows that AI can deliver meaningful clinical value by helping radiologists find additional aneurysms while also revealing important differences in performance across care settings. These findings underscore the importance of ongoing monitoring and evaluation after implementation.”
Marty Stempniak has covered healthcare since 2012, with his byline appearing in the American Hospital Association's member magazine, Modern Healthcare and McKnight's. Prior to that, he wrote about village government and local business for his hometown newspaper in Oak Park, Illinois. He won a Peter Lisagor and Gold EXCEL awards in 2017 for his coverage of the opioid epidemic.