Imaging AI vendors’ marketing hype doesn’t always match materials submitted to the FDA
A new analysis has unearthed discrepancies between what imaging artificial intelligence vendors market to radiologists and other physicians and the claims made in their 510(k) application with the FDA.
Among 119 recently cleared devices analyzed, about 1 in 8 were discovered to have marketing materials that made claims differing from their premarket approval, experts wrote July 5 in JAMA Network Open [1]. In some cases, vendors advertised devices as having capabilities not approved by the FDA for use of artificial intelligence or machine learning.
New York University researchers believe the findings underline the need for further studies and more uniform guidelines around marketing of such devices
“This systematic review found that there was significant discrepancy in the marketing of AI- or ML-enabled medical devices compared with their FDA 510(k) summaries,” Phoebe Clark, a biomedical informatics expert who studied at NYU Langone Health, and co-authors concluded. “Further qualitative analysis and investigation into these devices and their certification methods may shed more light on the subject, but any level of discrepancy is important to note for consumer safety.”
For the study, NYU investigators manually reviewed 510(k) clearance summaries and any accompanying marketing materials for devices cleared between November 2021 and March 2022. Of the total, nearly 13% were considered “discrepant,” with public webpages stating they were enabled for AI while no mention of such capabilities appeared in FDA-submitted materials. Another almost 7% of devices were tagged as “contentious,” meaning they were not flagged by the FDA as AI-based and had no mention of such technology in their application, despite vendors marketing them as such. Clark et al. identified the remaining 81% included in the study as showing consistency between advertising and their FDA application.
The vast majority, at about 82% or 75 devices, were related to radiology, the authors noted. Of those, 83% were deemed adherent, 4% contentious and 13% discrepant.
“How concerned should we be about their findings? A glass-half-full perspective might note that most devices in their sample (approximately 80%) did not involve discrepant marketing claims. But among those that did, the inconsistencies have disquieting implications,” Nigam H. Shah, MBBS, PhD, with the Department of Medicine at Stanford University, wrote in a corresponding editorial [2]. “To fully fathom the implications of the discrepancies that Clark and colleagues identify, it is necessary to examine the decisions that would be made (and the action that would be taken or withheld) based on the AI- and ML-enabled devices’ output,” they added later. “In essence, we need to evolve our unit of examination from the model to the model plus the care workflow it drives.”
Read much more at the links below.
Marty Stempniak has covered healthcare since 2012, with his byline appearing in the American Hospital Association's member magazine, Modern Healthcare and McKnight's. Prior to that, he wrote about village government and local business for his hometown newspaper in Oak Park, Illinois. He won a Peter Lisagor and Gold EXCEL awards in 2017 for his coverage of the opioid epidemic.