Of 300-plus imaging-based AI models for COVID-19 diagnosis, zero suitable for clinical use
Out of more than 300 published machine learning models for detecting COVID-19 from chest images, zero passed a recent research review, experts detailed Monday in Nature.
As the pandemic took hold last year, numerous publications touted the use of AI to help pinpoint the novel coronavirus on X-ray and CT scans. U.K. researchers, however, are casting doubts on these claims amid concerns of “Frankenstein data sets,” biases and methodological flaws.
“Any machine learning algorithm is only as good as the data it’s trained on,” first author Michael Roberts, PhD, with the University of Cambridge's Department of Applied Mathematics and Theoretical Physics, said in a statement. “Especially for a brand-new disease like COVID-19, it's vital that the training data is as diverse as possible because, as we've seen throughout this pandemic, there are many different factors that affect what the disease looks like and how it behaves.”
To reach their conclusions, Roberts and co-authors conducted a systematic review of scientific studies published between Jan. 1 and Oct. 3 of 2020. They specifically sought those that described machine learning models, claiming to be able to diagnose or prognosticate for COVID-19 from chest X-rays and CT scans.
They identified 2,212 studies, whittling the list down to 415 after initial screening and 62 in the final review. None of those 62 were suitable for clinical use, they noted, a major concern given the pandemic’s urgency. Roberts and colleagues cited numerous issues in published literature—with poor quality of data, application of machine learning methodology and reproducibility. Many, they noted, were trained on data sets that were far too small to be effective, while others failed to involve radiologists or other clinicians.
“Whether you're using machine learning to predict the weather or how a disease might progress, it's so important to make sure that different specialists are working together and speaking the same language, so the right problems can be focused on,” Roberts said.
Despite their concerns, the study authors see promise in using machine learning to combat the pandemic. Possible data set fixes could include moving away from “naïve” use of public information and striving for more diversity.
You can read more about their findings in Nature Machine Intelligence here.
Marty Stempniak has covered healthcare since 2012, with his byline appearing in the American Hospital Association's member magazine, Modern Healthcare and McKnight's. Prior to that, he wrote about village government and local business for his hometown newspaper in Oak Park, Illinois. He won a Peter Lisagor and Gold EXCEL awards in 2017 for his coverage of the opioid epidemic.