ACR, top health systems form collaborative to help radiologists assess AI solutions

The American College of Radiology and several top healthcare systems have formed a collaborative to help radiologists and other physicians assess artificial intelligence offerings. 

Mass General Brigham in Boston is hosting the Healthcare AI Challenge, which will roll out a series of virtual “events” allowing healthcare professionals to explore these technologies in real-world-like scenarios. Others participating include Emory Healthcare in Atlanta and the departments of radiology at the University of Wisconsin and UW Medicine in Seattle. 

They’ve chosen radiology as the initial area of focus, with the specialty accounting for the highest number of algorithms cleared by the U.S. Food and Drug Administration. 

“We are facing an overwhelming influx of FDA-approved AI tools in healthcare, especially in radiology. Forming an academic collaborative could play a crucial role in validating and selecting these tools, ensuring they adhere to the highest standards of efficacy and safety,” Dushyant Sahani, MD, chair and professor of the Department of Radiology at the University of Washington School of Medicine, said in a Nov. 13 announcement.

ACR joined the collaborative as a founding organization to ensure that its 42,000 members have access to the Healthcare AI Challenge. Those involved said they plan to add additional institutions in forthcoming phases. Participating professionals will gain access to late-breaking AI solutions they can assess for effectiveness in a simulated environment. They then can provide feedback on software performance, generating publicly available information for providers and patients. Those involved hope this crowdsourcing approach will create a repository of reliable artificial intelligence reviews. 

“AI has the potential to interpret and distill…data at a new scale and speed, but what we need is the ability to quickly test and compare different AI solutions,” Richard Bruce, MD, associate professor of radiology and the vice chair of informatics at the University of Wisconsin School of Medicine and Public Health, said in the announcement. “The Healthcare AI Challenge will offer a platform to evaluate and compare tools across various clinical situations.”

Radiologists and other healthcare professionals at the participating institutions can register and log into the platform and select one of several clinical scenarios. The image interpretation challenge includes questions focused on draft report generation, key findings, differential diagnosis and more. They then can rate the skill level of foundation models’ responses. 

Members of the public can track the results at HealthcareAIChallenge.org. Those involved expect to generate valuable insights to help AI developers further refine their models. Facilitators plan to continuously add new solutions, events and data into the challenge. Pathology and genomics are two of the professions expected to be incorporated in the future.  

“Participating in the Healthcare AI Challenge is another important step in ACR’s efforts to facilitate the safe, responsible and transparent evaluation of AI, especially as the use of newer foundation models is poised to unfold in healthcare,” Christoph Wald, MD, PhD, MBA, vice chair of the ACR Board of Chancellors and head of its Commission on Informatics, said in the announcement.