LLMs better than docs at providing clinical context for imaging orders
Large language models could improve protocoling processes by providing readers with valuable clinical information sourced from patients’ electronic health records, according to a new analysis.
Numerous studies have highlighted the benefits of including appropriate clinical context alongside imaging orders to help guide radiologists in their interpretations. Despite this, the information provided is often vague, brief and leaves out useful context that could help rads.
The authors of a new paper published in Radiology believe LLMs could help address this issue by extracting data from patients’ clinical notes to be included in imaging requisitions. A retrospective analysis by the group revealed that, in many cases, LLMs provide more accurate patient histories than the referring providers.
“Clinical histories accompanying imaging orders guide protocol selection and diagnostic focus,” Jae Ho Sohn, MD, MS, with the department of radiology and biomedical imaging at the University of California, San Francisco, and colleagues wrote Tuesday. “However, they are often incomplete, potentially compromising diagnostic accuracy and workflow efficiency.”
For their research, the group analyzed more than 28,000 deidentified patient records from UCS spanning 2012 to 2024. They compared imaging indications written by multiple sources, including referring clinicians, radiologists and AI, before prompting multiple proprietary and open-source LLMs to create clinical histories for imaging orders using patient notes. A group of 20 radiologists with between 2 and 25 years of experience compared the clinical indications produced by each group to assess their comprehensiveness, factuality and conciseness; they also ranked the indications for usefulness in protocoling, interpretation and overall performance.
Claude 3.5 Sonnet and the leading open-source model, Qwen 2.5-7B Instruct, yielded the best performance; they both were rated as significantly more comprehensive and factually accurate than referring provider-supplied indications. Claude 3.5 Sonnet was ranked as the most useful for exam protocoling, image interpretation and overall clinical utility. The research determined that comprehensiveness was the most important factor influencing the overall usefulness of an imaging indication.
“LLMs generated radiology-relevant indications from clinical notes that were more comprehensive and factual than clinician indications, and when generated by the proprietary LLM, were ranked most useful in protocoling and imaging interpretation,” the group noted.
The team believes these findings suggest that LLMs deployed in this care setting could benefit both ordering providers and radiologists.
The study abstract is available here.
In addition to her background in journalism, Hannah also has patient-facing experience in clinical settings, having spent more than 12 years working as a registered rad tech. She began covering the medical imaging industry for Innovate Healthcare in 2021.