Rad Partners scores $1M FDA grant to test new way of evaluating AI-generated radiology reports
Rad Partners subsidiary Cognita Imaging Inc. has scored a $1.29 million grant from the U.S. Food and Drug Administration to test a new way of evaluating AI-generated radiology reports.
The Nashville, Tennessee-based imaging group said the project will focus on a “growing challenge in medical AI.” Under the contract—which took effect June 22 and runs 18 months—experts will develop and validate a framework that uses large language models to assess human-drafted and AI-generated radiology reports.
Rad Partners notes that, while traditional reader studies are essential, expert reviews take time. They also are difficult to extend to the number and variety of cases needed to understand how a model may behave in day-to-day clinical practice.
“Once an AI system starts writing the entire report, the evaluation problem changes,” principal investigator Akshay Chaudhari, PhD, Cognita co-founder, an associate professor at Stanford University, said in a Sept. 16 news announcement from RP. “A few hundred cases may tell you whether a model works in a narrow setting. They do not show every way it can fail in practice. We want to test whether a group of language models, with radiologists reviewing the difficult cases, can give us a clearer and more repeatable picture at real-world scale.”
Cognita—part of Rad Partners’ larger Mosaic Clinical Technologies division—is calling the method “LLMs as a jury.” Instead of asking a single large language model to grade another model’s report, the framework instead compares the judgments of several different LLMs. RP said the goal is to make evaluation more consistent, reduce dependence on any one LLM and identify areas where automated review can “meaningfully extend expert evaluation.”
Those involved plan to apply the framework to approximately 1 million patient exams from a large and diverse U.S. cohort. Researchers hope to examine performance across various patient groups, healthcare settings, imaging equipment and diseases. Radiologist will review clinically important disagreements and determine if the problem came from the AI-generated report, the LLM jury, or the original rad’s findings.
“A model can look strong in a carefully selected study and still struggle at another hospital, on different equipment or with a rare presentation,” “Nina Kottler, M.D., chief medical AI officer at Mosaic Clinical Technologies, said in the announcement. “This work lets us look for those differences at a scale that would be very difficult to reach through reader studies alone, while keeping radiologists involved where their judgment matters most.”
As part of the contract, Cognita will provide the FDA with software code and practical guidance for building LLM juries, analyses comparing large and small validation cohorts, radiologist-reviewed discrepancy studies, and reports on the project’s results. Rad Partners said the contract was awarded through the FDA’s Broad Agency Announcement program for advanced research and development in regulatory science.
Cognita—also co-founded by Louis Blankemeier, PhD, CEO of the company and co-investigator on the FDA project—was first acquired by RP in November 2025 for a reported $80 million.
Marty Stempniak has covered healthcare since 2012, with his byline appearing in the American Hospital Association's member magazine, Modern Healthcare and McKnight's. Prior to that, he wrote about village government and local business for his hometown newspaper in Oak Park, Illinois. He won a Peter Lisagor and Gold EXCEL awards in 2017 for his coverage of the opioid epidemic.