AI accurately routes interventional radiology procedure requests at pennies apiece

Large language models can accurately route interventional radiology procedure requests and do so for pennies apiece, according to new research published Monday. 

Various subspecialists typically perform interventional procedures in hospitals, with changes in weekend and evening coverage further complicating matters. Referral-intake models can vary among institutions and range from a phone call with a covering radiologist to an EHR consult aided by electronic decision support. 

Scientists with Duke University Medical Center are exploring the use of LLMs to aid in this task—with the models able to recognize requests and generate responses. They detailed their early learnings March 24 in the Journal of Vascular and Interventional Radiology. 

“LLMs are potentially well-suited to translate complex call schedules and coverage between multiple services into a simple chat-interface for clinicians and other staff in a hospital system,” Brian P. Triana, MD, MBA, with Duke’s Department of Radiology, and co-authors advised. “This work assesses the accuracy and cost of a proof-of-concept prompted LLM to route procedure requests to the appropriate phone number or pager at a single large academic hospital.” 

Duke researchers utilized existing teams, pager and phone numbers and schedules to create text-based rules for IR procedure requests. They created a “prompted LLM” to route IR procedure requests at specific dates and times to the appropriate available teams. AI was then retrospectively tested on 250 “in-scope” requests that were explicitly defined by the provided rules. Triana and colleagues submitted an additional 25 “out-of-scope” requests for IR procedures such as an “epidural blood patch,” which were not part of the defined rules of the LLM. 

The team performed their experiment using GPT-4 and GPT-3.5-turbo from OpenAI, along with four more open-weight models. Their LLM correctly routed 96.4% of in-scope interventional procedure requests and 76% of out-of-scope requests using GPT-4. This version outperformed all other models including GPT-3.5-turbo, the authors noted. What’s more, it cost just $0.03 per request using GPT-4. Triana and co-authors calculated the figure using OpenAI API pricing, which was $30/1 million input tokens and $60/1 million output tokens. 

Duke researchers plan to further refine the model and are eyeing a pilot to test it in real-world clinical use. They believe their AI program could work at other institutions, given its adaptability. 

“A key novel component of this implementation is the ease of set-up and flexibility in altering the input covering teams and phone numbers,” the authors noted. “This methodology does not depend on training a dedicated model for the task and instead passes a complete set of rules for each request. This approach is highly adaptable to implementation at other sites where the procedural teams and phone numbers will differ.”

However, Triana and co-authors cautioned others not to structure their own programs in ways that grant outsized control to referrers. 

“Interventional radiology is increasingly functioning as a clinical specialty that operates with referrals and consultations, rather than procedure orders, and caution is advised against structuring intake referral systems that might encourage referring physicians to order procedures,” the authors advised. “Diagnostic radiology procedural teams may operate as technicians with less patient ownership for more basic procedures such as biopsies or lumbar punctures. This tool can be used to direct referring physicians to the appropriate contact, whether it is an order for a fluoroscopic lumbar puncture if one cannot be performed without imaging, or a consult to an interventional radiologist for assistance managing a patient with portal hypertension.”