Can mandatory peer review or new VR software improve the quality of radiology reports?

Radiologists rely heavily on voice recognition (VR) software when creating radiology reports, but VR use can lead to less accurate reports. According to a recent study published by the Journal of the American College of Radiology, implementing mandatory peer review is one way to reverse that trend and improve the quality of radiology reports. Introducing a new a VR application, on the other hand, may make less of an impact than anticipated.

Lead author Jonathan H. Chung, MD, University of Chicago Medicine in Chicago, and colleagues noted that report quality is suffering as radiology departments use VR more and more while remaining focused on improving turnaround times. This, they said, is bad for both patients and the profession.

“Error-ridden radiology reports not only confuse clinicians and create a poor impression among patients who read their reports, but they may also have medicolegal ramifications,” the authors wrote.

The authors attempted to boost the quality of their own medical center’s reports in two different ways: by implementing peer review and introducing new VR technology.

Phase 1: Peer review

Chung et al. set up a system to closely monitor the quality of each radiologist’s reports. Ten reports were selected each month and scored by “other anonymous radiologists in the same subfield of expertise” as being either “good,” “fair,” or “poor.” This continued for nine months, and the overall effect was that radiology report quality increased significantly.

“Good” scores made up more than 85 percent of the reports during the first month, but by the ninth month, that was up to more than 91 percent. Also, the percentage of “poor” reports dropped from 2.7 percent to 1.6 percent in that same nine-month timeframe.

The researchers said one may be tempted to cite the Hawthorne effect, which theorizes “the performance of workers can improve solely on the basis of their knowledge that they are being assessed,” but that it’s not quite that simple.

“The consistent improvement in radiology error rate over nine months suggests that the Hawthorne effect did not play a major role in the improvement evident in our department, given the sustained nature of the quality improvement beyond an initial observation period,” the authors wrote. “Moreover, the verity of the Hawthorne effect has recently been questioned; recent reanalysis of data from the landmark study used to develop the theory has shown that no confident conclusions could be made from the original methodology and results.”

A respect for authority did likely play a role in the improved quality, the authors said, as did the fact that radiologists saw others improving and were then likely motivated to do the same.

Phase 2: The new VR application

With the initial nine-month period complete and quality already showing improvement, the team turned to introducing “a new and potentially more accurate version of our VR system.” The results were not what they anticipated.

“The expectation was that the new version of our VR would result in better speech recognition and fewer errors, though this might be tempered initially by the transition to a new system,” the authors wrote. “On the contrary, there was no substantial or significant difference in the radiology report scores before and after the introduction of the new dictation system.”

Seven months after implementation of the VR application, “good” scores were down from 91.5 percent to 90.3 percent, with “poor” scores remaining low.

The authors think this is likely due to the law of diminishing returns.

“Significant improvement in error rates within radiology reports had already been realized during the first nine-month time period, likely because of multiple factors as previously described,” the authors wrote. “Therefore, any incremental improvement in error rates with additional resource investment would likely be small and, therefore, difficult to measure statistically. In addition, small improvements in report accuracy may not be evident, on the basis of an already thorough proofreading system.”