Could this system make it easier to evaluate the screening mammography skills of residents?

When teaching screening mammography techniques, it can be difficult for educators to assess a resident’s overall performance. To solve that problem, researchers have developed a new system that provides quick, helpful feedback to residents as they view exams. The team published its findings in Academic Radiology.

“Based on the challenges in learning, teaching and evaluating screening mammography, we decided to develop a system that would provide active independent resident viewing of screening examinations, timely resident feedback of performance metrics and an objective/quantitative system to assess resident screening skills, based on volumes, recall rates, and concordance and discordance rates,” wrote lead author Petra J. Lewis, MBBS, with the Dartmouth Hitchcock Medical Center, and colleagues. “This system would also define resident level-appropriate screening performance goals and be used by faculty to evaluate residents and enable early intervention if necessary.”

Residents and faculty separately interpreted screening mammograms using a batch reading process to identify non-callbacks and callbacks. Following each screening batch, the resident performed a concordance assessment to review discordant cases. Both concordant and discordant interpretations were entered into a “macro-driven” spreadsheet.

“These macros weigh the data dependent on the perceived clinical impact of the resident’s decision,” the authors wrote. “Weighted outcomes are combined with volumes to generate a weighted mammography performance score. Rotation-specific goals are assigned for the weighted score, screening volumes, recall rate relative to faculty and concordance rates. Residents receive one point for achieving each goal.”

Using this method, more than 18,000 mammography exams were reviewed by 31 residents, within 71 rotations, over 246 resident weeks. The average resident recall rate was approximately 10 percent, but decreased by resident level; second-year residents had an 11 percent recall rate while third- and fourth-year residents had recall rates of 9 percent.

The average resident-faculty discordance rate was 10 percent and decreased by resident level. Second-year residents had a discordance rate of 12 percent, while fourth-year residents had a rate of approximately 10 percent.

“This method provides residents with easily accessible case-by-case individualized screening outcome data over the longitudinal period of their residency and provides an objective method of assessing resident screening mammography performance,” the authors wrote.

Additionally, weighted performance scores ranged from 1.1 to 2, but did not change with rotation experience. Residents had an average goal achievement score of 2.6.

“We present a model for resident education in screening mammography that appears to have great benefits for both resident learners and faculty teacher-clinicians,” the authors concluded. “In a practical sense, the separation of screening interpretation into independent faculty and resident review sessions allows more efficient and less distracted faculty interpretation, and most importantly, for active resident learning.”