Since the publication of the study by McCormick, et al., in
Health Affairs challenging the notion that healthcare IT (HIT) necessarily results in fewer studies being ordered,
mainstream media healthcare blogs have been
buzzing with
discussion about the value of his research.
The simple conclusion of the study—as currently implemented, electronic access does not decrease test ordering in the office setting and may even increase it—has been drawn into a number of different conversations,
and even drew a high-profile critique from Farzad Mostashari, the National Coordinator for Health Information Technology.
Lead author Danny McCormick, MD, says he expected some push-back, given the stakes.
“I think it’s a study that perhaps flies in the face of common wisdom, and so I think some people are surprised,” McCormick says. “There’s a lot of people with investments, monetary and other, in the cost savings of health care IT.”
“Some of this is semantics,” he says. “We have here for the first time a federal database that collects data on a nationally representative sample of physicians practicing in the U.S., in actual practice, not in cutting-edge institutions.”
According to a press release from Health Affairs: “For their study, McCormick and colleagues analyzed data from the 2008 National Ambulatory Medical Care Survey, which includes 28,741 patient visits to a national sample of 1,187 physician-based offices. The survey excludes hospital outpatient departments and offices of radiologists, anesthesiologists, and pathologists.”
McCormick says his research makes “pretty measured conclusions,” and stops far short of any definitive claims “as to what might happen in the future.”
Speculating on Root Causes
Although McCormick acknowledged the study doesn’t help understand the causes of the increase in test orders he observed, he was willing to speculate about what might be behind it. HIT clearly makes ordering tests and retrieving their results a more streamlined process for physicians, and “if you make something easier for people to do, they will do it more often,” he says.
“The frustrating thing about a study like this is that the data were collected for a different purpose, so we just can’t get at the motivation piece,” McCormick says.
“Obviously, future studies are going to have to try and sort out this whole issue of how physician behavior is changed by health IT.”
Mostashari alleged that McCormick’s research was good for generating “attention-getting media headlines” but “tells us nothing about the impact of EHRs on improving care.”
(The same issue of Health Affairs in which McCormick’s study appears also published
several complementary stories on both sides of the question,
including one from Mostashari on the rapid advancement of HIT throughout the country.)
“I think what [Mostashari] is implying there is, ‘Hey, you guys described the world in 2008, but now we have something that we believe is going to be a game-changer’,” McCormick says. “That is yet another hypothesis that needs to be tested.”
Not a Comment on Meaningful Use
Interpreting the results of the study to imply that it somehow disputes the value of meaningful use measures is misguided, McCormick says.
“Our study doesn’t have anything to do with meaningful use,” he says. “We don’t talk about it and it didn’t exist in 2008.
“It’s logical that if you get physicians harmonized in the way they use health IT that that would help improve the effectiveness of HIT and decrease costs,” McCormick says. “The fact is that the same arguments have been made for 50 years prior to 2008 and that test ordering has gone up.”
Mostashari’s rebuttal also dings McCormick and his team for overstating the value of their research, saying the study “falls prey to the classic fallacy of using association to suggest causality.”
McCormick, who teaches courses on research methods at Harvard Medical School, says that if that’s the case, then so do the majority of published medical studies.
The only scientifically valid way of proving the existence of a causal relationship between HIT and increased test ordering would be to conduct a randomized control trial, which McCormick says would never happen due to the costs and logistics of arranging such an experiment.
If a systematic difference existed among patients who went to doctors that had HIT, analysis of those findings would have needed to account for that, McCormick says. As most policy studies are association studies rather than true experiments, he says, the question becomes one of how successful researchers are in controlling for additional variables.
“Bad association studies don’t do any controlling; good ones really go after it at a thoughtful way at every patient, doctor, and practice level,” he says. “Ninety percent of the analysis we describe in the paper was multi-variant modeling to make sure that those patients weren’t different."
“Without an experimental design, you can never say that it’s proven, and we couldn’t possibly say that here, but we didn’t say causality anywhere, and we wouldn’t,” McCormick says. “Irrespective of financial arrangements, at the end of the day, test ordering went up.”
Fielding Misinterpretations
McCormick says he is uncomfortable with interpretations of his findings as a statement on the overall value of HIT. As a primary care doctor who uses Epic software in his own practice, he says there are clear physician benefits from HIT, and that his research didn’t seek to challenge them.
“The conclusions of this paper don’t say anything about physician convenience, quality of life, patient outcomes [being improved or not by HIT],” he says. “They may exist, but our study didn’t examine them.”
Rather, McCormick says, what his study aimed to do was supplement research touting the benefits of HIT “in cutting-edge institutions with whole teams of HIT people designing these systems with the needs of clinicians in mind” with a look into “what in practice actually exists.”
“The conclusion is that despite the claims that HIT will decrease test ordering, we found that, in actual practice, test ordering seemed to go up,” McCormick says.
“We think it should raise a note of caution for those who claim forcefully that HIT in actual practice around the U.S. is likely to decrease costs,” he says. “Clearly, the National Coordinator thinks that we’re definitely headed there.
“But all the modeling studies that were done in the mid-2000s from Rand and others made quite staggering claims as to the dollars saved with HIT, and we need better data,” McCormick says.
“People often point to HIT as the thing that’s going to bail us out of the spiraling costs crisis that we have. From my point of view, HIT may do many things, but it’s possible that bailing us out of the cost crisis isn’t one of them,” he says.
