By Katherine Baicker, Ph.D., and Amitabh Chandra, Ph.D.
The New England Journal of Medicine, December 21, 2017
In these times of heated rhetoric about what various health care reforms can and cannot accomplish, both hopeful and doomsday stories abound. Proponents and opponents of reforms often claim that their views are grounded in evidence, but it’s not always clear what they mean by that — particularly given the wide range of often incompatible views. Voters, physicians, and policymakers are left to wade through a jumble of anecdotes, aspirations, associations, and well-designed studies as they try to evaluate policy alternatives. Having a clear framework for characterizing what is, and isn’t, evidence-based health policy (EBHP) is a prerequisite for a rational approach to making policy choices, and it may even help focus the debate on the most promising approaches.
EBHP, we believe, has three essential characteristics. First, policies need to be well-specified; a slogan is not sufficient. For example, “expand Medicaid” isn’t a policy. “Expand existing Medicaid benefits to cover all adults below the poverty line” is closer — but, of course, moving to a specific, implementable program requires vastly more detail.
Second, implementing EBHP requires us to distinguish between policies and goals. This distinction is important in part because different people may have different goals for a particular policy.
Third, EBHP requires evidence of the magnitude of the effects of the policy, and obtaining such evidence is an inherently empirical endeavor. Introspection and theory are terrible ways to evaluate policy.
What makes for “rigorous enough evidence”? Professional medical societies have developed gauges of the strength of evidence to support clinical guidelines, and we should demand nothing less for health policy. No study is perfect, and important policy questions are rarely answered definitively by any one study. Nor does pointing to a large literature with similar results prove a point if those studies share a common weakness such as an inability to control for confounders.
There is also a key difference between “no evidence of effect” and “evidence of no effect.” The first is consistent with wide confidence intervals that include zero as well as some meaningful effects, whereas the latter refers to a precisely estimated zero that can rule out effects of meaningful magnitude. These nuances are often lost when “evidence” is deployed in policy debates.
The effect of a policy, of course, also depends on the design and implementation details and the program particulars, and evidence needs to speak to those particulars. It is also important to consider the full range of a policy’s effects — its costs and benefits, and how each of these evolves over time.
Making health policy on the basis of evidence will always be a fraught and uncertain endeavor, and each component we outline here comes with challenges.
In addition, just as the distinction between policies and goals is often muddied, interpretations of the evidence are often flavored by the implicit goals of the analyst.
Finally, even a rich body of evidence cannot guarantee that a policy will achieve its goals, and waiting for that level of certainty would paralyze the policy process. In health policy — as in any other realm — it is often necessary to act on the basis of the best evidence on hand, even when that evidence is not strong.
Just because something sounds true doesn’t mean that it is, and magical thinking won’t improve our health care system. EBHP helps separate facts from aspiration. But as important as evidence is to good policy choices, it can’t tell us what our goals should be — that’s a normative question of values and priorities. Better policy requires being both honest about our goals and clear-eyed about the evidence.
***
Comment:
By Don McCanne, M.D.
What role should evidence-based health policy play in the design of our system of health care financing? Before answering that, we need to decide what the goals are. Then we need to survey various policies that might achieve those goals. Finally we should look at the evidence as to how effective each of those policies might be. How are we doing?
Of course, the goals are most important. At PNHP we believe that everyone should have access to comprehensive health care that is of reasonable quality and is affordable – goals achievable only through the collective effort of government working with the public and private health care delivery system. Others believe that health care should be an individual responsibility thus they have a goal of minimizing the role of government. Obviously the selection of policies, whether evidence-based or not, would be quite different for these sharply divergent goals.
When you have a system that is designed to cover absolutely everyone, what evidence-based studies do you need to be sure everyone really is covered? Silly question – you don’t need any studies, everyone is automatically covered. Accessible? Regional planning with appropriate allocation of resources – no policy studies, you just do it. Comprehensive? Simply cover appropriate care (which might be guided by evidence-based health care studies, but you don’t need evidence-based policy studies). Affordable? Fund the system using progressive tax policies – you do not need an evidence-based policy study to show that those who can afford it need to pay more than those who can’t.
A prime example of placing evidence-based policy before goals is the classic RAND Health Insurance Experiment. That study showed that requiring cost sharing payments when receiving health care reduced total spending on health care. Although it is conceded that cost sharing (deductibles, copayments, or coinsurance) does reduce the use of beneficial health care, little attention has been paid to that. Instead, the policy of cost sharing, especially high deductibles, has been used for the goal of reducing spending, while ignoring the goal of affordable access to health care. This demonstrates why the goals are so much more important. This tradeoff satisfies the supporters of individual responsibility in health care but fails for those who support health care for everyone.
We are seeing a similar process with ACOs, MACRA, MIPS, APMs and other policies that supposedly replace volume with value, or quality instead of quantity. What are the goals? Although some lip service is given to quality, reduced spending is the real goal. The goals of health care justice as supported by PNHP are nowhere in sight. So we have moved forward in implementing these policies, but are they evidence-based? Many studies have been done and many more are in process, but the evidence is that they are relatively ineffective. Yet the public and private bureaucrats are implementing them because of their perceived goal when, in fact, they do not have adequate evidence that these policies are effective.
Evidence-based health policies may be of value in limited circumstances. But the real debate is over the goals. Let’s fight that one out now. Universality, access, equity, affordability, comprehensiveness, and administrative efficiency are the goals to which we should aspire. We do not need decades of building an evidence-based case for implementing the policies that effectuate these goals.
Stay informed! Visit www.pnhp.org/qotd to sign up for daily email updates.
