Internal and external validity and the RAND HIE
By Don McCanne, M.D.
In a KPBS debate between James Knight and Don McCanne on consumer-directed health care and health savings accounts, Dr. Knight stated, “… the Rand Health Insurance Experiment… shows that people who pay for their own health care services themselves… will spend about 45 percent less on health care with no significant change in their health care status…”
Actually, the study showed that people who paid 95 percent of their health care costs out-of-pocket had annual expenditures that were 28 percent lower than those who paid nothing for health care. But the much more important point to be challenged is the implication that reduced utilization due to out-of-pocket spending never impacts health status.
Dr. Knight’s statement requires a response which the structured radio format did not permit. Below, internal and external validity are defined, followed by my response to Dr. Knight.
The KPBS debate:
http://www.publicbroadcasting.net/kpbs/news.newsmain?action=article&ARTICLE_ID=873506
The Economics of Health Reconsidered, Second Edition
By Thomas Rice, Ph.D.
Health Administration Press
Two major classifications designate the validity of research studies: internal validity and external validity. Internal validity refers to how well study results accurately represent what actually occurred in the specific setting being examined. External validity refers to how well the results from a particular study can be generalized to other settings and/or populations.
It is generally agreed among the health services research community that the RAND Health Insurance Experiment was very strong in terms of its internal validity. External validity is more of a concern, however.
http://www.ache.org/pubs/rice2.cfm
Email message from Don McCanne (not always known for his diplomacy):
Dr. Knight,
It was a pleasure to debate you on KPBS on the issues of CDHC and HSAs.
We’ll certainly never agree on health care reform, but we really should agree on the objective data in the health policy literature. Bending or breaking the data does not advance the dialogue on reform.
As with other CDHC supporters, you cited the RAND HIE as demonstrating that the decreased utilization due to patient cost sharing did not change health care status. But you are likely aware that economists rightfully have questioned the external validity of this study. Innumerable other studies have demonstrated a significant impairment in access and outcomes resulting from cost sharing. Even a closer look at the RAND HIE data confirmed that cost sharing did have a negative impact on sub-sectors such as hypertensives. The internal validity of the RAND HIE study would apply only to very healthy populations followed for a short enough period of time such that chronic disease would continue to have a very low prevalence in that population.
Applying the RAND HIE results to a cross section of our entire population with its high prevalence of chronic disease is not appropriate, and if done so deliberately to deceive, then is even dishonest.
Though you won’t enjoy it, you should read my Quote of the Day from yesterday that discusses the new RAND study which demonstrates that elimination of cost sharing reduces health care spending.
Based on what I said above, you should immediately challenge my statement. It has very high internal validity for the population using cholesterol-lowering drugs, but its external validity is very limited. Instead of stating, “reduces,” it should have stated, “can reduce.”
If you wish to read more, my message on the new RAND study is at the following link:
http://www.pnhp.org/news/2006/february/rand_eliminating_cop.php
We’ll certainly continue the debate over single payer, CDHC, and the various incremental approaches. At PNHP we pride ourselves on the fact that our single payer advocacy is based on an honest, comprehensive assessment of the health policy literature. If CDHC is to have any credibility, you’ll have to show us solid data demonstrating that your model will not have the negative impact that has been predicted and is already being confirmed. Your task will not be easy because you are beginning with highly flawed policy theory.
Peace,
Don
Don McCanne, M.D.
Senior Health Policy Fellow
Physicians for a National Health Program www.pnhp.org