Dealing with conflict

It occurred to me yesterday that I haven't done much straight medblogging in a while. I suppose you could count all my posts lambasting the thimerosal-autism crowd for conspiracy-mongering, shoddy science, and fear-mongering, but I'm not sure that really counts, certainly not after the Hitler zombie made an appearance, anyway.

Fortunately, hot off the presses, a new study from the Journal of the American Medical Association came to my attention yesterday, and it has relevance to a lot of medical news. Written by John P. A. Ioannidis, MD and entitled, Contradicted and Initially Stronger Effects in Highly Cited Clinical Research, this study took a rather clever but straightforward look at contradictory studies. Dr. Ioannidis looked at all original clinical research studies published in three major general clinical journals (JAMA, the New England Journal of Medicine, and Lancet) and selected high impact specialty journals. Articles published between 1990-2003 and cited more than 1,000 times in the literature were chosen for analysis. He then looked at the results of these highly cited studies and then compared them to the results of subsequent studies of comparable or larger sample size with similar or better-controlled experimental designs. For comparison, he also examined a set of matched studies that were not nearly as widely cited. He sets up the background here:

Clinical research on important questions about the efficacy of medical interventions is sometimes followed by subsequent studies that either reach opposite conclusions or suggest that the original claims were too strong. Such disagreements may upset clinical practice and acquire publicity in both scientific circles and in the lay press. Several empirical investigations have tried to address whether specific types of studies are more likely to be contradicted and to explain observed controversies. For example, evidence exists that small studies may sometimes be refuted by larger ones.

Similarly, there is some evidence on disagreements between epidemiological studies and randomized trials. Prior investigations have focused on a variety of studies without any particular attention to their relative importance and scientific impact. Yet, most research publications have little impact while a small minority receives most attention and dominates scientific thinking and clinical practice. Impact is difficult to measure in all its dimensions. However, the number of citations received by a publication is a surrogate of the attention it has received in the scientific literature and its influence on scientific debate and progress. Citations are readily and objectively counted in established databases. High citation count does not necessarily mean that these studies are accepted; citations may sometimes be critical of an article. Nevertheless, citation count is ameasure of how much a study has occupied the thinking of other scientists and has drawn attention—for good or bad.

I would tend to agree that citation count is not a bad rough estimate for how influential a study is. Dr. Ioannidis ended up choosing 49 highly cited studies. All the usual later refuted studies are there, including the Nurses' Health Study, which showed a decrease in cardiovascular events in women receiving hormone replacement therapy but was later contradicted by the Women's Health Initiative, a large randomized trial; the intial studies showing a benefit of vitamin E on cardiovascular health (the trial contradicting that study has recently been in the news); a study showing human IgM monoclonal antibody to endotoxin could decrase mortality in gram-negative sepsis, later contradicted by large studies; and the initial study showing a supposed benefit of nitric oxide in respiratory distress syndrome, also later contradicted. Of these 49 studies, Dr. Ioannidis noted that 45/49 studies claimed that the studied intervention was effective. Of these "positive" trials, 16% were contradicted by later studies; 16% reported much stronger effects than what subsequent studies reported; 44% were eventually replicated with results similar to the initial trial; and 24% remained unreplicated and "unchallenged" for now. There was no significant difference in these numbers in less widely cited studies, except that a lower percentage of them showed "positive" results.

What this study suggests is that at least 1/3 of widely reported clinical trials may either be incorrect or show a much larger effect due to an intervention than the "real" effect. Possible reasons for discrepancies between initial results and later trials may include publication bias (positive studies are more likely to see publication in high-impact journals than negative studies) or time-lag bias (which favors the rapid publication of interesting or important "positive" results). Also, high impact journals like JAMA and NEJM are always on the lookout for "sexy" findings, findings likely to have a strong impact on medical practice or that challenge present paradigms, which may sometimes lead them to overlook flaws in some studies or publish pilot studies with small numbers. Dr. Ioannidis' results are probably not surprising to most doctors, who have been trained to understand that no one single study can be the final word, no matter how seemingly compelling the results, but I rather suspect that the lay public will be shocked. Of course, as Dr. Ioannidis points out in the article:

We should acknowledge that there is no proof that the subsequent studies and meta-analyses were necessarily correct. A perfect gold standard is not possible in clinical research, so we can only interpret results of studies relative to other studies. Whenever new research fails to replicate early claims for efficacy or suggests that efficacy is more limited than previously thought, it is not necessary that the original studies were totally wrong and the newer ones are correct simply because they are larger or better controlled. Alternative explanations for these discrepancies may include differences in disease spectrum, eligibility criteria, or the use of concomitant interventions. Different studies on the same question are typically not replicas of each other. In fact discrepancies may be interesting on their own because they require careful scrutiny of the data and reappraisal of our beliefs.

On the other hand, he notes that, compared to all 14 studies whose results were contradicted or softened, the subsequent studies were either larger or better designed. More importantly, none of the contradicted treatments is currently recommended by medical guidelines, which implies that later studies usually do make up for spurious findings of initial studies.

I think the most important message for lay people to take home from this study is what I've been saying all along. With rare exceptions, you can't rely on any one study for the definitive answer to any clinical question. As Dr. Ioannidis put it, there is indeed no gold standard in clinical research. All a physician can do is to look at the preponderance of evidence and apply his/her best judgment. Moreover, we should all look at the results of highly touted studies with a bit of skepticism. It's when multiple studies start coming to the same conclusion that one can be more confident in the results.

One thing this study does highlight is the difference between evidence-based medicine and much of "alternative" medicine. Alties frequently accuse us "conventional" doctors of being "dogmatic" or otherwise unwilling to consider different ideas or treatments (specifically their ideas and treatments) about medicine and the treatment of disease. In actuality, it is alties who tend to be more dogmatic and unchanging. After all, if the results of 1/3 of seemingly very important papers over 13 years were later refuted, resulting in the abandonment on the basis of new evidence of accepted treatments once thought sound, doesn't that tell you something? It tells me that "conventional" medicine changes its treatments on the basis of new evidence and, more importantly, abandons treatments found to be ineffective or not as effective as newer therapies. It may not happen as fast as we'd like, but happen it does eventually. Contrast this to alternative medicine, where there are still alties pushing Laetrile (despite the fact that it was shown to have no efficacy against cancer in well-designed clinical trials 25 years ago) and chelation therapy for coronary artery and peripheral vascular disease (despite multiple randomized studies during the 1990's showing it to be no better than placebo). Indeed, it is not uncommon to see alties quoting work that is over 100 years old, conveniently ignoring the advances that have occurred since then. A prime example is altie favorite Antoine Beauchamps, about whom alties like Bill Maher like to repeat the myth that Louis Pasteur made a deathbed admission that Beauchamps was right and he was wrong about the germ theory of disease.

This study is also relevant to the thimerosal-autism debate. Mercury-autism hawkers like to point to early analyses of studies that appeared to show an epidemiological link between thimerosal exposure and autism rates. They make much of the fact that the relative risk for autism due to thimerosal in early studies started out seemingly high and then decreased with successive iterations until it was not significantly different than 1.0, using this observation to claim that such studies were "reworked," presumably as a result of a conspiracy or pressure to "cover up" a "connection." David Kirby also makes this implication in the PowerPoint presentation he includes on his website, in which he shows the Verstraeten study starting out with a relative risk of 7.6, decreasing to 2.6, and then to 1.0 over successive iterations as the study progressed and more subjects were added. However, it is not uncommon for relative risks to start out high (or, in the case of studies of protective agents, low) in early epidemiological studies of various risk factors for or protective factors against diseases and then to trend towards 1.0, as some of the studies in Dr. Ioannidis' article (such as initial studies showing cardioprotective effects of vitamin E or flavonoids) show themselves. It is not necessary to invoke any sort of implausible grand conspiracy between the CDC, WHO, FDA, and IOM to suppress a link between thimerosal and autism to explain the evolution of the data away from indicating a causal link. It's just the way such studies not infrequently play out, as smaller studies find a possible risk factor initially and then later, better designed, larger studies fail to confirm the same risk factor. So it has been with thimerosal and mercury. Early reports and preliminary analyses of studies were suggestive that thimerosal might be a contributory factor to autism, but the latest studies do not bear out the initial concerns, no matter how much the thimerosal-autism activists try to misrepresent, for example, the Danish study that showed no link. Fortunately, as this excellent overview of the controversy mentions, this issue should soon be resolved once and for all scientifically, given that no thimerosal has been in vaccines since January 2003, providing the ultimate epidemiological experiment to confirm the trend of the data for the last 5 years towards no connection. Unfortunately, science is probably irrelevant to the controversy, as Robert Carroll points out:

There is no way to close this issue of mercury and autism. Whatever data is available can always be mined for some gem that supports the conspiratorial theory and there is always hope that some future study will provide some support for the causal belief. No study will ever be able to show with absolute certainty once and for all that thimerosal or any other substance does not cause autism in some people some of the time.

Indeed. As I've pointed out, even if autism rates stay flat or continue to rise in the coming 5-10 years after the removal of thimerosal from childhood vaccines and multiple studies continue to fail to find a link, you can safely bet the farm that the conspiracy theorists will still be blaming it. The same is true of many other putative harmful agents. Science can never definitively prove a negative. It can only say that the overwhelming evidence is that there is no link.

What those not directly involved in research don't always know is that science can be quite messy while in progress. Medical research is no exception. This messiness is especially pronounced in clinical research, where there are so many more confounding variables than in most basic science research and study design is much more difficult to control. It should therefore come as no surprise that studies often disagree; true evidence-based medicine thus relies on the preponderance of evidence from controlled scientific trials evaluated in its totality, a synthesis the Cochrane Collaboration attempts to provide. That is the science of medicine. As physicians, we have to accept sometimes that there is no good data to apply to certain clinical situations and we have to rely on our experience and judgment. Deciding what to do in such situations is the art of medicine.