A little criticism directed Orac's way

I really have to stop checking Technorati searches on my blog and my Sitemeter referral logs. I really do. If I didn't check them a couple of times a week, then I would probably never come across posts mentioning me that raise my blood pressure like this one (unless, of course, readers alert me to them). Or maybe I should just learn to ignore such posts, rather than responding and risking provoking a blog flame fest. Too bad last night that I was in just the right mood to pay attention to one post in particular that mentioned me and feeling just cranky enough to lay down a little Respectful Insolence about it.

For a blogger who says he works in the pharmaceutical industry and doesn't like Kevin Trudeau (although I would disagree with his qualification of his dislike in which he said that there's "a lot to Kevin Trudeau that's admirable"--I consider Trudeau to be a scam artist), a blogger going by the 'nym Random John sure doesn't seem to understand one of my major recurring points. He starts here:

Posts like this simply make my blood boil. Perhaps I should keep my “buttons on the inside” (i.e. not let other people push my buttons), but I get frustrated when doctors get their priorities all out of whack and misinterpret the results of experiments.

That's two of us who should perhaps keep our "buttons on the inside," but I get frustrated when other bloggers make big, fat straw men out of what I say and then angrily (or happily--or both) tear down said straw men. The post of mine to which he was referring is here. In it I pointed out how easy it is for doctors to delude themselves into believing a treatment works on the basis of anecdotal observations when it may not and why evidence-based medicine as exemplified by well-designed clinical trials can help avoid that trap and allow doctors to confirm or reject early anecdotal observations. John took issue:

Statistics measure population tendencies, and are bad at predicting individual results. I ought to make every doctor on the face of this planet write this on a chalkboard until they say it in their dreams. Maybe it should be a semester course in medical school and biostatistics. A three-hour course required for graduation. Three hours a week of writing this on the board, supervised, and a recorder stuck under their mouths at night for verification. To pass the class, they have to be saying this in their sleep.

Orac seems to take the most extreme example of the alternative-medicine-is-good-and-conventional-medicine-is-bad nutcase (and, the person who fits the description he gives on the page is a nutcase, I’ll give him that), and equate people giving testimonials.

Geez, I wonder what I said that set him off so much.

Uh, no, that's not what I said or meant. My piece on the altie to which you refer was humor that was indeed directed at the most extreme example, but it did not say that anecdotes are without value. What it did say that is relevant is this:

If you accept vague and/or poorly documented anecdotes and testimonials as sufficient evidence that an "alternative" therapy "works," you just might be an altie.

How is that unreasonable? And, yes, John, there are a fair number people like that. I've encountered them on misc.health.alternative. They inspired the piece, to which many old regulars of m.h.a. contributed, whether they knew it or not. Another relevant excerpt:

If you dismiss every well-designed randomized clinical study that failed to show a benefit for an alternative medicine or therapy over placebo control as either not proving that the therapy is ineffective or as having been manipulated by nefarious forces (conventional medicine, the pharmaceutical companies, the government, etc.) to produce a negative result, you may well be an altie.

Hmmm. Note the words. "Vague and/or poorly documented anecdotes and testimonials." "Dismiss every well-designed randomized clinical study." (Emphasis mine.) I was making fun of people who buy vague and/or poorly documented testimonials for their favorite cure du jour while at the same time automatically rejecting well-designed studies either because they were done by the hated conventional medical community or because don't show what they want them to show. But John isn't finished:

Let’s look at this argument then: clinical trials good, testimonials bad. Do you see the bias here? No?

Uh, no, John, it's not "bias" to understand that testimonials are inherently less reliable than clinical trials in identifying which treatments work and which do not and then putting them into their correct place in the hierarchy of medical evidence. It's just good science. And, no, John, I did not say that anecdotes are "bad." I merely pointed out that they are far less likely to be widely applicable than the results of clinical trials. I've explained multiple times why most testimonials don't even meet the minimum standard of medical evidence, and why personal observation alone is prone to many more biases, including selective thinking and confirmation bias. Finally, contrary to John's characterization of what I said, I have not entirely dismissed anecdotal evidence. Actually I have said repeatedly, that such evidence, if better documented than just a testimonial, can be useful to guide further research. Consider, first, what I said about the difference between testimonials and anecdotes:

The other problem with testimonials is that they don't rise even to the lowest level of medical evidence, the anecdotal report. Anecdotal reports in medicine require a careful documentation of symptoms, lab tests, diagnoses, exact courses of treatment, and a patient's response to treatment. Testimonials almost never present these elements in sufficient detail to judge whether the treatment actually did anything. There's just no way of telling truth from exaggeration or fiction.

I reiterated the faults of testimonial-based "claims" here:

So what's wrong with testimonials? Well, as I like to say, the plural of "anecdote" is not "data," and testimonials usually don't even rise to the level of anecdotes. Testimonials are often highly subjective, and, of course, practitioners can and do pick which testimonials they present. Even in the case of cancer "cure rates," testimonials often mean little because they are given for diseases that surgery alone "cured." (Also, dead patients don't provide good testimonials.) Worse, testimonial-based practice tends to preclude the detailed observation and long-term followup necessary to identify which patients benefit from treatments and which do not, complication types and rates, or long-term results of the treatment. Anecdotes are really good for only one thing, and that's developing hypotheses to test with basic scientific experimentation and then clinical trials.

Note the last sentence. That's me pointing out the proper weight that should be given to anecdotal evidence, which is not zero but is much less than the weight that should be given to a well-designed clinical trial. I suspect the problem here is that John seems to be equating testimonials and anecdotal data. Given that John works for the pharmaceutical industry, I would have thought that he would know the difference (namely that testimonials are usually poorly documented or not at all and are usually primarily intended to sell a product), but apparently not. In any case, what I said was not all that different from what John himself said, except for my emphasis on distinguishing dubious, highly subjective testimonials from the more objectively observed medical anecdote:

The sparks for ideas that lead into scientific revolutions come from odd, anecdotal observations that are outside of what statistics predicts. Individuals give important information that statistics will miss, and cutting out anecdotal or testimonial information and relying solely on clinical trials for our research is like cutting off our legs because the car gets us there faster.

John's also constructing and attacking another huge straw man here. I never advocated "cutting out" anecdotal information (although I definitely do advocate cutting out testimonials, for the reasons I described in detail here, here, and here). I merely pointed out that anecdotes are usually pretty weak evidence. John did have a point in mentioning that clinical trials may not adequately predict the response of any single individual to treatment. That does not invalidate my point, however, because anecdotes are considerably worse than clinical trials at such prediction. In addition, it isn't more anecdotes that will help us predict more accurately the response of any individual to any given therapy. It will be clinical trials that identify factors that might help us predict which individual patients will respond better to which treatment. Indeed, that's the whole point of genomic medicine and molecularly targeted therapeutics like Herceptin.

Now that I think about it, it's rather tempting to build up a straw man of my very own and take John to task for recommending that we abandon clinical trials as medical evidence in favor of anecdotes and testimonials. (Insert much triumphant ranting and raving about said straw man here.)

If I were to do so, though, I just hope it would be a real straw man and not the truth.