A little criticism directed Orac's way
For a blogger who says he works in the pharmaceutical industry and doesn't like Kevin Trudeau (although I would disagree with his qualification of his dislike in which he said that there's "a lot to Kevin Trudeau that's admirable"--I consider Trudeau to be a scam artist), a blogger going by the 'nym Random John sure doesn't seem to understand one of my major recurring points. He starts here:
Posts like this simply make my blood boil. Perhaps I should keep my “buttons on the inside” (i.e. not let other people push my buttons), but I get frustrated when doctors get their priorities all out of whack and misinterpret the results of experiments.
Statistics measure population tendencies, and are bad at predicting individual results. I ought to make every doctor on the face of this planet write this on a chalkboard until they say it in their dreams. Maybe it should be a semester course in medical school and biostatistics. A three-hour course required for graduation. Three hours a week of writing this on the board, supervised, and a recorder stuck under their mouths at night for verification. To pass the class, they have to be saying this in their sleep.
Orac seems to take the most extreme example of the alternative-medicine-is-good-and-conventional-medicine-is-bad nutcase (and, the person who fits the description he gives on the page is a nutcase, I’ll give him that), and equate people giving testimonials.
Uh, no, that's not what I said or meant. My piece on the altie to which you refer was humor that was indeed directed at the most extreme example, but it did not say that anecdotes are without value. What it did say that is relevant is this:
How is that unreasonable? And, yes, John, there are a fair number people like that. I've encountered them on misc.health.alternative. They inspired the piece, to which many old regulars of m.h.a. contributed, whether they knew it or not. Another relevant excerpt:If you accept vague and/or poorly documented anecdotes and testimonials as sufficient evidence that an "alternative" therapy "works," you just might be an altie.
Hmmm. Note the words. "Vague and/or poorly documented anecdotes and testimonials." "Dismiss every well-designed randomized clinical study." (Emphasis mine.) I was making fun of people who buy vague and/or poorly documented testimonials for their favorite cure du jour while at the same time automatically rejecting well-designed studies either because they were done by the hated conventional medical community or because don't show what they want them to show. But John isn't finished:If you dismiss every well-designed randomized clinical study that failed to show a benefit for an alternative medicine or therapy over placebo control as either not proving that the therapy is ineffective or as having been manipulated by nefarious forces (conventional medicine, the pharmaceutical companies, the government, etc.) to produce a negative result, you may well be an altie.
Let’s look at this argument then: clinical trials good, testimonials bad. Do you see the bias here? No?
The other problem with testimonials is that they don't rise even to the lowest level of medical evidence, the anecdotal report. Anecdotal reports in medicine require a careful documentation of symptoms, lab tests, diagnoses, exact courses of treatment, and a patient's response to treatment. Testimonials almost never present these elements in sufficient detail to judge whether the treatment actually did anything. There's just no way of telling truth from exaggeration or fiction.
Note the last sentence. That's me pointing out the proper weight that should be given to anecdotal evidence, which is not zero but is much less than the weight that should be given to a well-designed clinical trial. I suspect the problem here is that John seems to be equating testimonials and anecdotal data. Given that John works for the pharmaceutical industry, I would have thought that he would know the difference (namely that testimonials are usually poorly documented or not at all and are usually primarily intended to sell a product), but apparently not. In any case, what I said was not all that different from what John himself said, except for my emphasis on distinguishing dubious, highly subjective testimonials from the more objectively observed medical anecdote:So what's wrong with testimonials? Well, as I like to say, the plural of "anecdote" is not "data," and testimonials usually don't even rise to the level of anecdotes. Testimonials are often highly subjective, and, of course, practitioners can and do pick which testimonials they present. Even in the case of cancer "cure rates," testimonials often mean little because they are given for diseases that surgery alone "cured." (Also, dead patients don't provide good testimonials.) Worse, testimonial-based practice tends to preclude the detailed observation and long-term followup necessary to identify which patients benefit from treatments and which do not, complication types and rates, or long-term results of the treatment. Anecdotes are really good for only one thing, and that's developing hypotheses to test with basic scientific experimentation and then clinical trials.
The sparks for ideas that lead into scientific revolutions come from odd, anecdotal observations that are outside of what statistics predicts. Individuals give important information that statistics will miss, and cutting out anecdotal or testimonial information and relying solely on clinical trials for our research is like cutting off our legs because the car gets us there faster.
Now that I think about it, it's rather tempting to build up a straw man of my very own and take John to task for recommending that we abandon clinical trials as medical evidence in favor of anecdotes and testimonials. (Insert much triumphant ranting and raving about said straw man here.)
If I were to do so, though, I just hope it would be a real straw man and not the truth.