Frontier science versus textbook science
How then can the fraudulent claims by Dr. Hwang Woo Suk have been accepted by Science, a leading journal that rejects most papers submitted to it? How can the community of stem-cell scientists have allowed a very visible claim to have stood unchallenged in their field for 20 months? Little wonder that Richard Doerflinger, an official of the United States Conference of Catholic Bishops, ridiculed the dreams of therapeutic cloning in a statement last week, scoffing that scientists were chasing miracle cures "in pursuit of this mirage."
The contrast between the fallibility of Dr. Hwang's claims and the general solidity of scientific knowledge arises from the existence of two kinds of science - a distinction that is often blurred when new advances are reported first by scientific journals and then by the news media. There is textbook science and frontier science, and the two types carry quite different expiration dates.
Textbook science is material that has stood the test of time and can be largely relied upon. It may include findings made just a few years ago, but which have been reasonably well confirmed by other laboratories.
Mr. Wade makes the point that research such as the kind that Dr. Hwang does is what he characterizes as "frontier" science; that is, science at the very edge of what is known or possible and warns against overreacting:
Science from the frontiers of knowledge, on the other hand, is wild, untamed and often either wrong or irrelevant to future research. A few years after they are published, most scientific papers are never cited again.
Scientific journals try to impose order on the turbulent flow of new claims by having expert reviewers assess their merit. But even at the best journals, reviewers provide only a rough screen. Many papers slip through that later turn out to be innocently wrong. A few, like Dr. Hwang's, are found to be fraudulent.
This rough screening serves a purpose. Tightening it up, in a vain attempt to produce instant textbook science, could retard the pace of scientific advance.
This leads to a paradox. The science that is getting published in the highest profile, most prestigious journals is almost by definition the most tentative science. Given that, it is surprising how much of what is published in such journals actually does stand the test of time, but it should not be surprising that much of it does not. However, the very prestige of such journals gives such research seemingly more authority than research published in less prestigious journals. It is often said that one Nature, Science, or Cell paper is worth five or even ten papers in more pedestrian, middle-of-the-road journals as far as improving a scientist's CV (and chance of a good job or promotion) goes. Perhaps that is because publications in such journals are viewed as an indication that the work a scientist is doing is on the cutting edge. That perception, built up over time, is likely the major reason that it is very, very difficult to get a paper accepted and published in Science, Nature, or Cell. The vast majority of submissions are rejected, many without even being sent out for peer review because an editorial decision is made that they are not "interesting" enough (something that happened to me once). However, scientists understand that papers published in the most cutting edge journals are tentative. They're interested in the papers because such work is the most likely to advance the frontiers of science, but they also know that the papers have a higher than average probability of being wrong, either in part or in whole, or a dead end. Wade nails it when he writes:
I would also point out that, because of the imprimatur of Science, many scientists and physicians, myself included, considered Dr. Hwang's results to be major breakthroughs. Of course, part of this could be due to wish fulfillment, given the promise of fantastic new treatments for a variety of diseases that Dr. Hwang's results and new technique seemed to offer, but that's exactly the sort of situation when we as scientists should really be the most skeptical.But the roughness of the proceedings is not prominently advertised by journal editors, except when cases of blatant fraud are detected, whereupon they proclaim that peer review cannot reasonably be expected to detect fraud. They do not protest so much when newspapers report their journals' claims as if they were certifiably true. Because of Science's authority, Dr. Hwang's claims to have cloned human embryonic cells were prominently reported and presented to the public as if they were important breakthroughs.
Many ideas for reforming peer review have been floated, but in reality I doubt that any of them would catch a determined fraud. Science and peer review inherently depend upon trust that the investigator presenting his data for publication has not fabricated it. The only real way to detect fraud would be to put such an onerous burden on peer reviewers that it would make finding qualified scientists willing to do be peer reviewers difficult unless they were paid. It would require seeing the raw data, and anyone who has done research knows just how hard it is to go through another's scientist's laboratory notebook to evaluate the raw data. One proposal, however, for reforming publication procedures and peer review that might actually help somewhat is this:
Medical journals, including JAMA and several surgical journals, have been doing just this for a while now, with no undue burden or generation of strife. It may not prevent fraud, but it definitely makes one feel accountable as an author. I can say from personal experience that, when I sign off on one of those statements for a paper that I am a co-author on, I want to make damned sure that I have read the manuscript in its entirety carefully and that I do indeed agree with it, at least in general.But last week Dr. Kennedy announced he was considering revising the journal's publication procedures, though not with any great hope of preventing future cases of fraud. He suggested that authors would be required to state in writing their specific contributions to a report, a reform perhaps aimed at Dr. Gerald Schatten of the University of Pittsburgh. Dr. Schatten accepted senior authorship of - and thus responsibility for - one of Dr. Hwang's papers, even though Dr. Schatten had performed none of the experiments and was not in a position to vouch for them. All the work was done in Seoul.
A second proposed change is to have all authors state that they agree with an article's conclusions.
Both procedures may seem to include a certain potential for generating strife. Each author could overstate his or her contribution, arousing the wrath of all the others. Some authors may think a conclusion too timid, while others consider it an overstatement.
As Wade points out:
Tightening up the reviewing system may remove some faults but will not erase the inescapable gap between textbook science and frontier science. A more effective protection against being surprised by the likes of Dr. Hwang might be for journalists to recognize that journals like Science and Nature do not, and cannot, publish scientific truths. They publish roughly screened scientific claims, which may or may not turn out to be true.