Apr 16, 2012

peer pressure

Dr. Casadevall, now editor in chief of the journal mBio, said he feared that science had turned into a winner-take-all game with perverse incentives that lead scientists to cut corners and, in some cases, commit acts of misconduct.

Carl Zimmer writes in the Science column in the New York Times that there is a rise in misconduct in the sciences; a ten-fold increase in journal retractions over the last decade. This is very troublesome. He outlines some of the reasons based on findings from some who have researched the phenomena.
  1. Biomedical research where the bulk of the issue lies, consumes a larger and larger amount of government funding.
  2. Researchers feel a need to publish many papers and in prestigious journals (I don't see how this has changed much so I don't think it's a driver). There is a direct link between how 'important' a paper is (as measured by the number of citations) and the likelihood of retraction. The worst offender was The New England Journal of Medicine.
  3. In 1973, more than half of biologists had a tenure-track job within six years of getting a Ph.D. By 2006 the figure was down to 15 percent. In other words a high profile paper is a requirement for 'admission'.
  4. Promotions are driven by number of papers written, quality of journal the papers is printed in, and the amount of funding a researcher can claim.  Not the quality of the research.
  5. More and more researchers are coming out of India and China creating more competition. And those countries offer cash rewards for papers in prestigious journals.
  6. And perhaps the online availability of papers is leading more faulty research to be rooted out.
I don't know that this answers the question of why the increase in retractions. Most of this was true when I was a grad student considering becoming a professor. Quantity of publications and the quality of the journal we topics of discussions many times. I was lucky in that my professor agreed that my work warranted two complete and broad papers rather than a bunch of smaller ones stretched out. I simply refused to cut up the work because it didn't make any sense.

The other subcontext to this article is that it only talks about MEDICAL research. Sean Carroll has already come out and said that he doesn't think this is a trend in physics. And I believe him. Having read lots of medical research the quality of papers is simply mindbogglingly atrocious. It is almost universally bad. Including stuff coming from the top universities. Most of this is due to the reliance on observational studies over clinical studies. Clinical studies are hard and expensive. New observational research can be culled from age old studies with a computer and some ideas.  In sort the research techniques available, suck. That's why I don't believe any medical research unless I read that specific article and look around for criticisms.

But in general the bigger problem is the methodology of journal peer review. You write your paper and the journal sends it to 3 researchers in the same field. It's a system with the potential for gaming and more likely laziness. We need a Wikipedia or research where everyone can see and comment and criticize papers before they are sanctioned by a journal. And maybe even sanctioning of papers is not really necessary at all.

No comments: