The PLoS journals do a lot of things really well. I love that they are open access. And I consistently find interesting, important, and well done studies in them. One of the big disappointments has been that people don’t really use the comment feature, though. In the beginning, the dream was that publication would only represent the beginning of the peer review process. After publication, the entire scientific world could weigh in on the strengths and weaknesses of a given study by using the comment feature.
There are all kinds of reasons why the PLoS comment section remains a wasteland. Scientists are reluctant to offend their peers–writing something critical using their own names may jeopardize their grant applications and their journal submissions. Scientists have a lot to do, and writing comments takes time. Not many people look at the comments, so many scientists probably look at comment writing as a waste of time.
But recently, a scientist named has described what happened when he actually tried to USE the comment feature on the PLoS Biology website.
Exercise in a Pill
A high profile paper came out in PLoS Biology this year: Effects of resveratrol and sIRT1 on PGC-1a activity and biogenesis: A reevaluation. It got covered widely. For example, it featured in a story called Exercise in a Pill in the New York Times. Exciting stuff!
Paul Brookes decided to take a closer look at this interesting paper, and he found some problems that seemed to be pretty significant. For example, it looks like many of the gel bands featured in their western blots may have been recycled between figures. Hmmm. You can find a full description in his blog post on this subject.
He went ahead and posted his observations in the comments attached to the article. Obviously, the problems he pointed out DID seem pretty damning. So what happened then? A prompt response from the authors and the journal? Nope. PLoS decided it violated their terms (apparently, they were concerned that the nature of the problems discussed implicated the authors in research misconduct) and unilaterally took it down. Makes you wonder what the comments section is for, no? He tried to follow up with the PLoS staff for months to get this matter resolved, but there was total radio silence.
Brookes has since published his concerns on PubPeer, and they have been getting some attention on Twitter. But obviously, only a small percentage of readers of the original paper are aware of the concerns Brookes has posted on an entirely different website. Looks like there is still no response from the authors of the original paper. And PLoS Biology appears to have stated that they are “investigating this matter” only AFTER Brookes and other scientists started to raise a stink.
My Own Experience with PLoS Journals
I have actually had my own not-so-nice experiences with comments & corrections in PLoS journals. For example, a colleague found a small mistake in an article I had published in PLoS ONE. Luckily it didn’t impact any of our conclusions (it actually made our findings a little stronger). But having the mistake there could be confusing to readers, so I wrote up a correction and submitted it in July. I got an email from PLoS stating that I should expect the correction process to take a while. OK. But now it is early November, and the correction still hasn’t gone up! I’ve emailed PLoS twice to ask about its status. Nothing, no reply. I guess at this point I should just post the correction in the comments section attached to the article. I think the problem with this solution is that nobody looks at the comments.
My other (even worse) experience was with PLoS Neglected Tropical Diseases. When the journal was just starting out, I published a paper there. It was a fairly controversial paper, and a group of scientists wrote a very critical commentary in the same issue. As it happens, there were some factual inaccuracies in that commentary (for example, the authors got the number of SNPs investigated in our study wrong, saying it was 17 instead of 82, and then claimed to reanalyze our data). When we pointed out these errors, the editor that had handled our article suggested we respond in the Comments section of the article. We pointed out that few people read the comments, so this didn’t seem like a great solution.
After a back and forth that went on for a while, we had a conversation with the editorial staff in which it was agreed that we would bring up these errors in a sort of debate forum. We would respond to the Commentary, citing the errors it contained, the authors would get a chance to respond, then we would respond to their response. Honestly, it seems like factual inaccuracies in an article should be clarified in a formal Correction… but a published “debate” seemed like a better solution than writing a comment that would just be ignored, so we agreed to this solution. After that conversation, we turned in our contribution to the debate. And we never heard from the editors again. Not a word, despite a number of follow-up emails on my part.
PLoS NTDs didn’t publish our contribution, and they didn’t even take the time to respond to repeated inquiries about the status of the debate we had all agreed upon. As Paul Brookes experienced, there was total radio silence. Eventually I gave up. We posted a detailed response in the Comments section of the critical commentary. I doubt anybody looks at it. I suspect that PLoS NTD’s refusal to publish the debate or to even grace us with a justification for their failure to follow through had something to do with the fact that I was a lowly grad student at the time and one of their section editors was involved in writing the critical commentary. I love that PLoS NTDs is drawing attention to much neglected tropical diseases–but after that experience, I would never, ever publish there again.
I don’t think the political problems surrounding comments & criticisms are unique to the PLoS journals. But certainly, these experiences suggest that the PLoS journals have some important work to do. We are all in this together. Mistakes in the scientific record need to be corrected. PLoS doesn’t seem to be facilitating this important process, even though this was initially one of their biggest goals.