We Can Fix Peer Review Now

Imagine a software company that solicits user feedback with, “Please let us know what does and does not work in the current release and what you would like to see in the future. However, keep in mind that we will not be making any updates to our products and the version you have is the final one.” This is the state of post-publication peer review today. We ask scientists to comment on static, final, published versions of papers, with virtually no potential to improve the articles. We ask scientists to waste their time and then take the lack of participation as evidence against post-publication peer review. For two years now, I have heard the argument that efforts to encourage post-publication commentary have failed and therefore cannot succeed in the future. This is the classic “has not worked so far and therefore never will” mentality (just as people tell me that lack of mobile devices in the lab right now is proof that scientists will not use phones and tablets for research in the future). Proponents of post-publication versus pre-publication review are still viewed as the crazy fringe that is about to derail all that is good about science publishing. Much has been written on the failure of the current publishing model in science ((Peer review is f***ed up – let’s fix it)) ((Stop deifying "peer review" of journal publications)) ((The Seer of Science Publishing)) ((End the wasteful tyranny of reviewer experiments)) ((Is peer review broken?)) ((I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals)) ((The Cost Of The Rejection-Resubmission Cycle)) ((The gift that keeps on giving)) ((The magical results of reviewer experiments)) ((Dear Academia, I loved you, but I’m leaving you. This relationship is hurting me.)). I want to focus here on the ways to incentivize post-publication peer review, and specifically, on ways to incentivize constructive criticism. By far, the best demonstration of the power and potential of post-pub review is the PubPeer website. Not only did PubPeer succeed where most journals failed – encouraging comments after publication – but the comments on their site have led to a number of high profile retractions. PubPeer is a clear demonstration of the power to catch problems that pre-publication peer review is simply incapable of flagging. A common criticism of PubPeer is that the comments are overwhelmingly negative. However, as should be clear from the above, this is not a problem of PubPeer; this is the fault of our publishing structure. Scientists are already sleep-deprived and overwhelmed with workload. Why spend time commenting on a paper if the paper won’t improve? Naturally, the comments are those that are likely to lead to a retraction or a major correction, as these are effectively the only actions that can still be applied to a published manuscript. The good news is that the solutions to this are already live. The journal F1000Research has broken ground with support for versions of manuscripts. Authors can return after publication and edit their papers with clearly-tracked and stamped versions (example here). This is a big deal. I have gotten countless e-mails after publishing my papers with important references that I missed and with great questions that suggested easy clarifications to the manuscripts to strengthen them. If only I could easily edit and improve my papers! As more publishers enable versioning, the incentive to provide constructive rather than destructive feedback on PubPeer will increase exponentially. Luckily, you don’t have to wait a decade or two for other publishers to catch up to F1000 and enable versioning. Just submit your manuscript to a preprint server like bioRxiv or arXiv before you send it to a journal, and then solicit reviews from scientists and encourage them to publish these on PubPeer. Both bioRxiv and arXiv have versioning, and you can continue to improve your paper even after it is published in a traditional journal. There are many great reasons to deposit to a pre-print server, but even if you don’t, or for papers you have previously published, you can easily contribute to productive and constructive post-publication commentary. We are constantly answering questions about our publications at seminars, conferences, and via e-mail. These discussions are so helpful if made public. By e-mail, you answer to one person, but on PubPeer, you answer 1,000. You can quickly make an FAQ section on your paper based on the questions you commonly get, or you can copy-paste entire e-mail threads (I have done just that on my recent publication). Ideally, in the future, these discussions will happen directly on PubPeer instead of privately by e-mail. Finally, there are thousands of journal clubs happening each week with deep and careful discussions of papers. This is post-publication peer review! You spend hours preparing to present the paper. You have concerns, questions, and positive feedback. Why not share it openly or anonymously on PubPeer? After all, PubPeer calls itself the “online journal club”. PubPeer engages the authors for you so you can get clarifications and additional information. You can help other scholars interested in this work. You can help the authors to improve the understanding of their work, and if they published on a pre-print server or with a journal that has versions, possibly help the authors to improve the manuscript itself. There is no reason to wait for publishers to innovate. With the exception of a few, innovation is neither the forte nor the goal of the publishers. As scientists, with just a few minutes of our time, we can contribute to the online annotation and discussion of published research already. We can push for constructive post-publication discussions and peer review as authors and readers. The tools are at our disposal. Let’s use them. Let’s elevate the tone of the commentary and let’s comment on the vast majority of papers that are good and not headed for Retraction Watch. If we make an effort as scientists now, we will validate the post-publication peer review naturally and will lead to a healthier scientific publishing and discourse. ----------------------------- [Because I am a fan of  making frequently asked questions public, below are the common motifs in defending the status quo of peer review and publishing.]

•  Pre-pub peer review improves papers

This seems obvious on the surface. Certainly all of my manuscripts improved thanks to peer review. But by how much? And at what cost (see here  and here)? Is the 9-month average delay helping or hurting science? I am a strong advocate for academic peer review. I don't know anyone who argues against improving papers and quality of science through review. But why pre-publication? I think the current system does more harm than good to the quality of science. Just consider the fact that the paper does not see the light of day until the reviewers and the editor have been satisfied. This is a ludicrous level of pressure; pressure not to improve but to provide the desired results. Not only does this contribute to outright fraud, but as Arjun Raj points out, this constantly leads to inadvertent bad science. And any good from pre-pub peer review will still be there with post-publication peer review. In fact, papers will improve more rapidly and will gain more reviews with the post-pub structure with versions. We’ll have higher quality manuscripts, faster, with fewer retractions, and fewer dubious results.

• The current system is stretched and has weaknesses, but is it really broken? Good science still gets published.

It’s broken beyond repair. The average 9-month delay from the moment of submission to the publication is inexcusable with our current tools. We are publishing the same way Gregor Mendel did, despite the advent of computers, internet, social networks, and mobile devices. Good science gets published despite, not because of the current system. The current publishing structure is pushing people out of science. It is demoralizing, exhausting, and destructive.

Professor at University of Washington, in response to the invitation to share a story about published research on our PubChase Essays: “Thanks for the invitation to share a story [about my research].  I’ll see if I can come up with one, but to be honest, publishing has become such a war that once a paper is out, I think I try to forget the details of what went on as soon as I can.”

Professor at Brandeis, also in response to the story request: “Not sure what dirt I want to dish about my papers. I could tell how it took 6 tries to publish our recent manuscript or how other of our most frequently cited papers were rejected from various journals.” Professor Arjun Raj: “For myself, I can say that by the time a paper comes out, I usually never want to see it again. The process just takes so long and is so painful that all the joy has long since been squeezed out of the paper itself.”

• Pre-pub is a filter so we don’t have to read crap. Too much is published already.

It’s a bad filter. It approves bad papers and rejects good ones. Yes, the volume of publications is overwhelming. Over 100,000 papers are deposited into PubMed each month. The solution isn't to reject more. That just leads to delays. Most rejected papers are still published, just with a delay. Rejections are often random.  And how much more would we have to reject to make the information flow manageable? With internet and today’s technology, we now have better filtering than 300-400 years ago. The solution is to improve tools like PubChase that solve the problem of discovery via personalized recommendations. And, of course, post-pub review can serve as the same filter, only faster and better.