Peer review – a system that needs revision
Publishing is a fundamental part of science and the main goal of researchers globally. The crucial part of this process to ensure credibility and scrutinise the claims is peer review.
Peer review makes science stronger and more objective since other perspectives in the field are incorporated. There is a reason that thoroughly peer reviewed articles are more credible, and most researchers agree that this extra time makes the publication better. As researchers we can get very attached to our work and other perspectives will often highlight improvements which makes the work stronger overall. However, this process can drag on and adds an unpredictable amount of time until publication.
In theory this process is sound. However some concerns about the process are worth examining. But first, a quick guide to peer review.
From manuscript to published article
Briefly, after submitting the manuscript to a journal the editor decides whether to reject it or to proceed to peer review. An immediate rejection like this is not just about quality but also about suitability for the specific journal. If this passes the manuscript is sent anonymously to some (typically 2-3) fellow researchers in the field who have relevant experience. You have the option to suggest reviewers for the study or to suggests reviewers that should be excluded (such as competitors).
What follows is a discussion and rebuttal where the anonymous peer reviewers comment on the manuscript. This means discussing its strengths and weaknesses and possibly suggest additional data that could strengthen it. They also suggest on whether to accept it or reject it. The editor then takes these suggestions from the reviewers and decides whether to reject or accept it provided they address the reviewer’s suggestion. This discussion goes back and forth until the peer reviewers are satisfied. The editor of the journal who is not necessarily an expert in the field that may step in as a final arbiter if there is disagreement between authors and editors.
This whole process may take 2 weeks or up to 2 years depending on many factors. The length of the discussion, the new data requested, how long it takes to find peer reviewers, and the responsiveness of the journal.
Note that this is a generalised summary and there are many styles of peer review with different levels of anonymity. Read this summary if you want to know more.
Challenges with Peer review
Time delays: Each of these steps take time. The discussion and rebuttal between reviewers and authors need time to develop, especially when complementary experiments are needed. However there are some delays that are entirely in the hands of publishers. In some cases, it takes many months to find good peer reviewers with relevant expertise. This whole process of peer review, from initial submission to that sweet, juicy publication can take anytime from 1 week to a year. Every step adds time and uncertainty. Moreover the number of submissions journals receive can vary wildly during the year so the demand for peer reviewers may exceed the supply. This time delay is quite a concern for researchers who often need papers out by a deadline for grant applications or in time for a PhD/Postdoc to leave.
Ambiguous answers: The peer review process works well when submissions get a clear acceptance with little extra work needed, or a clear rejection. Either way they get a certain informative answer without a time investment. However sometimes reviewers give answers in the middle where they accept it but with major criticisms that ask for a large amount of work in a small-time window. This adds a lot of time pressure to plan and perform experiments to quickly address the reviewers’ comments. Then the manuscript is hopefully accepted but in the worst cases after all this crunch time you may resubmit and then get a heartbreaking rejection.
Untrained Peer reviewers: Peer reviewing is a difficult balance, and a very specific type of constructive feedback is needed. The best feedback is brief, constructive and shows the weaknesses in the argument and how to quickly improve them. Remember, there is limited time and money to work on the manuscript, so suggesting an experiment that takes more than 3 months just isn’t feasible. A three-page criticism of grammar and word choice is not especially helpful, and neither is a diatribe on how this has no clinical purpose. Worst of all are the peer reviewers who simply don’t understand the text.  However, there is very little training on how to do peer review and is most often learnt through peers.
I think this is a reason why the peer review comments are frequently complained about. Peer reviewers aren’t formally trained therefore the feedback isn’t standardised and there is a misunderstanding from the authors and peer reviewers on what exactly their role is. So peer reviewers may feel unforgiving, cold, unconstructive strangers rather than helpful peers just like this article shows. This was explored in a humorous fashion here.
Conflict between authors and reviewers: There are also cases where there is big disagreement and heated online discussions between authors and reviewers. Usually, the editor steps in and can resolve the situation. There have been high profile examples of rejected papers that eventually led to Nobel prizes such as Kary Mullis PCR paper in 1993, Hans Krebs’ paper on the citric acid cycle in 1937, and Enrico Fermi’s paper on weak interaction in 1993. Here is a collection of some funny examples. Imagine if some of the peer review process was done in person? Now that would make for some good TV (for a specific audience)
Luck: The reality is that there is an element of luck on your chances of publishing depending on which reviewers are chosen. Some reviewers will be overly critical or have professional bias or ask for some unreasonable changes or arbitrarily disagree with your work. On the other hand, one reviewer may be super enthusiastic about your work and accept it immediately. This adds an element of luck to publishing. Chance plays a part in all aspects of life, so it’s not a huge shock that science is affected. But it is worth noting.
Peer review is unpaid work for reviewers: From the reviewer’s perspective, peer review is not just a 30-minute proofread of a text. It is a in depth discussion that takes many hours over the course of several month up to an entire year. While reviewing papers can be seen as an act of collegiality that benefits the research field and science at large, publishing companies benefit financially from this largely unpaid work. This expectation to do extra unpaid work is a pervasive problem in academia.
Predatory and semi predatory journals: These are journals that have high publishing fees but provide little to no peer review. Some of these are more obvious to spot as email spam. However some journals blur the line between reputable and predatory. For example, the peer review process may be unreasonably sped up with reviewers expected to say yes to every manuscript given to them, or the barrier for acceptance is made very low, or the only way for reviewers to suggest a rejection is to withdraw from the conversation completely. Therefore, journals advertise a peer review system but in reality, they charge high prices and don’t deliver on the promise. There is no global regulation on these journals so some deeper research into each journal and researchers experience of them is sometimes needed.
Pre submission peer review: Systems like review commons allow for a smoother peer review process. In this system the peer review comments are done before submitting to a journal. This way, the authors get a quick indication to judge journal suitability and impact, before committing to a journal, then ensures a smoother and quicker process once submitted. This can be especially useful for highly multidisciplinary projects where it is not clear which journal is the best.
Summary – peer review benefits science, but the system needs revision
References
https://www.sciencealert.com/these-8-papers-were-rejected-before-going-on-to-win-the-nobel-prize
https://www.nature.com/articles/d41586-020-00381-1
https://www.nature.com/articles/d41586-023-00403-8
https://sitn.hms.harvard.edu/flash/2022/peer-review-in-science-the-pains-and-problems/
https://www.sciencedirect.com/science/article/pii/S0164121200000200
https://www.degruyter.com/document/doi/10.1515/ip-2020-5001/html?lang=en. https://shitmyreviewerssay.tumblr.com/
0 comments