Why I Decline to do Peer Reviews (part one): Re-reviews

Like pretty much everyone fortunate enough to occupy a faculty position in psychology at a research university, I am frequently asked to review articles submitted for publication to scientific journals. Editors rely heavily on these reviews in making their accept/reject decisions. I know: I’ve been an editor myself, and I experienced first-hand the frustrations in trying to persuade qualified reviewers to help me assess the articles that flowed over my desk in seemingly ever-increasing numbers. So don’t get me wrong: I often do agree to do reviews – around 25 times a year, which is probably neither much above nor below the average for psychologists at my career stage. But sometimes I simply refuse, and let me explain one reason why.

The routine process of peer review is that the editor reads a submitted article, selects 2 or 3 individuals thought to have reasonable expertise in the topic, and asks them for reviews. After some delays due to reviewers’ competing obligations, trips out of town, personal emergencies or – the editor’s true bane – lengthy failures to respond at all, the requisite number of reviews eventually arrive. In a very few cases, the editor reads the reviews, reads the article, and accepts it for publication. In rather more cases, the editor rejects the article. The authors of the remaining articles get a letter inviting them to “revise and resubmit.” In such cases, in theory at least, the reviewers and/or the editor see a promising contribution. Perhaps a different, more informative statistic could be calculated, an omitted relevant article cited, or a theoretical derivation explained more clearly. But the preliminary decision clearly is – or should be – that the research is worth publishing; it could just be reported a bit better.

What happens then? What should happen, in my opinion, is that the author(s) complete their revision, the editor reads it, perhaps refreshing his or her memory by rereading the reviewers’ comments, and then makes a final accept/reject decision. After all, the editor is – presumably and hopefully – at least somewhat cognizant of the topic area and perhaps truly expert. The reviewers were selected for their specific expertise and have already commented on the article, sometimes in great detail. Armed with that, the editor should not find it too difficult to make a final decision.

Too often, this is not what happens. Instead, the editor sends the revised paper out for further review! Usually, this entails sending it to the same individuals who reviewed the article once already. Sometimes – in a surprisingly common practice that every author deeply loathes – the editor also sends the revised article to new reviewers. Everyone then weighs in, looking to make sure their favorite comments were addressed, and making further comments for further revision. The editor reads the reviews and perhaps now makes a final decision, but sometimes not. Yes, the author may be asked to revise and resubmit yet again – and while going back to the old reviewers a third time (plus yet new reviewers) is less likely, it is far from unheard of.

What is the result of this process? What an editor who acts this way would no doubt say is, the article is getting better and the journal is as a result publishing better science. Perhaps. But there are a few other results:

  1. The editor has effectively dodged much of the responsibility for his/her editorial decision. Some editors add up the reviewers’ verdicts as if they were votes; some insist on unanimous positive verdicts from long lists of reviewers; in every case the editor can and often does point to the reviewers – rather than to him or herself – as the source of negative comments or outcomes.
  2. The review process has been extended to be epically long. It is, sadly, not especially unusual for this process of reiteration to take a year or more. Any review process shorter than several months is considered lightning-fast for most journals in psychology.
  3. The reviewers have been given the opportunity to micro-manage the paper. They can and often do demand that new references be inserted (sometimes articles written by the reviewers themselves), theoretical claims be toned down (generally ones the reviewers disagree with), and different statistics be calculated. Reviewers may even insist that whole sections be inserted, removed, or completely rewritten.
  4. (As a result of point 3): The author is driven to, in a phrase we have all heard, “make the reviewers happy.” In an attempt to be published, the author will (a) insert references he/she does not actually think are germane, (b) make theoretical statements different from what he/she actually believes to be correct and (c) take out sections he or she thought was important, add sections he or she thinks are actually irrelevant, and rephrase discussions using another person’s words. The author’s name still goes on the paper, but the reviewers have become, in effect, co-authors. In a final bit of humiliating obsequience, the “anonymous reviewers” may be thanked in a footnote. This expressed gratitude is not always 100% sincere.

These consequences are all bad, but the worst is number 4. A central obligation of every scientist – of every scholar in every field, actually – is to say what one really thinks. (If tenure has a justification, this is it.) And yet the quest to “make the reviewers happy” leads too many authors to say things they don’t completely believe. At best, they are phrasing things differently than they would prefer, or citing a few articles that they don’t really regard as relevant. At worst, they are distorting their article into an incoherent mish-mash co-written by a committee of anonymous reviewers — none of whom came up with the original idea for the research, conducted the study, or is held accountable for whether the article finally published is right or wrong.
So that’s why, on the little box at the bottom of the peer review sheet that asks, “Would you be willing to review a revision of this article?” I check “no.” Please, editors: Evaluate the article that was submitted. If it needs a few minor tweaks, give the author a chance to make them. If it needs more than that, reject it. But don’t drag out the review process to the end of time, and don’t let a panel of reviewers – no matter how brilliant – co-author the article. They should write their own.

11 thoughts on “Why I Decline to do Peer Reviews (part one): Re-reviews

  1. Exactly. Drives me crazy.

    A revise/resubmit decision letter should be more or less a contract: If you can do these things, I will accept your paper. If you can’t, I won’t. The editor should then be able to easily assess whether or not the terms have been met without sending it out for further review. This is how I operate and how I instruct all of our associate editors to operate.

  2. I totally agree. One possible remedy:
    In some fields (e.g., physics or computer science) the scholar also uploads the manuscript on ArXiv. This has one potential benefit.
    They can submit it to a journal, and the revised version (say, after multiple rounds of review) will appear on the journal, but they can keep the ArXiv version to be their personal version, the one in which every word is theirs and no imposition is exerted.
    This of course only works for journals for which the copyright thing is not super-stringent, that is, you won’t be forced to take down your own version from everywhere.

  3. I don’t think it is reasonable to have a rule (e.g., “never send a paper out for more than one round of reviews”) to apply to all manuscripts. The above example of many rounds of reviews (in which the reviewers, and sometime the editor, in essence seek to act as authors) certainly is easy to find fault with. On the other hand, sometimes an initial submission may lay right at the reject/revise border, and the authors may submit such a substantial revision that the fairest thing to do, in my opinion, is to allow others (rather than only the action editor) to weigh in on the overall question of whether the revised paper should be accepted or rejected. The action editor then is free to make a decision informed by those evaluations *without* requiring that the author further revise the paper to address all of the petty grievances/tangential citations that might be raised in the second round of reviews. You might counter that this will upset the reviewers, but this can be done tactfully.

  4. Excellent post! As an author, this drives me nuts because, in the case of sending it to one of the same reviewers again, it weights that unique reviewer’s perspective too heavily. And, in the case of having new reviewers added to the mix, it is comparable to submitting the paper anew.

    From the point of view of a reviewer, I often feel that re-reviews are a drain on the system. Reviewers are a limited limited resource. I allocate time review a paper every 1 to 2 weeks. If one of those slots is eaten up by a re-review, then that is one less “new” paper that I can review.

  5. Thanks David. I agree wholeheartedly with your article. I work in a very different area of science but it is exactly the same. Maybe in the future articles will be accepted if they are interesting or potentially important, but the job of the editor will be to moderate and publish comments which improve understanding – all done online, rather like a blog. Very few of us read paper journals now, and the follow up comments would be available.

    I’ll read some more of your posts when I have time

    Dropsy of wardroundstuff

  6. Right on! I just performed a very detailed review of a manuscript, rushed as always by automatic emails by the journal, only to find the decision letter simply referring to my review and those made by two additional reviewers. The editor never even read the manuscript, he only decided “Major revision” and will surely send it back to us…alas, I read your post too late to check “no” on the appropriate box!

  7. I, too, agree. As a sometime editor at PSPB, I never sent a single paper back to reviewers. There is, however, one exception where I would have, and that is when the review involves new data using methods that were not part of the original review. Such a resubmission deserves peer review. Minor studies, manipulation checks, slight rewording of text does not require re-review, but brand new studies using novel methods require further scrutiny.

  8. Pingback: Why I Decline to do Peer Reviews (part two): Eternally Masked Reviews | funderstorms

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s