The Dysfunctional Incentives of Academic Peer Review

Academic peer review is a highly dysfunctional process, replete with perverse incentives and maddeningly Kafkaesque outcomes. We chase after peer reviewed outlets nonetheless because they are also the closest thing that academia has to a vetting mechanism for research productivity, and thus promotion. This creates a number of incentives toward increasing the volume of output, including papers on obscure topics that are neither read nor cited in any meaningful way. Note that it is possible to reach this conclusion and yet hold that the existence of peer review is still superior to no peer review at all. But peer review has many problems.

The Case of the Misbehaving Referee:

To illustrate, let’s consider a hypothetical scenario in which a well-regarded journal solicits reports from two referees.

Referee A is a lazy procrastinator who has expertise in a topic but also strong and opinionated views on the same. He waits until the last minute to even read the paper – some 6 months after he receives it – and does a sloppy and desultory job. He doesn’t substantively engage with the assigned paper’s content and even misrepresents some of its points through simple inattention and carelessness. When making a recommendation at the end of his report, he instead defaults to his own prior opinions about the subject of the paper. Those opinions are strong and precede his analysis. He therefore recommends rejection.

Referee B is a more conscientious and thoughtful reader. She too is an expert in the topic, but is more open to new and innovative angles of looking at familiar problems. She’s impressed by the manuscript and spends several hours reading it, writing up thoughtful and constructive comments. Her comments indicate the paper is good and should eventually be published – indeed, she thinks it would be a tragedy to reject it and she offers multiple detailed suggestions that are intended to polish it up into final form. She recommends modest Revise & Resubmit with an inclination to publish.

The editor of the journal in this scenario is strapped for time, as most editors tend to be. The journal in question is also high ranked and receives far more submissions than it can ever hope to print. It may reject 95 out of every 100 submissions it receives, and view that as a point of pride – its reputation for exclusivity. Yet in order to manage these considerations from sheer volume alone, the editor must also take steps to reduce his time commitments to each piece. He therefore takes on a largely deferential and sometimes absentee approach. He uses referee reports to screen and reduce the strain of volume on the journal’s resources. For perfectly rational reasons of time management, the editor adopts a default position of rejecting any manuscript unless the referees reach a consensus and both decide to advance it to publication.

Since this hypothetical article only has one favorable report and one hostile – albeit lazy – report, it automatically gets moved into the reject box.

But recall that in our scenario, not all referees are giving the article an equal amount of attention or weight. Recall that Referee A, the lazy guy, recommended a rejection on grounds that had more to do with his own priors than the manuscript’s merits. He clearly turned in an inferior product to the editor. Referee B, the conscientious reader who offered detailed and thoughtful comments, not only saw promise in the piece – she took time to improve it toward a better scholarly product. But her views were dismissed nonetheless, on account of a built-in bias toward the other referee’s veto.

In short, the rejection of this piece does not reasonably reflect upon anything about its academic merits – it is strictly a byproduct of a lazy reviewer doing a very poor job, and a journal editing convention designed to reduce the stack of submissions that must be considered at length.

Editing Incentive Structures:

What effectively emerges in this sort of a referee process is a veto-by-laziness. Or any number of other reasons that could produce bad behavior by the referee, be it willful mendacity, laziness, incompetence, craziness, or some combination thereof.

Now thoughtful editors might exercise a little discretion at several points in the process to rein in this type of referee. They could discount the weight assigned to a suspect reader’s report. They might choose not to use a referee who develops a reputation for producing lazy and mistaken results. They could override a bad referee report and instruct the author to ignore it. They might even send the report back to the referee, explaining its deficiency and asking for a higher quality product.

And yet the incentives of editing a journal are probably aligned against all of these things, all else equal. Time is scarce, and editors of prestigious journals tend to be accomplished scholars with heavy research expectations of their own. Ask any one of them and they will usually tell you it’s a thankless task. Unfortunately, that often means the path of least resistance to clearing out a queue of submissions to a high-prestige journal is to default to the rejection pattern described above. Unless a piece garners unanimous support in the first round, it becomes a pass – even if one or more of the referees engages in what any sensible person would independently recognize as bad behavior. Or sloppiness. Or dereliction of duty in the review process.

The problem that emerges is fundamentally a matter of incentives, and specifically the fact that the misbehaving referee has almost no social incentive to improve his or her behavior. Yes, that referee might not be asked to review for the same journal again. But that’s about the worst thing that could happen to him/her, and since refereeing is a largely thankless task with no compensation and very little external prestige, being dropped by the editor over poor performance is hardly a penalty – in fact, some academics might even desire the relief of not being bothered again!

There’s also very little visible cost suffered by the editor (although there might be unseen costs, especially if a journal passes on a piece that later becomes an important and widely cited article elsewhere – here’s a case in point). The rejection helps to clear out the submissions pile in which a vastly disproportionate number of authors are seeking an extremely scarce amount of space in a prestigious journal. The editor is also in a secure position of power over a deeply subjective publishing process, meaning his own editing “errors” – i.e. the rejection of a promising piece on flimsy reasons and bad referee behavior – will not meaningfully register upon his or her continuation in that position. There’s basically no other feedback loop than for the rejected author to plead his/her case to the editor after the fact – a practice that may be easily ignored if the editor wishes, and that is heavily frowned upon in some academic disciplines.

So who does bear the cost of a the lazy or misbehaving referee described in the scenario above? Aside from abstract concepts of quality and the value of the lost academic discourse, there are two readily identifiable losers from the misbehaving referee scenario.

The first and most obvious is the author of the submission. He or she must endure the slow pace and other inefficiencies of a prolonged review process, only to emerge from it with essentially nothing – a rejection by the editor and a largely useless reader report from the misbehaving referee.

The author might gain helpful comments from the other referee who did her job in the scenario described above (Referee B), but even this is delivered in an exceedingly inefficient and cumbersome manner that could likely be achieved at much less cost and effort through other means. A helpful referee report is indeed a consolation prize, but it’s a very modest one that could just as easily be obtained by contacting a few academics over email and asking them to read a draft of a paper, or by presenting that paper at a conference that also solicits formal comments.

Let me also make the suggestion though that the conscientious referee is also penalized in the scenario described above. She does in fact perform a helpful service, but she gains no reward for doing so other than the distant satisfaction of advising a nameless and faceless author who now possesses an unfairly rejected paper and must begin the entire process again elsewhere. That might be sufficient for some people, but I’d wager that most referees who devote several hours to providing feedback on a paper would become less inclined to do so if they learned that their efforts were essentially being ignored by the editor. I can also attest to this experience myself – there are certain journals I have refereed for in the past only to find out that the editor was dismissive of my comments save to pass them on to the rejected author. If asked to review for some of these journals again, I know I’d devote less time to typing up thoughtful suggestions that I fully expect to be ignored. I might even decline to review in the first place. Now make the reasonable assumption that other potential conscientious referees would respond similarly to having their comments brushed aside and largely ignored by an editor. The overall incentive is to avoid spending the time that is necessary to provide conscientious comments.

A modest improvement:

I would not expect there to be an easy fix to any of the problems described above, nor do I have one to offer. But I will suggest a way to alter some of the incentive structures slightly and in a direction that improves them.

Specifically, journals might adopt a process that allows referees to review the comments of other referees before a final decision is made on the piece. (Note: some journals do currently share the reports between referees after the review process is complete, which is better than nothing but still a very minimal check upon referee misbehavior). In the scenario I describe here though, I envision something along the lines of a 1-week review period in which referees are invited to read their fellow referees’ reports, and then permitted to send a short supplemental note to the journal’s editor addressing any issues of quality that may be evident in those reports. It wouldn’t be a magic bullet, but it may create a touch of internal pressure in the review process as a conscientious referee could call attention to laziness, mendacity, or other types of misbehavior – all of which are currently costless to the misbehaving referee.

Doing so would also add more work for the editor, but I’d argue in a good way – an editor might be perfectly rational in deferring to unanimity or allowing the lazy referee to veto a decent article for suspect reasons, simply because it reduces the submission queue. But I think we can all agree that this too is a lazy practice that reduces the quality of a journal. A small internal incentive provided by the aforementioned round of referee review might change that slightly for the better. Besides, a lazy referee is also more likely to ignore the secondary round thus amplifying the voice of the conscientious referee. This in itself could become its own signal to the editor on a more efficient and appropriate weighting of their respective comments.