In Part 1 of this post I credited Cengher and LeBlanc (2023) for presenting a detailed survey of how the peer review system works, and offered some of my own thoughts about how reviewers should function in that system. Now I present some thoughts about the role of the action editor. Throughout this series of posts, if you think I’ve missed or misconstrued some important aspect of the peer review process, I’d love to hear from you (tscritc@ilstu.edu).
Rules for Action Editors
Many of the rules that apply to reviewing apply to editing, but there’s also the following.
- It doesn’t take four reviewers to decide whether most manuscripts can work. Having too many reviewers, in fact, simply results in a cacophony of voices to which no author can respond effectively in a revision. Some of our journals are in the habit of involving as many people as possible in the peer review process, and that’s well-intentioned, but reviewing by committee is no more productive than doing anything else by committee. Two reviewers almost always tell you what you need to know about a manuscript, and they can do this in a far less complicated way than four reviewers can.
- More reviewers is an even worse strategy when, to fill out a roster of reviewers, you select people who lack the basic expertise to evaluate the research. Unqualified reviewers interfere with the goal of advancing the literature. They ask a lot of questions that the literature has already answered and they impose methodological and evidence standards that are not appropriate to the research area. Note, too, that asking someone to review a manuscript for which they aren’t qualified isn’t fair to that person; it’s asking the person to function professionally outside of their domain of competence, something that all clinical ethics codes forbid (and perhaps research ethics codes should too).
- To be clear, there is a role for the generalist reviewer, especially in journals with wide circulation and that publish on widely varied topics. For these, a generalist reviewer should be expressly charged with looking out for the interests of nonspecialist readers. To put that another way, a generalist reviewer should help the author communicate in ways that will keep nonspecialist readers reading. This being said, however, not every reviewer is a generalist… that’s a special skill set, so you can’t simply pick someone who works on Topic A to review a manuscript of Topic C and expect that person to be helpful. If you want a generalist perspective, pick someone with a demonstrated track record of bridging research areas and communities.
- When inviting reviews, tell reviewers why you chose them. I send a customized invitation letter to each reviewer — not one of those horrible boilerplate invites that editorial manager systems generate. I say explicitly why I’m asking each person for their input. The boilerplate letter will say “because of your subject matter expertise and experience in the field” or some such nonsense — those are weasel words that communicate nothing. I say things like, “The study employed signal detection analyses to evaluate the specificity and sensitivity of the diagnostic tool. I really need someone to tell me if the analyses were conducted competently.” Funny enough, if you tell people what their job is, they do it better.
- For reviewers generally, telegraph specific questions you have about the manuscript that you’d like them to answer. I do an initial read of the paper as if I were a reviewer, and often there are big-picture issues that concern me, about which I would really like reviewer input. This often involves how the manuscript fits into the existing literature. Once I was action editor for a paper on the use of punishment in the penal system. Knowing that our distinction between operant and legal punishment traces back to Skinner and Science and Human Behavior, I immediately wanted to know whether reviewers thought the manuscript extended our understanding or merely recapitulated old points. Also, because of Rule #4 for Reviewers (concerning “Who might do what differently, with respect to what problem, as a function of reading this article?”), I was curious whether this conceptual paper would have any practical utility for readers versus simply demonstrating that an author could think behaviorally about some complex topic. Pointing reviewers toward specific issues is really useful because the truth is that a reviewer’s job is super ill-defined. As a reviewer, you receive this complex stimulus called a manuscript and then have to figure out, from among a myriad of possibilities, how to respond to it. Too many degrees of freedom are involved, and this creates opportunities for reviewers to go down unnecessary rabbit holes. I always assure reviewers that their comments should be whatever their high-probability verbal behavior dictates, but if they’re so inclined I’d love guidance on those specific issues I asked about. I’ve found that reviewers generally like being told what I’m wondering about because they generally want to be helpful, and this is just defining one way for them to help. Again, funny enough, if you tell people what their job is, they do it better.
- You can disagree with the reviewers. The whole point of an action editor is to serve as a conscience for the peer review process. You’ve been selected for this role because someone thinks you have a special perspective on the discipline and on the journal’s place in it. So be the conscience. If you think reviewers have treated a manuscript too harshly, accept it despite their “reject” recommendations. If you think a manuscript has a fatal flaw that reviewers missed, reject it despite their “accept” recommendations. Their recommendations are advice, not votes in a democratic system. To be sure, the default strategy is to trust your reviewers, and I have “overtuned” mine perhaps only 5 or 6 times in my whole career. But I have slept well at night after doing so and, when manuscript acceptance was involved, I have been proud of the resulting published articles. By the way, when I explain my contrary decision to reviewers (see Rule # 10), they usually take the news in stride, and what results is often a really productive conversation about our shared goals as a review team.
- Focus! Focus! Focus! I can’t emphasize this one strongly enough: Upon receipt of reviews, your job is to define for the author, precisely and in uncertain terms, what are the make-or-break points for revision. It’s a dereliction of duty to simply say, “Here are the reviews. Respond to them. Good luck” — a strategy that I see being implemented with increasing frequency by action editors. This strategy is a dereliction of duty for two reasons. First, you’re failing the journal because journals need authors and you’re making the author’s job unnecessarily hard. The author should not have to guess about how to weigh the importance of a host of recommendations — especially given that reviewers sometimes render incompatible advice (“Half of the Introduction section can be deleted,” one reviewer will say, while another wants Paragraph 3 of that section expanded to book length). Second, you’re failing the reviewers because, by not imposing any clear vision on what the final product should look like, you’re depriving them of feedback on which of their suggestions were most valuable. Overall, if all you’re going to say is “Respond to the reviews,” a bot can do that, and there is no need for a human action editor.
- Dealing with revisions, Part 1. There is one further benefit of setting clear revision priorities: This makes YOUR job easier in evaluating a revision. If you tell an author that the revision absolutely must include Things 1, 2, and 3, it’s a simple matter to decide whether it does. But if you tell an author to consider Things 1 to 100 (impossible to incorporate all of them in a journal-length paper!), how in the world do you decide whether the revision is a success? Failing to take a stand at the point of initial manuscript decision is simply kicking the can down the road, for both you and the author. It’s a waste of everyone’s time.
- Dealing with revisions, Part 2. If you’ve set clear priorities for a small handful of required changes, and those changes get made, you free yourself up to proceed with what I call a holistic evaluation of the revision. Here’s how I do this. First, because I asked for only a few mandatory changes, it’s easy for the author to say how the changes were made and where in the manuscript I can find evidence of this. So I quickly verify the changes. Second, I do a careful read of the revision without consulting the previous reviews or my decision letter. The idea is to see if the updated draft holds together as a coherent whole — which by the way is critical to the point raised in Reviewer Rules #5 and #6 (the end product of good peer reviewer should be a paper in which readers can easily perceive value to themselves). If I instead check reviews and the decision letter first, I’m now under control of all sorts of details that may or may not matter to the new draft. To say this another way, once you begin changing certain things in a manuscript, the relative importance of other things can wax or wane, and that should be a stronger driver of the revision than any itemized list of issues from the reviews and decision letter. Third, sometimes after the holistic read I re-check the reviews and decision letter to see if there’s anything critical I missed — but I don’t always do this. If the revised manuscript is a bang-up piece of work that is going to benefit the literature, I don’t care much what kind of correspondence it has to details of the reviews. I suppose that might sound disrespectful to the hard work reviewers have invested in their evaluations, but an editor’s first responsibility is getting good product into the journal’s pages, however that may be accomplished.
- Reviewer followup: Once I write my initial decision letter, I contact each reviewer to share the decision, offering a few words about how I arrived at it. When reviewers agree, this is easy and I tell them the truth, that they made my job easy. When reviewers disagree, the message is more complicated, in part because egos can be involved. However, I find it’s almost always the case that each reviewer offered something that shaped my decision — for instance, a reviewer who recommended rejection might’ve identified the very change that makes the manuscript salvageable, or a reviewer who recommended publication might have worried about the same issue that another reviewer thought to be a deal-breaker. I rarely have to stretch to let each reviewer know they rendered a valuable service. Overall, we must never forget that reviewers work for free and editor gratitude is their only proximal “payment” for services rendered. Also, one hopes that this kind of followup helps to tweak the way reviewers approach future evaluations, which, selfishly, makes them more valuable to me for future submissions.