War Stories from the Blind Review Process
This discussion forum is for authors to share quotes or brief excerpts from funny, outrageous, or just plain awful reviews. Did one of your blind referees obviously forget his/her meds the morning of reading your paper?
Let's hear about the blind review process from people who have survived.
posted by Political Scientists for Scholarly Journal Ethics @ 3:37 PM
36 Comments
36 Comments:
Experimental design. Showed internal validity ect...
"Analysis needed to control for X"
So what you're saying is that they didn't understand the randomization element of the standard experimental design? Or were they challenging the adequacy of the randomization?
If it's the first, yeah, that's pretty dumb.
Do you get that a lot?
I've gotten it before (I'm a different person than the first guy).
It's not common, but it's not rare either.
I submitted a piece at the beginning of June and I was told that I'd hear within two months - it's now almost October and still nothing. I have no real question, or useful information to offer, just whining.
Why not email them?
Yeeeouch! I've had a couple of reviewers dismissively and flippantly suggest that my paper would be better placed in a magazine -- a commentary on what they perceived as the lack of political science or other social science theory to ground the empirical work.
Points about the underdevelopment of theory could be addressed in a way that is far less haughty and angry. They would carry far more weight that way.
These reviewers were obviously abused as children, or they wouldn't be so disdainful.
Yeah, and these are the same reviewers who couldn't begin to write clearly or coherently enough for a magazine....which explains what little value is placed on their own work.
I and co-authors submitted a paper to Political Analysis in August, 2005. We received an R & R in June, 2006. We resubmitted in August, 2006. We just received a rejection (February, 2007). I could understand taking 18 months to process a submission if we received thoughtful reviews; but, intead, the reviewers generally demonstrated lack of understanding and no interest in taking time to write professional reviews. With a little luck, Christopher Zorn will run that journal with greater ethical integrity when he takes-over this summer.
Ugh! Disgusting! This represents the worst of journal management. An unaccountable and diffident editor who doesn't care about service to helpless authors who have no other choice once they've submitted than to wait out the process (since simultaneous submissions are not allowed).
Send the scumbag outgoing editor a stink bomb, some rotten candy, or whatever else you think he deserves......
A rejection on the second round should take a few months _MAX_!
Those reading this site may be interested to know that there will be a conference in July:
The 2nd International Symposium on Knowledge Communication and Peer Reviewing: KCPR 2007. Reading the conference blurb, it seems to be put together primarily by people in the natural sciences.
It also has lots of citations to the effect that "a surfeit of publications has documented the deficiencies of this sytem"
Peri Schwartz-Shea, U of Utah
I did some research on peer review. Most of the studies conclude that the system just doesn't work. So the conference blurb doesn't surprise me.
But it CAN work, if journal editors would follow a few professional norms. This system remains superior to many alternatives.
There are a lot of authors across political science and other social science fields who would testify to how the process did improve their work, and did so in a timely manner.
The problems arise when editors are careless, unprofessional, arrogant and insufficiently attentive to the journals they are managing -- the editors who are taking on journals as a hobby. These are the people who are fooling around with people's careers.
I would certainly agree that peer review can improve papers, but that's not what the studies in science, medicine, sociology etc. look most at. They try to answer the question of whether acceptance via peer review is essentially stochastic; the results are kind of disturbing on that front.
Based on a recent experience of mine: when reviewing an article, *please* read the footnotes! I say this because I recently had an article rejected (2 of 3 referees recommended R&R) in which all of the criticisms raised by the one referee to recommend rejection were actually addressed in my notes! In truth, the editor really should have noticed this as well.
Don't get me started on the uneven quality of reviewers. Yes, someone should have noticed. Indeed, if a paper has two relatively strong R&Rs that might be a reason to let the manuscript go forward, i.e., give the author a chance to rebut/engage with the criticisms offered by one of the reviewers.
A lot of people receive polarized reviews that lead to rejections. Some of the time, the "good" review should be discounted. But other times, the argument is simply controversial. You'd think journals would want to publish articles like that...
Just had a paper rejected by _IS_ (by both reviewers).
Reviewer B provided 1.25 pages, single-spaced, of generally useful comments, most challenging my conclusions (which s/he rejected) and which identified some structural weaknesses of the paper.
Reviewer A provided 9 double-spaced pages that can best be described as "incoherent," though that might be too kind. The review was lousy with vague things like "part II is 'thin'" (whatever that means).
Without going into detail on the paper, Reviewer A nearly became hysterical -- the rage was so palpable that typographical errors (the kind you g enerate b ywriting reallyquickly) were rampant -- because s/he took the paper to be a defense of the Bush Administration (which it isn't) and a rejection of the journalistic Neocon Master Plan narrative for the Middle East (which it is) -- something that obviously troubled him/her on a deeply personal level.
Ideologically driven reviews are pretty common in work that has policy implications, even if these policy ramifications are rather indirect.
I don't know how you get around that, since many political scientists do, after all, have views of their own that color their reading of manuscripts.
You can take care to obscure the policy implications so that they are less obvious.
But ultimately this is why it's a good thing that there is more than one journal out there. You might actually find reviewers who lean favorably toward whatever policy implications your work might have. This points toward the random component of much of the review process.
Some might be discouraged by this fact, but the randomness /can/ work in one's favor too. Only when it does, it's never because the process was random, but because the paper was just really, really, really, good! ;-)
I just received 3 R&Rs and the editor rejected my article.
These are the kind of experiences that make you think about a new career.
Frustration understood.
Hopefully this is rare. But when you're the one person in twenty that it happens to, it still takes awhile to get over it.
This may be a sign that the paper is close, though. And this is why there is more than one good journal out there. Journals do reject good papers on a regular basis.
I know, this sounds cliched like "there are other fish in the sea...." But with 3 R&Rs, clearly this is not a hopeless paper.
3:25 pm: As a journal editor, I've occasionally needed to reject submissions with the equivalent of three R&R's. It's not pleasant, but sometimes necessary for various reasons. If you have a concern, you should contact the editor and ask for more explanation. In that kind of situation, the editor really should provide detailed reasons for the decision. And 9:43 am is absolutely right. Three R&R's at a good journal indicates that the paper is probably publishable. There's no reason to lose confidence in the essential fairness of the referee process or the quality of your work.
I have to second 9:43am and 5:00pm. I have worked as an assistant in a larger journal office and have to say, many good pieces of research are turned away, none are for the exact same reasons, and none are taken lightly by the editors unless they are completely out of touch (I will concede that is happens now and then). However, just about anything that gets 3 R&Rs at a top journal should be in range, with sufficient revisions, to get accepted to another journal.
A couple of reasons I have seen rejects in this situation before include; 1) the MS is very similar, if not nearly identical, to another work that has recently been accepted but is not yet published, 2) the reviews all see promise but the requisite changes are just too much to accomplish in a single round of revisions (this might happen if all reviewers suggesting R&R see the need for revisions in different parts of the manuscript), and 3) the topic of the MS merits publication, thus it was considered for review, but the premise as argued by reviewers is just too far outside of the mission of the journal.
Many of the other problems that are pointed out here (confusing reviews, assumed editorial indifference, long time to decision etc.) are in fact derivatives, at least in part, that some journals have between 300 and 600 manuscripts come through the office per year. When you consider the number of journals there are, you can recognize that there is in fact an argument to be had that good reviewers are not just dialed up, they are almost competed for. Many of the “top” researchers never agree to perform reviews, thus you are relegated to others. Meanwhile, the process of a reviewer rejecting a request, then securing another reviewer, takes time. The process of securing substantive reviews, I have to say, is the most stressing part of my job. Some real jerks will agree to a review, will not respond to reminders and phone calls for a long time, and then rescind their agreement, leaving the office, and author(s) hanging. It’s a very interesting process.
As 5:00 noted, none of these are reasons to be shaken about the review process as a whole. Simply understand to where you are submitting your work.
I'll follow up on 6:11's very insightful comments, as I'm also doing managing editor work for a major IR journal.
One of the most important factors is that there is simply a glut of submissions right now. Many of the major IR journals are experiencing a 25-50% annual increase in the number of submissions. The space constraints are still the same, however, which means that the standards are exceptionally high.
There are many, many papers that receive what I'll call the "standard R&R review." The ref says the paper is good but needs a bit of work before it's publishable. This sort of review used to result in an R&R from the editor, but no longer. Two or more such reviews will often elicit a rejection, even if the third review is largely positive.
What's needed are at least two exuberant reviews, along the lines of "publish as is, or with minor changes." I suspect that such reviews for a first submission were rare in the past, but in light of the increasing number of submissions, they're becoming increasingly common. So editors can afford to sit and wait for the exuberant reviews. They still have the same number of issues to fill.
That all sounds very pessimistic, but I suppose the comfort is that once you've gone through the process of receiving solid R&R reviews from a few journals, you'll have had the chance to revise your paper to the point where it will start getting the exuberant reviews.
I'll refrain from commenting on whether this trend is ultimately good or bad for the profession.
Had anyone dealt with the new editorial staff at East European Politics and Societies? I submitted an article there, got a revise and resubmit, resubmitted, and had it accepted. The catch was my article was too long, and so, after talking to the managing editor, I broke it into two pieces, and submitted the second part as a separate article. The first part I got a proof back on, and it's up on the journal's website, though it hasn't been printed yet.
After that, I trimmed the second part of the article to the desired length, and re-submitted it, along with a letter explaining what it was. In the meantime, the journal changed editorial staffs. A few weeks later, after sending three copies, I got a request to send an electronic copy for the reviewers (if they're gonna do that, I don't know why they don't change the submission guidelines), and then a few weeks later, I got an email from the managing editor ("signed" by the two editors) saying that the journal had many submissions and couldn't print them all. No reviews, no explanation, no nothing.
I immediately sent the managing editor an email saying I was baffled, given the situation. Then a little later I sent an email to the two editors, saying the same thing, as well as requesting copies of the reviewers' comments.
A couple of days later, I got an email back from the managing letter, saying "The editors have seen your message and stand by their decision." That was it--nothing else at all. I emailed her again, an cc'ed the two editors, asking for reviewer comments again, but that was two days ago, and I haven't heard anything since.
I should add that the managing editor hasn't been overly polite, and that being pointedly ignored by the editors when I email them directly (twice) doesn't win points with me either.
Your paper was not sent for review. You received what is commonly called an editorial or administrative reject. It can happen for many reasons, but basically the paper is rejected by the editors without them wanting or needing reviewer input. Your recently submitting a closely related paper was likely a factor.
As far as I know, the paper was submitted for review--I was specifically asked for an electronic copy for that purpose. However, it's hard to tell for sure since I've gotten no feedback whatsoever. Don't you think that's a little unusual?
This ambiguity definitely calls for an inquiry to the editorial offices.
If email is not answered, a follow-up phone call, even to leave a voice-mail message, is in order.
Thanks for the advice. I still haven't gotten a reply, more than a week after the last email. It's clear that the editors don't want to deal with me (only the managing editor has responded at all), but I guess a phone call would if nothing else force them to consider the consequences of being rude to the next person (and I'm not sure what I have to lose in all this). Should I call the editors directly, though, or the managing editor? Actually, the managing editor is apparently out of the country.
5:36 is spot-on, the likelihood of this happening is likely tied to the fact that they recently published a piece of yours that was closely related enough to the current that they decided not to give it full consideration.
Whether or not this was as noted, as "administrative reject" or not is hard to tell if the editorial team is not willing to allow the requisite transparency into the process.
One other thing to consider, is that new editorial teams are only loosely bound to manuscripts/policies that carry over from the previous team. That your submissions occurred during the transfer could indicate several things: 1) That they were not entirely understanding with why you should submit two decidedly similar works, or 2) the process of the new team does not call for formal decision letters, and what you received was in fact the reviewers decision, as odd as it was.
Of note, and this goes for all...
If in fact you received a reject, even if you couldn't tell that that is what you received, and your inquiry to the editor came off in any way rude or presumptuous (note that sarcasm and emotions are rarely conveyed accurately across email), this is a quick way to get very terse responses.
There is no excuse for rudeness, but an answer is called for. Make a phone call. Start with the doormen, the managers, and if you don't get a response, go to the editor.
Yes, I would start with placing a call (leaving a message) with the managing editor.
Does the journal have a policy about shutting its doors for July-August? Some do.
But the professional thing to do is to advertise this on the jounal's website, as the /Journal of Politics/ does.
An "out-of-office reply" to my last email from the managing editor said she was leaving the country until August, but would be checking email on occasion. Since it's been a couple of weeks and she still hasn't replied to me, I would assume that she's purposely ignoring me--and I can't call her, since I don't know where she is. Should I call one of the editors, or just wait?
As for rudeness on my part...when the managing editor first asked for an electronic copy, and I sent it to her (this was a couple of months ago now), after she called me "Mr." twice, I sent her a note saying that, while I wasn't offended personally (which is true--I'm not normally hung up on titles), she might want to be careful about doing that with academics. I never got a reply from her on that, and she did it again with the rejection email, and I noted that in my first response to her (I said, "I'm still no a 'Mr.'"), after which she referred to me by first name. On the one hand, I feel absolutely ridiculous about mentioning it, or even feeling upset over it--but on the other hand, when it happened with the rejection email, I was actually a bit offended, I suppose because it comes off as a slight when the person ought to have known better.
I don't know: maybe that upset her, but I didn't say anything else untoward, and I don't see why the other people involved would be upset with me.
This is getting just a little weird and contorted.
No author should have to put up with any of this silly nonsense.
I'd withdraw and go to another journal, assuming you have options. Time to move on.
If you don't, then I'd just wait until they decide to come back to their hobby -- running this journal. And then you let them call you whatever they want.
But with well over 200 journals in the field, you should have options.
Oh, I know I know I have other options (though one annoyance to all this is that I'll have to change large parts of the text to avoid a copyright issue). I just would like to know what was wrong with the article so that I can avoid that mistake the next time (with this article at another journal--East European Politics and Societies is not getting any more articles from me while this editorial board is around). Of course, I'm also just annoyed.... :)
Publishing articles in respected (i.e. peer-reviewed) journals is not only a means for disseminating the results of our research, but has also become an important indicator of our value on the job market. While the process of (double-blind) peer-review is often criticised, there is very little factual information available. Together with a number of colleagues from the universities of Essex, Keele, Mainz, Mannheim and Salford, we have set up the Political Science Peer-Review Survey. If you author, review or edit political science manuscripts, you should really take part. It takes less than ten minutes of your time and should give some answers which are highly relevant for all of us. Here are some preliminary results.
In the last 18 months I have received three different reviewer reports from two different journals where I suspect sexism. In each of these reports, all of the critical comments are aimed at the author rather than the manuscript with frequent references to the author as "she" and "her". Now if the review was truly blind, they would not be so certain of my (female) gender. Clearly these reviewers know who I am and want me to know that fact.
This creates a chilling atmosphere for female political scientists and I am baffled that the editors didn't note the reviewer's behavior. Is this kind of bullying common in the political science peer review process?
No, I don't believe it is common.
It's more likely to occur, though, at lower tier, more specialized journals, than at top level journals. The management of some journals is not very professional, I'm afraid.
Best advice is to keep trying. Take whatever you can from these reviews, and submit to another journal.
Thanks for sharing this info.
As a journal editor I am a bit distressed by 6:19 above, and (less so) by 9:51 in response. The simple fact of the matter is that we often have reviewers use "she", "her" as a matter of preference. I suspect that most are trying to make a statement against sexism. I would go so far as to say two reviews in 18 months (obviously it depends on how many articles have been submitted) using the feminine rather than masculine form is, well... totally unremarkable. So how does this translate into "clearly" knowing the author's work, and perpetuating a chilling effect? Ouch! I would simply suggest that maybe the author had not considered some things in the article, and perhaps is a bit touchy about criticism thereof...? I will say that at our journal we are diligent about anything in reviews that smacks of ad hominem attacks against authors, and will ask reviewers to remove or modify their language.
Post a Comment
Subscribe to Post Comments [Atom]
<< Home