Do the referees in science journals get payed for their work? Is it a full time job, or are they asked to referee specific papers which fall within their area of expertise? How many referees referee a paper?
Suppose I submit a revolutionary paper that is rejected by the referees. How do I go about defending my paper?
I’ve never read a journal, so sorry if some of the questions are naive.
It differs from journal to journal. My experience is that reviewers are not paid, it is done mostly as a scientific courtesy. Specific referees/reviewers are invited from fields of expertise, for instance our lab, an eye development lab, gets papers from other eye development labs as well as other fields of development that involve similar signalling pathways.
It is not a full time job, and the experience is often done in collaboration with graduate students and postdocs in the lab – it is a very valuable skill to be able to read, critically analyze, and decide to support or reject a paper. It means that you 1) must know your background well 2) must be able to process and understand the data 3) decide whether to believe the data 4) decide whether the data agrees with the observations and conclusions 5) find all the shortcomings in the paper 6) know what the journal you are reviewing for is looking for in a paper 7) decide if all of these things make a paper worthy of acceptance. I reviewed a paper for Science – my first pass through, I found the shortcomings, I believed the data, but I didn’t know if it was Science material (even after the shortcomings were addressed). My boss, with more experience than I, was able to look at it and say “This is the first time this type of regulation has been described in this system” which instantly made it Science worthy, so we recommended acceptance with changes. It generally takes at least a few days of full time work to do this, making reviewing a dozen or so papers a year perfectly reasonable (especially with grad student/postdoc help).
The process works as follows. One writes a paper, according to the format of the journal you wish to submit to (for instance here). One submits to an editor (often online), with a cover letter. Usually, one can suggest reviewers to use or to avoid. The editors read the paper and decide to send it out. If there are gaping holes in the paper or the paper should go in a more niche or different journals, it is rejected without review by the editor. Otherwise, it gets sent out to anonymous reviewers (a point of contention – there are lots of debate whether to allow the writers to know who the reviewers are right now) for a month or so, and then it comes back. Usually, it is 3 reviewers. If two recommend acceptance, the paper gets in. Reviews come back, and the editor usually asks for both responses to the criticisms and changes asked for by the reviewers.
Any part of this is debatable. Rejection without review can be appealed. Often times reviewers can ask for revisions that are beyond the scope of the paper and unreasonable. Each point is a debate between the writer and the editor and occasionally the reviewers. If a particular reviewer is way off base with the others, one can request a new reviewer. It is a fairly open, fairly plastic process that allows a collegial back-and-forth between all parties with a minimum amount of political rancor and bickering and a lot of protection of data.
So if you submit a revolutionary paper, your major foci will be addressing why your paper disagrees with previously accepted work and explaining why your work is revolutionary. If you do not do these things appropriately, or your reviewers are unfair, you have opportunity to retune the paper and resubmit, submit elsewhere, or appeal to the editor. If you are unable to do these things, then the review process has worked and your paper should not be published.
A clear description of the process by edwino. Some other stuff:
I have not heard of any cases where referees are paid.
Rejection without review is really just the purview of the mega-huge journals, i.e., Science & Nature. Sometimes I think other journals should reject without review though…!
Reviewers are often (usually) given the choice of not being anonymous, though nearly all maintain their secrecy.
A month or two?! For specialist journals (i.e., not Science or Nature), receiving first reviews can easily take 3+ months.
If two recommend acceptance, the paper will usually get in - but not invariably.
Yes, although the editor’s decision is, in the end, final.
Ha! Reminds me more of Usenet at times, with a thin veneer of politeness of a Core of Pure Evil.
I worked for the ornithology journal once. None of the reviewers got paid. As a matter of fact the running joke was that the competition was so fierce because the stakes wer so low.
At the Auk, it was generally three before publication. It went to a general reviewer first. If the paper had major problems, it went no further and was reurned with comments and th ereasons why to the author(s). If it made it past the first general reviewer it went to at least two specialty reviewers. Then it may have gone on to publication, be rejected by one or both of th ereviewers or be sent to another reviewer.
Letter writing to the reviewers.
Btw, the biggest source of hold ups in the Auk’s review process was incorrectly formatted MS.
You’d think that after making it into or through grad school they could read and follow the directions. It wasn’t complicated at all. Yet…
After that it was reviewers who procrastinated.
Really? In my field (Earth sciences), most reviewers do not ask to remain anonymous. In fact, it’s common practice to thank the reviewers by name for their comments in the acknowledgements section, even including the names of reviewers of earlier versions of the manuscript that might have been rejected by another journal. Anonymity seems pointless when the authors could very well tell by the review comments just who has been looking at their paper. 
If the editor feels that there are grave concerns that remain after the authors respond to reviewers’ comments, the manuscript is going to be rejected, period. There’s nothing to keep the authors from resubmitting to another journal though, and that’s pretty common when the first journal you try to submit to is highly competitive, like Science or Nature.
I’ve written a few reviews for bird journals. If any journals pay for it, I’d like to know about it!
Not in my experience. To confirm what sunfish says, as far as I recall, only one reviewer of any of my articles chose to remain anonymous - the others all signed their reviews.
To the reviewers, or to the editor? In my experience, I’ve always corresponded with the editor alone. I’ve never corresponded with a reviewer directly.
I just had a minor article accepted by an African bird journal that was a real pain, considering the size of the article and the size of the journal. One reviewer thought it was fine, and had a few minor comments. The other two, a married couple, gave it an extremely nit-picking and highly disdainful and outright insulting review, and recommended it be published in a ridiculously abbreviated form, if at all. I think a good bit of their problem was that they were not familiar with my name (it’s the first time I’ve published on African birds) and they were being territorial.
Anyway, I was able to answer all of their objections in detail by writing a very extensive letter to the editor. He agreed to publish the article more or less in the original form (with some revisions). As far as I know, he did not send my own comments back to the reviewers for their response, and I assume he just assessed their validity on his own. (These 2 reviewers have a reputation for being difficult in general, and I think he was almost embarassed by how over-the-top the review was. I met the first reviewer some time later at a conference, and when I told him the problems I was having with those guys he just laughed.)
The process has been pretty thoroughly described already, I’d just like to add a few points.
Paying referees: As already stated, the reviewers are not paid. However, being listed as a reviewer for a quality journal looks very good for your career. It’s an acknowledgement that you have specific knowledge of the field and name recognition. Always a good thing.
Anonymity: In my field (microbiology), reviewers remain anonymous. However, it’s usually not that had to figure out who reviewed your paper based on their comments. Along these lines, you can request a certain editor which can help grease the wheels as some editors are more willing to fight for a paper than others. Also, you can ask that certain people not be reviewers due to research competition, conflict of interest, etc.
You can ask, but sometimes that will result in the editor (or, God forbid, the NSF program director for a proposal) deliberately sending the paper to a person you wanted to have kept out of the loop, just because they are curious as to what that person will say. It’s not necessarily a death knell for a paper because an editor can always get a fourth reviewer (if three are standard), but it sucks when a proposal gets killed this way (speaking from experience).
My experience to date, so far. My boss likes to submit high and get rejected. So our papers begin at Science, Nature, and Cell, invariably get rejected, and end up in Neuron and Development. I just had a second author published in Dev Biol after minimum amounts of revision. My thesis work was submitted to Cell and just like I suspected, it was bounced back without review. My boss got the bright idea to submit a nearly identical paper to the more niche Cell-press Developmental Cell to the same editor. Unsurprisingly, it came back just as fast. Now it is at Genes and Development, which, with an impact factor of around 19, is actually ranked higher than Dev Cell. I suspect it will get reviewed but rejected although who knows. It is more of a Development paper, really.
It will be nice to get reviewers comments, though.
It is not a full time job but it can feel like it. Generally I get two paper a week to review (chemistry), and I wait till I have four or five sitting on my desk and the editors are screaming at me. It can take a day to review 6 papers properly especially if it is not your specialist area, but if I am lazy and the papers look reasonable 5 minutes.
I published in Nature late last year and the whole process has got much faster, initial submission, refereeing, manuscript revision to print publication took close to 3 months.
I think these comments go hand-in-hand. In my field (fluid dynamics), I frequently saw very nit-picky reviews from people who were trying to stifle competition in their specialty. All the reviews we got and gave were anonymous, but it wasn’t hard to figure out who the reviewer was based on the authorship of all the tangentially-related papers the reviewer insisted should have been cited.
My wife sees this frequently in experimental psychology as well. While she’s well known in her specialty, she frequently supervises student research in other areas and they can have a lot of trouble getting good papers past territorial reviewers.
Yes, my two reviewers mentioned that I should have cited several of their own extremely obscure and non-peer-reviewed (self-published) reports from an area several hundred miles away from my own study area. The biggest delay in my revision was getting copies through interlibrary loan, since very few institutions had them. But I needed to read the reports in order to demonstrate to the editor that they were irrelevant to my article. Even if they hadn’t signed their review, it would have been glaringly obvious who had written it.
There are clearly differences between fields. In my field (cognitive neuropsych / experimental psych) it’s unusual for a reviewer to identify themself. In >100 sets of referee’s comments I’ve received, I can only recall the reviewer identifying themselves twice. In both cases, the reviewers were very senior figures. I have never revealed my identity when I review articles.
Again, differences between fields in the detail of the process. In my field the response to reviewers comments directly addresses the points raised, and the response is essentially a direct reply to the referees (and sometimes to the editor’s comments, if there are any such). Referees see the authors reply to them, indeed each referee sees the response to each of the different referees - and the process can circle the merry go round as many times as needed.
And as others have said - territoriality and nitpicking do occur, though only in certain areas, those where competing theorists are repeatedly poking each other in the eye over a picket fence (and probably have been for the previous 43 years)…
Interesting. Only once have I remained anonymous in giving a review. (The [invited] paper was so bad it required a total re-write, and both the editor and I agreed that the particular authors would likely take the criticism better if they assumed it was coming from a male. Cultural issues, and all that.) On the other hand, someone I know in who works in medical genetics once told me that her papers are sent out for review minus the details of how her investigation was done, to help keep her from getting scooped by the competition, i.e., her reviewers! Still trying to figure out the usefulness of the review process, in that case.
My experience is like Colibri’s, where the reviewers won’t see my reponses to their comments unless I send them directly to the reviewer (which would be unusual). In my field, the editor reads over my responses to the reviewers and is the one who makes the final call.
Agreed that some territoriality is a given, only in my field it’s sometimes actual territory that gets quibbled over. 
Am I the only one amused by the idea of ornithologists being “territorial”?
I am?
OK. Just checking.
Not an answer (as many have already been provided) but a story…
A few years ago, one of the faculty here was sent a paper to review, and it turned out that the author had plagerized, from the reviewer.
Ooops. As a further oops, the faculty member’s secretary faxed the review to the author, instead of the journal editor, with the comments “This paper was almost completely plagerized from my paper, blah de blah de blah.” Hillarity ensued.
I almost want to cross post this in the “Stupid Cheaters” thread in the pit…
You’re not the only one. I was amusing myself picturing the two mentioned reviewers singing loudly at Colibri.