What are you talking about?
It is my experience, as well, that academics are busy people. Why do you think otherwise?
-FrL-
I am going to have to go back and read this thread more carefully, but I may have some things to say later. I am an editor for an international medical journal, and we do double-blind reviews: The reviewers don’t know the names of the authors, and the authors don’t know the names of the reviewers. We have an editor-in-chief and 18 associate editors. Our database holds many thousands of practitioners who may be assigned to review.
I will check back later when I have time.
Exactly the links I was going to post. The OP bears more than a whiff of “pointed agenda.”
I agree that reputation does play a role in an academic’s career, but is it large enough to balance out the all-consuming need to publish. I remember that the first time I went to a conference in grad school, the department chair warned me beforehand about several attendees who were known for publishing shoddy work, stealing ideas from others, and so forth. But while their reputations suffered, they were still apparently able to get tenure and appear at conferences.
Furthermore, the academic world is quite heirarchical, and you have to be careful about whose toes you step on. For instance if a grad student found a flaw in a paper written by someone on his or her committee, he or she might think twice before making a stink about it.
The fields that I worked in were math, theoretical physics and computer science. In all of those, papers were typically a proof (or a series of proofs). The whole point of having a proof in print was (allegedly) that after a theorem is proved once, nobody has to do it again. Now a single proof could often run to 30 pages or more of very dense mathematics. The amount of effort required to thoroughly check every single equation, every single statement, and every single definition is extraordinary. In fact, it’s not unusual for a grad school class to spend an entire semester grappling with one proof or one chapter of a book. And I find this problematic because I doubt that the reviewers are going to be thorough enough. Simple time constraints prevent it.
There is also the issue that in any particular field or sub-field or sub-sub-field, the academics within it tend to form a tight-knit community, sometimes without much communication with the outside. This is probably a more typical problem in the humanities and social sciences. A community of that sort can easily produce self-reinforcing prejudices. Everybody shares the same bedrock assumptions, hence those bedrock assumptions do not get questioned, or at least not as much as they should. There’s an excellent description of the problem in one field in the essay How to Deconstruct almost Anything. The author, Chip Morningstar, says:
This is precisely the sort of thing that I’m worried about.
Academics are far from a tight-knit community in the humanities. They have cliques, but those cliques are often in a war with other cliques due to theoretical arguements (and the occassional personal issues).
My wife is tenured, reviews, has been an editor, and is on a grant review panel for NIH. This means that our home is regularly covered in stacks of articles she is reviewing, or grants she is reviewing. She reads every single one of them. She is also reviewed herself several times per year. She gets an article back with the comments of 3 reviewers. There is one person who is completely against the theory that my wife uses for some of her work - and when that person reviews her work it comes out in the review. This FORCES my wife to be extra careful in her writing to get past that person. There is no “easy pass” going on - there is an outright war.
I don’t think its the peer review that is the problem, but rather our perception of what it means.
Peer review is just a way of vetting an article to make sure it’s not whacked out. The reviewers don’t repeat experiments, recalculate all the data, etc.
Being published in a peer reviewed journal simple means that the author has made a respectable argument. Kinda of like a indictment.
The actual court case doesn’t happen until after it’s been published. When others may dispute the results, repeat the experiment, etc.
But the simple fact that the article has been published in a peer review journal does not make it inherently correct, just like being indicted for crime doesn’t mean you are actually guilty.
For the record, I recently submitted a paper to a theoretical computer science conference with a complex “proof” (around 15 pages long in abbreviated form). The “proof” we thought was watertight when submitted, but after closer inspection, we found that there was a serious flaw in it, and that necessitated rewriting a large portion of it quickly.
Unfortunately, for us, we thought that we may get away with this—what reviewer is going to actually read through this “proof” closely? After the referee’s reports come back, we can just submit our final, corrected paper instead, right? Wrong! One reviewer spotted the error, giving the correct location and reason why the proof failed and rejected the paper. We’d been working on this for over 8 months, and were convinced that the results were sound, but peer review prevailed 
Not that I particularly desired to bring those threads up again, but I’ll say this much. SentientMeat told a great many nasty lies about me and also scratched his head in wonderment at my refusal to trust him. The fact that the two things might be related does not seem to have crossed his mind. Tempting as it may be, though, I will not bash academia in general based on his thinking alone.
My goal for this thread was not telling the academic world what to do. I have left that world. I am merely a human being who desires to learn things and know things. My purpose here is to make known my opinions about peer review.
But if you want to know how I feel about gathering knowledge, I’ll say this much. My technique is to learn from everything around me, and to try to make my experiences as broad as possible. Among authors, I’ve come to prefer generalists rather than specialists. I figure that the more things an author knows, the less likely that they’ll make mistakes. When I think about the list of authors that I consider great, I see that few were among the highly educated, or if they were their education was often in a different field from what they wrote about.
One way anonymity is the general rule in TCS, as far as I can see. I don’t see how this is an impediment—a proof is either correct or it is not.
I think this is a very good analogy.
I agree. I would say, though, that my field is a lot more cathedral than bazaar. The chemistry literature is a venerable edifice that is made of stone, you have to be considered about your papers. It doesn’t have the organic qualities of some younger fields - dashing stuff off and then correcting / re-considering is not encouraged. This is particularly true for correcting / re-considering data - if you’ve made something you definitely don’t want to be mis-characterising it in print.
This ponderous structure is a drawback in some respects, but a great strength of it is in peer review. There is great reverence for the literature in the community - authors and reviewers uphold both ends of their contract, so to speak. ITRChampion is not wrong when he says it’s tough refereeing a dense paper, so some mistakes creep in; but very few lies. Mistakes in important / mainstream areas will get spotted by other chemists, which is the self-correcting strength of wider peer review in the community.
Pah. This forum is Great Debates, not MPSIMS (although perhaps you’re simply witnessing?). The fact that you posted it here entails an argument, which means you’re advocating a position. And specifically, that position is: peer review leads to the opposite of “high-quality, truthful articles”. Or, restated, peer review inevitably results in ill-conceived falsities. At best, that’s asisnine hyperbole; at worst, it accuses the entire academic world of fundamental dishonesty and corruption.
Again, I’m not saying that the peer review process is perfect (it isn’t), nor that peer reviewed publications are beyond question (they’re not). But if you’d really like a debate and not just squeal out adolescent whines, you might formulate alternatives and give reasons for why they’re better. Y’know…lay the groundwork for an argument.
And you propose this – with a straight face, I presume – carrying with it the implication that someone’s being expert enough to participate in the peer review process is mutually exclusive with their gaining “broad experience” (not just limiting, but exclusive of breadth). More disturbingly, you claim that being ignorant in a particular discipline somehow actually provides expertise in that discipline – for how else can authors I consider great were either not highly educated or educated in a different field be interpreted? You really don’t see the inanity (insanity?) of that opinion?
But if you want to know how I feel about your view of others’ technique of gathering knowledge, I’ll say this much. Humans are not only fallible, but limited in both time and understanding. Becoming well-versed on a topic necessarily requires sacrificing some experiences and activities for others, a trade of breadth for depth. It strikes me that your issue isn’t so much with the peer review system. Rather, it stems from egotism that borders on solipsism; a self-absorbtion that refuses to acknowledge the value and validity of others’ expertise (and even expertise in general). A sort of “who are they to claim expertise?” indignance, so narcissistic that you elevate relatively benign warts to the level of fatal flaws.
Actually, having just written that last sentence, it occurs to me: with a little adjustment, I think you might’ve been an awfully successful academic. 
Exactly. It doesn’t even mean that the peers who did the reviewing necessarily agree with the conclusions of the paper.
Marc
Here is an example I had with peer review.
I wrote a very simple paper on extending the Playfair system beyond digraphs. The editor I sent it to wanted more depth to it. I kept adding and adding until it got to his standard. Then he sent it to peer review. Their reply was that it was too complex of a paper and not fitting for the Playfair, itself a simple system. When I asked the editor if there was a middle ground between what he wanted and what the other reviewers wanted, he said probably not and refused to take a resubmission.
The debate that I started was not about alternatives. The debate that I started was about evaluating the worth of peer review. If you don’t like the debate I started and you’d prefer to have a different debate, you are welcome to start your own thread. Your are not welcome to order me to have your debate in my thread.
I see the insanity of the position which you dreamed up and tried to pin on me, but as it has no relation to anything I wrote I don’t really care. Your first sentence assigns to me an “implication” that I never implied. Your second sentence assings to me a “claim” that I never claimed. Indeed I claimed the exact opposite. I never said that being ignorant is a discipline is a good thing. I praised the generalists, who are precisely those people who are not ignorant in any discipline. As for that quote of mine, it could be interpreted as meaning what it says, rather than something that it does not say. It is a statement about those authors who had the greatest effect on me, nothing less and nothing more.
It’s abundantly obvious that you want me to be arguing for total anti-intellectualism. But as stated in the thread you already linked to, things don’t become true just because you wish they were true.
Digital Stimulus, let’s not get personal in our attacks on the ideas, here.
(And on preview), ITR champion, you have posted some stuff that a lot of folks are going to consider pretty silly. If they let their emotions get in the way of their posting, you would do well to refrain from posting on the same emotional level.
[ /Modding ]
This shows you don’t know anything about peer review in the humanities. I have published a dozen articles and a book in the field of philosophy, and know a lot about peer review, having been a referee a bunch of times and having received numerous referee reports. People work hard on them; you can tell. Most referee reports I receive are very thorough and indicate that someone read my paper very carefully. And they don’t try to spare your feelings. Peer review is a tough process. It’s not perfect, but by and large, the process works: the good journals are where the good articles are, and the less-good articles get shunted off to the less-good (and less-read and -respected) journals.
And of course peer review isn’t the only mechanism by which scientific error is corrected. A faulty paper that makes it through the peer review process might be refuted by other papers, or discounted because others cannot replicate the results. Science is rational because it is self-correcting. Unlike some other belief-systems.
Have you ever refereed an article or received a referee’s report? Do you have evidence for what you are talking about?
This I can absolutely agree with. Peer review would work perfectly if you had perfectly virtuous people doing it. If the people involved fail, then peer review fails to roughly the same extent. The same could be said, more or less, for any other avenue of publishing. But I do feel that the academic system, as it is currently set up can give people an extra nudge towards dishonesty in some situations.
Just for the record it is not true that Social Text did not peer review its articles at the time when Sokal’s paper was published. According to the editors’ account of the matter they decided against submitting the Sokal article to a physicist for peer review because they assumed that it was not a contribution to physics but, rather, the opinon of a credentialled physicist on the institution of scientific study and they were eager to include his perspective in their special issue.
That doesn’t make it a good decision–it was a bad decision and one the editors publicly (and doubtless privately) regretted. But Social Text was then as it is now a peer reviewed journal–making the Sokal article an unfortunate exception to the rule.