The Case Against Peer Review

[A] Peer review is defended as the mechanism that separates reliable work from the rest, and the evidence for that claim is thin. [B] Studies in which the same manuscript goes to several reviewers find agreement about acceptance barely above what chance would produce; papers later shown to contain fabricated data have passed review at the most selective journals; and the errors most often caught are matters of presentation rather than of substance. [C] The practice is also expensive in a way that appears in no budget, since it consumes the attention of the people whose attention is scarcest. [D]

The conclusion usually drawn is that review should be replaced by open publication and public comment, and the argument for that is weaker than the case against the present system. Comment after publication is not distributed according to importance; it gathers where a subject is already contested and is absent everywhere else, so the papers that most need scrutiny are the ones least likely to attract it. What the evidence supports is a smaller claim: that review is poor at ranking and tolerable at filtering, and that journals have advertised the first while performing the second. A system honest about that would keep the filter and stop treating the appearance of a paper in a particular journal as a statement about its quality.