What Counts as Repeating a Study

When a result fails to appear a second time, the argument that follows is rarely about the data. It is about whether the second study was the same study. This is not evasion on the part of the original authors, or not always: two experiments can differ in a hundred ways, and only somebody with a theory can say which of those differences ought to matter.

[A] That is what makes the disagreement so difficult to settle. A direct replication changes nothing that the theory says is relevant, but the theory is precisely what is in dispute, so its author is entitled to say afterwards that the season, the country or the age of the participants was relevant all along. [B] The reply that this is unfalsifiable is too quick, because sometimes the moderating condition is real and the objection is correct. [C] What separates a legitimate qualification from a rescue is whether it was stated in advance, and that is a matter of record rather than of judgement. [D] A discipline that registers its predictions before collecting data converts an argument that cannot be won into one that can, not by making anybody more honest but by removing the opportunity to decide afterwards what the theory had always said.