Ranking the Universities

Published rankings of universities are widely criticised, and most of the criticism misses. The usual complaint is that the weights are arbitrary, which is true and comparatively harmless, since a reader who disagrees with the weights can look at the components. A more serious objection is that the components are chosen for availability rather than importance, so that what is measured is what every institution already reports.

[A] The deepest problem is different again, and it is that the ranking is not a measurement but an intervention. Once a table is published, institutions manage the quantities it contains, and a quantity that is being managed stops describing what it once described. [B] Staff-to-student ratios improve when teaching is transferred to people who are not counted as staff; graduate earnings rise when admissions shift towards subjects whose graduates already earn well. [C] Neither change is fraud, and both are exactly what a rational administrator should do. [D] The consequence is that a table which was informative when it was first published becomes less informative every year that anybody pays attention to it, and the better it is designed, the faster this happens, because a well-designed table is one that institutions take seriously.