What is Interrater reliability?

Also called Inter-rater reliability, Rater agreement

Interrater reliability is the degree to which two independent raters judging the same person or the same piece of work arrive at the same score. It sets a ceiling on everything built from those ratings: a criterion no two raters agree on cannot be predicted well by any test, however good the test is.

The number, and the split inside it that matters

Supervisory ratings of job performance are the criterion most selection research is validated against, and their interrater reliability has been cumulated twice. Viswesvaran, Ones and Schmidt reported 0.52 for overall job performance in 1996. Salgado and Moscoso, cumulating 219 independent coefficients over a total sample of 43,203 people in Frontiers in Psychology in 2019, reported an observed 0.56, and 0.61 after correction for range restriction.

The finding worth acting on is not the headline figure but the split by why the rating was collected. Observed interrater reliability was 0.45 when the ratings were made for administrative purposes and 0.61 when they were made for research. After correction the research figure rises to 0.69 — and the administrative figure stays at 0.45. The correction that lifts one does nothing for the other.

Administrative ratings are the ones organisations actually run on: the appraisal that feeds a bonus, the talent-review placement, the readiness call before a promotion. So the ratings used to make decisions about people are, in the literature, the least reliable version of the rating that exists — and the gap is not a measurement artefact that a statistical correction removes.

How many raters it takes

The reliability of an average of several raters is higher than the reliability of any one of them, and the relationship is the Spearman–Brown formula applied to raters rather than to items: k times r, divided by one plus k minus one times r, where r is the reliability of a single rater and k is the number of raters.

Run it at 0.45, the administrative figure, and the practical answer falls out. One rater is 0.45. Two are 0.62. Three are 0.71. Five are 0.80. Nothing about the rating form changed across those four lines. This is why adding a second and third independent rater is almost always a better investment than redesigning the scale, and why a single-manager judgement should not be the sole basis for a promotion decision.

Extend the same arithmetic and the curve flattens where you would expect: four raters are 0.77, six are 0.83, eight are 0.87. Each rater after the third buys less than the one before, which is the argument for a floor of three rather than a target of ten — and it is why a multi-rater instrument that merges a group of two into a combined view is not being timid, it is declining to print a figure carrying about 0.62.

None of this reaches the artefact, and that is checkable. Nine published sample 360 and multi-rater reports were read at source on 15 September 2026 — Mercer Mettl, three SHL templates, Hogan through its UK distributor, DecisionWise, Appraisal360, Info-Tech, hr-survey.com and 360feedback.app. Several print the spread of the raters’ answers, and Hogan’s goes furthest, tabling a standard deviation per item with a plain-English explanation of what it means. Not one of the nine prints a reliability coefficient, a standard error of measurement or a confidence interval, so nothing on any of those pages distinguishes a bar built from one rater from a bar built from six — and both are set to one decimal place in the same typeface. A panel-by-panel walkthrough of one such report shows exactly where that distinction would sit.

What it does to a validity coefficient

An unreliable criterion drags down any validity coefficient measured against it, which is why published validities are often corrected for criterion unreliability before being compared. That correction is legitimate and it is also where a great deal of the field's inflation has come from, because the size of the correction depends entirely on which reliability figure is fed into it.

So when a vendor quotes a validity coefficient, two questions decide whether the number means anything: what the criterion was, and which interrater reliability was used to correct it. A validity corrected using a research-grade 0.69 will look considerably better than the same study corrected using the 0.45 that describes the ratings the client actually keeps.

The 2024 update, and what it does to everything above

There is a third cumulation, published after the two this entry has cited since it was written, and it changes the advice rather than the arithmetic. Zhou, Sackett, Shen and Beatty report an updated meta-analysis in the Journal of Applied Psychology (2024) over 132 independent samples, using a weighting method chosen so that a few very large studies cannot dominate the result. They put the interrater reliability of supervisory performance ratings at 0.65 — above the 0.56 of the 2019 cumulation and well above the 0.52 the market has quoted since 1996.

The finding they lead on is not the headline figure. It is that the coefficient varies by the kind of job being rated: 0.57 for managerial positions against 0.68 for non-managerial ones, converging with a 1997 result they describe as important and often neglected. Their recommendation follows directly and it is uncomfortable for everyone who quotes a single number, this page included: they advise against using an overall grand mean of interrater reliability at all, and recommend job-specific or local reliabilities instead.

Read that against the rater arithmetic above and it has a price attached. At 0.68, two independent raters reach 0.81. At 0.57, the managerial figure, two raters reach 0.73 and it takes four to pass 0.80. The target whose rating is hardest to collect — a manager, rated by people who report to them — is the one that needs the most raters, and a 360 cycle is almost always run on a manager.

Three cautions, so this does not simply become a newer number to quote. The 2024 estimate is an overall figure across rating purposes; it does not retire the purpose split, which is the part of the 2019 paper this site argues from and which the 2024 authors also examined. Its full text is paywalled and only its abstract was read, which is recorded on the evidence entry for this literature. And every coefficient in all three cumulations describes two supervisors — none of them measures peer, subordinate or self ratings, so the figures used on this site to argue for a rater floor in a multi-rater cycle are borrowed from a different rater population, and that is stated rather than smoothed over.

What a second and third rater buy you — run on both of the figures in circulation

On the 2019 administrative figure, r = 0.45. Two raters: (2 × 0.45) ÷ (1 + 0.45) = 0.62. Three: 1.35 ÷ 1.90 = 0.71. Four: 1.80 ÷ 2.35 = 0.77. Five: 2.25 ÷ 2.80 = 0.80.
On the 2024 cumulation's managerial figure, r = 0.57. Two raters: (2 × 0.57) ÷ (1 + 0.57) = 0.73. Three: 1.71 ÷ 2.14 = 0.799. Four: 2.28 ÷ 2.71 = 0.84.
Raters needed to pass 0.80: five on the older figure, four on the newer one. Three raters reach 0.799 on the 2024 managerial coefficient, which is not 0.80.

Two things are worth taking from running it twice rather than once. The practical answer barely moves — a one-rater judgement is weak on either figure, and the fix is more independent raters rather than a better form. But the threshold does move, and in the direction that shows why the choice of coefficient is not academic: the same 0.80 target needs five raters on the administrative 0.45 and four on the 2024 managerial 0.57. Any vendor or programme that quotes a rater minimum should be asked which coefficient it was derived from.

Not the same as internal consistency

Internal consistency describes whether the items of one instrument hang together for one respondent. Interrater reliability describes whether two people scoring the same performance agree. A rating form can be internally consistent and still produce a different verdict depending on who filled it in.

Cronbach's alpha

How AssessAll handles it

The 360 report will not display a rater group's average until at least three people in that group have responded; below the floor the group is merged into a combined others, and if that is still short the figure is withheld rather than shown. The floor is set in code as at least three and cannot be configured lower. It was built for rater anonymity rather than for reliability, but the arithmetic above lands in nearly the same place: three raters take the 2024 managerial coefficient to 0.799 and two only reach 0.73, so the floor that protects the raters is also approximately where the number starts being worth printing. The two reasons are independent and they agree.

Sources

More terms beginning with I

Incremental validity Item analysis Item response theory

Check a selection process against the four-fifths rule

Free, no signup, computed in your browser — with the remedy, not just the verdict.

Open the calculator

Last reviewed 2026-09-28