--- title: "Interrater reliability" description: "Interrater reliability is the degree to which two independent raters judging the same person or the same piece of work arrive at the same score. It sets a ceiling on everything built from those ratings: a criterion no two raters agree on cannot be predicted well by any test, however good the test is." canonical: https://www.assessall.com/guides/glossary/i/interrater-reliability updated: 2026-09-28 source: AssessAll --- # Interrater reliability _Also known as: Inter-rater reliability, Rater agreement._ Interrater reliability is the degree to which two independent raters judging the same person or the same piece of work arrive at the same score. It sets a ceiling on everything built from those ratings: a criterion no two raters agree on cannot be predicted well by any test, however good the test is. ## The number, and the split inside it that matters Supervisory ratings of job performance are the criterion most selection research is validated against, and their interrater reliability has been cumulated twice. Viswesvaran, Ones and Schmidt reported 0.52 for overall job performance in 1996. Salgado and Moscoso, cumulating 219 independent coefficients over a total sample of 43,203 people in Frontiers in Psychology in 2019, reported an observed 0.56, and 0.61 after correction for range restriction. The finding worth acting on is not the headline figure but the split by why the rating was collected. Observed interrater reliability was 0.45 when the ratings were made for administrative purposes and 0.61 when they were made for research. After correction the research figure rises to 0.69 — and the administrative figure stays at 0.45. The correction that lifts one does nothing for the other. Administrative ratings are the ones organisations actually run on: the appraisal that feeds a bonus, the talent-review placement, the readiness call before a promotion. So the ratings used to make decisions about people are, in the literature, the least reliable version of the rating that exists — and the gap is not a measurement artefact that a statistical correction removes. ## How many raters it takes The reliability of an average of several raters is higher than the reliability of any one of them, and the relationship is the [Spearman–Brown formula](https://www.assessall.com/guides/glossary/s/spearman-brown-formula) applied to raters rather than to items: k times r, divided by one plus k minus one times r, where r is the reliability of a single rater and k is the number of raters. Run it at 0.45, the administrative figure, and the practical answer falls out. One rater is 0.45. Two are 0.62. Three are 0.71. Five are 0.80. Nothing about the rating form changed across those four lines. This is why adding a second and third independent rater is almost always a better investment than redesigning the scale, and why a single-manager judgement should not be the sole basis for a promotion decision. Extend the same arithmetic and the curve flattens where you would expect: four raters are 0.77, six are 0.83, eight are 0.87. Each rater after the third buys less than the one before, which is the argument for a floor of three rather than a target of ten — and it is why a multi-rater instrument that merges a group of two into a combined view is not being timid, it is declining to print a figure carrying about 0.62. None of this reaches the artefact, and that is checkable. Nine published sample 360 and multi-rater reports were read at source on 15 September 2026 — Mercer Mettl, three SHL templates, Hogan through its UK distributor, DecisionWise, Appraisal360, Info-Tech, hr-survey.com and 360feedback.app. Several print the spread of the raters’ answers, and Hogan’s goes furthest, tabling a standard deviation per item with a plain-English explanation of what it means. Not one of the nine prints a reliability coefficient, a standard error of measurement or a confidence interval, so nothing on any of those pages distinguishes a bar built from one rater from a bar built from six — and both are set to one decimal place in the same typeface. [A panel-by-panel walkthrough of one such report](https://www.assessall.com/guides/reports/360-feedback-report) shows exactly where that distinction would sit. ## What it does to a validity coefficient An unreliable criterion drags down any validity coefficient measured against it, which is why published validities are often corrected for criterion unreliability before being compared. That correction is legitimate and it is also where a great deal of the field's inflation has come from, because the size of the correction depends entirely on which reliability figure is fed into it. So when a vendor quotes a validity coefficient, two questions decide whether the number means anything: what the criterion was, and which interrater reliability was used to correct it. A validity corrected using a research-grade 0.69 will look considerably better than the same study corrected using the 0.45 that describes the ratings the client actually keeps. ## The 2024 update, and what it does to everything above There is a third cumulation, published after the two this entry has cited since it was written, and it changes the advice rather than the arithmetic. Zhou, Sackett, Shen and Beatty report an updated meta-analysis in the *Journal of Applied Psychology* (2024) over 132 independent samples, using a weighting method chosen so that a few very large studies cannot dominate the result. They put the interrater reliability of supervisory performance ratings at **0.65** — above the 0.56 of the 2019 cumulation and well above the 0.52 the market has quoted since 1996. The finding they lead on is not the headline figure. It is that the coefficient varies by the kind of job being rated: **0.57 for managerial positions against 0.68 for non-managerial ones**, converging with a 1997 result they describe as important and often neglected. Their recommendation follows directly and it is uncomfortable for everyone who quotes a single number, this page included: they advise against using an overall grand mean of interrater reliability at all, and recommend job-specific or local reliabilities instead. Read that against the rater arithmetic above and it has a price attached. At 0.68, two independent raters reach 0.81. At 0.57, the managerial figure, two raters reach 0.73 and it takes four to pass 0.80. The target whose rating is hardest to collect — a manager, rated by people who report to them — is the one that needs the most raters, and a 360 cycle is almost always run on a manager. Three cautions, so this does not simply become a newer number to quote. The 2024 estimate is an overall figure across rating purposes; it does not retire the purpose split, which is the part of the 2019 paper this site argues from and which the 2024 authors also examined. Its full text is paywalled and only its abstract was read, which is recorded on the [evidence entry for this literature](https://www.assessall.com/guides/evidence/salgado-moscoso-2019-interrater-reliability). And every coefficient in all three cumulations describes **two supervisors** — none of them measures peer, subordinate or self ratings, so the figures used on this site to argue for a rater floor in a multi-rater cycle are borrowed from a different rater population, and that is stated rather than smoothed over. --- Source: https://www.assessall.com/guides/glossary/i/interrater-reliability