What is Reliability?
Reliability is the consistency of a measurement: how much of the variation in scores is signal rather than noise. It is reported as a coefficient between 0 and 1, and it comes in several distinct kinds — internal consistency, test–retest, parallel forms and inter-rater — which answer different questions and are not interchangeable.
Which kind is being quoted
Internal consistency asks whether the items agree with each other in a single sitting. Test–retest asks whether the same people get the same result on a second occasion. Parallel forms asks whether two versions of the test rank people the same way. Inter-rater asks whether two humans scoring the same response agree.
A vendor who says 'reliability of 0.89' without saying which kind has not answered the question. For a behavioural questionnaire used to advise someone about their working style, test–retest is the figure that matters, and it is usually the lowest and the least often published.
Reliability caps validity
A test cannot correlate with an outcome more strongly than its own consistency allows; the theoretical ceiling on a validity coefficient is the square root of the product of the two measures' reliabilities. Unreliable measurement is therefore not merely untidy, it puts a hard limit on how useful the instrument can be.
The reverse does not hold. A perfectly reliable measure of the wrong thing is perfectly consistent and perfectly useless, which is why reliability is a precondition for validity and never a substitute for it.
Is there a threshold, and who says so
Most published thresholds are textbook convention rather than regulation, and vendors quote them as though they were rules. One exception is worth knowing because it is free to read and comes from the body that hires for the United States federal government: the Office of Personnel Management’s assessment strategy guidance states that "for most assessment applications, reliabilities above .70 are likely to be regarded as acceptable".
The same page defines reliability operationally — "assessment reliability is demonstrated by the consistency of scores obtained when the same applicants are reexamined with the same or equivalent form of an assessment" — which is a test–retest framing, not an internal-consistency one. A vendor quoting a .70-plus alpha against OPM’s .70 is not comparing like with like.
OPM sets the companion figure for validity in the same place: "a validity coefficient of .30 or higher is generally considered useful for most circumstances". Those two numbers together are a more defensible benchmark than any figure in a vendor brochure, because a buyer can open the page and read them.
The ceiling reliability puts on validity
- Maximum observable correlation = √(reliability of the test × reliability of the criterion)
- Test .80, supervisor rating .52 → √(0.80 × 0.52) = √0.416 = 0.64
- Test .70, supervisor rating .52 → √(0.70 × 0.52) = √0.364 = 0.60
- Test .90, supervisor rating .30 → √(0.90 × 0.30) = √0.270 = 0.52
Substitute the reliabilities you were actually given. The third line is the useful one: an excellent test scored against a noisy criterion is capped near .52 however good the test is, which is why a reported validity should always be read next to the reliability of the thing it was validated against.
Not the same as validity
Reliability is about consistency; validity is about whether the inference is justified. A bathroom scale that reads four kilograms heavy every time is perfectly reliable and invalid for the use it is put to.
ValiditySources
Read next
Related terms
Check a selection process against the four-fifths rule
Free, no signup, computed in your browser — with the remedy, not just the verdict.
Last reviewed