What is Criterion validity?

Concurrent and predictive designs

In a concurrent study, current employees take the test and their scores are correlated with their existing performance ratings. It is fast and it is the design most validity evidence actually uses, but the sample has already survived selection, promotion and attrition, so its range is narrower than an applicant pool's.

In a predictive study, applicants take the test, everyone is hired regardless of score, and performance is measured later. It matches how the test is used, and it is expensive and rare for exactly that reason.

The criterion is usually the weak link

Validity coefficients are bounded by the reliability of the outcome measure. Supervisor performance ratings, the criterion in most published studies, are themselves noisy and contaminated by tenure, likeability and rating leniency.

So a modest correlation may reflect a weak test, a weak criterion, or both, and a study reporting a validity coefficient without reporting the criterion's own reliability has left out half of the information needed to read it.

A coefficient is not an outcome, and two different sums convert it into one

Criterion validity is reported as a correlation, and a correlation is not a thing anyone can act on. Converting it into a practical result takes two further facts about your situation: the base rate, meaning the share of applicants who would perform satisfactorily if you hired at random, and the selection ratio, meaning the share of applicants you hire. Taylor and Russell published the conversion in 1939 and it has not been improved on. The same coefficient of 0.31, against a base rate of 50%, lifts the success rate among people hired to 71.5% if you hire one applicant in ten and to 60.0% if you hire one in two.

There is a second and older conversion that answers a slightly different question. Brogden showed in 1946 that where the outcome is left as a continuous measure rather than cut into satisfactory and unsatisfactory, the validity coefficient is literally the proportion of the maximum possible gain that the procedure captures: a test at 0.31 delivers 31% of what a perfect predictor would. That is a strikingly direct reading of a number usually described as hard to interpret.

The two answers do not agree, and the discrepancy is informative rather than a mistake. At a base rate of 50% and a selection ratio of one in ten, the success-rate conversion gives 43% of maximum and the continuous-criterion conversion gives 31%. Dichotomising performance into pass and fail changes what 'the gain' refers to. Any figure quoted as 'the improvement' should say which of the two it is.

Not the same as construct validity

Criterion validity asks whether the score predicts an outcome. Construct validity asks whether it measures the attribute claimed. A test can predict an outcome for reasons that have nothing to do with the construct it names.

Sources

• Sackett, P. R., Zhang, C., Berry, C. M. & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology, 107(11), 2040–2068
• Taylor, H. C. & Russell, J. T. (1939). The relationship of validity coefficients to the practical effectiveness of tests in selection: discussion and tables. Journal of Applied Psychology, 23(5), 565–578
• Brogden, H. E. (1946). On the interpretation of the correlation coefficient as a measure of predictive efficiency. Journal of Educational Psychology, 37(2), 65–76

Read next

• Selection utility calculator — turn a coefficient into a success rate
• How many candidates before a score means anything?
• What Sackett et al. (2022) actually says

Related terms

More terms beginning with C