A Validation Study With 60 Hires Finds a Working Cognitive Test 26% of the Time

Statistical power in a criterion-related validation study is the probability that a genuinely predictive assessment produces a statistically significant validity coefficient in your sample. For a cognitive ability test with an operational validity of .31, a sample of 60 hires has roughly 26% power. Three times in four, a test that works looks like a test that failed.

That is not a rounding problem. It is the single most common reason a defensible assessment gets retired, and it happens before anyone collects a score, in the sample-size decision nobody wrote down.

The number you planned around has already been corrected

When a meta-analysis reports that work sample tests predict job performance at .33 or structured interviews at .42, those are operational validity estimates: corrected for unreliability in the performance criterion and for range restriction in the sample. Sackett, Zhang, Berry and Lievens (2022), in the Journal of Applied Psychology, revised the whole table downward after showing that earlier range-restriction corrections had been systematically too aggressive. Their revised figures assume a criterion reliability of .60 for job performance ratings.

Your study will not observe those numbers. It will observe a raw correlation between predictor scores and whatever criterion you managed to collect, in a sample consisting only of people you hired. Two attenuations apply, in this order:

• Criterion unreliability. Multiply the operational validity by the square root of your criterion reliability. At the .60 Sackett et al. assume, a .31 becomes .24. If the criterion is one supervisor's rating rather than an aggregate, reliability is lower and so is the product.
• Range restriction. Multiply again by your u-ratio, the standard deviation of predictor scores among hires divided by the standard deviation among applicants. At a moderate u of .70, the .24 becomes about .17.

So the quantity your significance test is actually hunting is roughly .17, not .31. That difference is the whole study.

What that does to the sample size you need

The figures below are computed, not cited: required sample for 80% power using Fisher's z transformation, two-tailed, alpha = .05, starting from each revised operational validity and attenuating at criterion reliability .60 and u = .70. Change the assumptions and the numbers move, which is the point of doing the arithmetic rather than inheriting a habit.

Structured interview (operational validity .42)

• Expected observed correlation: .23
• Cases needed for 80% power: 141
• Power at 100 cases: 65%

Work sample test (operational validity .33)

• Expected observed correlation: .18
• Cases needed for 80% power: 235
• Power at 100 cases: 44%

Cognitive ability test (operational validity .31)

• Expected observed correlation: .17
• Cases needed for 80% power: 268
• Power at 100 cases: 40%

Situational judgment test (operational validity .26)

• Expected observed correlation: .14
• Cases needed for 80% power: 385
• Power at 100 cases: 29%

Conscientiousness measure (operational validity .19)

• Expected observed correlation: .10
• Cases needed for 80% power: 730
• Power at 100 cases: 18%

For scale, Bosco and colleagues (2015) coded 147,328 correlations from 1,660 articles in the Journal of Applied Psychology and Personnel Psychology between 1980 and 2010. The median absolute correlation was .16, and the median sample size was 224. Applied psychology's typical effect is about the size of an attenuated validity coefficient, and its typical published sample is more than twice what most in-house validation studies assemble.

Significance was never the right target

Suppose you get lucky at 100 cases and observe .30, comfortably significant. The 95% confidence interval runs from .11 to .47. You cannot distinguish "marginal" from "excellent" at that width, which means you cannot set a cut score, estimate utility, or compare two instruments on that evidence. At 200 cases the interval narrows to roughly .17 to .42, still wide.

This is why the Principles for the Validation and Use of Personnel Selection Procedures treat precision and power as planning questions, not reporting afterthoughts. Before a study begins, "one should determine whether a large enough sample size can be obtained to meet the desired level of statistical power," and the report should "describe procedures and results of relevant power analyses." The Principles are blunter still about asymmetric caution: "Testing professionals should give at least equal attention to the risks of Type II error."

Regulators got there first. The Uniform Guidelines on Employee Selection Procedures require a validity coefficient significant at the .05 level under section 1607.14B(5), and define technical feasibility in section 1607.16 as requiring "an adequate sample of persons available for the study to achieve findings of statistical significance" alongside "a sufficient range of scores" and "unbiased, reliable and relevant measures of job performance." A study that cannot clear those conditions is not a weak criterion-related study. By the Guidelines' own definition it is not technically feasible, and the Guidelines say plainly that users are not required to hire people merely to make one possible.

Sizing the study before you run it

1. Pick the operational validity for the method, not the vendor's brochure figure. Published meta-analytic estimates for the method class are the honest prior.
2. Attenuate for your criterion. Multiply by the square root of criterion reliability. Estimate it from your own data if you have two independent raters; otherwise state the assumption explicitly.
3. Estimate your u-ratio. Predictor standard deviation among hires over standard deviation among applicants. If you have applicant-pool scores, this is arithmetic rather than guesswork.
4. Compute required N on the attenuated value. Fisher's z gets you there in two lines of code; Aguinis and Harden document how badly rules of thumb perform in its place, noting published power as low as .14 for small effects in some literatures.
5. Decide the interpretation in advance. Write down, before data collection, what a non-significant result will and will not be taken to mean. This is the step that prevents an underpowered null from being read as disconfirmation.
6. Report both corrected and uncorrected coefficients with confidence intervals, as the Principles direct.

AssessAll's pay-as-you-go credits at ₹30 or US$0.50 per assessment, with 100 free individual and 250 free corporate credits on signup, make piloting a cohort inexpensive. No price makes an underpowered study informative, which is why the sizing calculation belongs before the pilot, not after it.

When a local study is the wrong instrument

If the arithmetic says 268 cases and you have 70 annual hires in the role, a local criterion-related study is not the tool. The defensible alternatives are already recognised:

• Validity generalisation and transportability. Meta-analytic evidence plus a job-analysis demonstration that your role matches the roles in the cumulated research. The EEOC's own guidance on tests and selection procedures and sections 1607.7 and 1607.8 of the Guidelines contemplate relying on validity evidence from other studies.
• Content validity. For work samples and job knowledge tests that sample job content directly, a documented content-oriented strategy under section 1607.14C needs no criterion data at all.
• Synthetic validity. Build the validity estimate from job-component evidence when no single job supplies enough cases.

When is a local study still right? When you have the cases, when your criterion genuinely differs from the performance measures in the published literature, or when the instrument is bespoke enough that no cumulated evidence applies. Those are real conditions and they do occur; AssessAll's custom assessment work exists partly for the third. The error is not running local studies. It is running one that was arithmetically incapable of returning an answer, then treating its silence as a finding.

The takeaway

Before you collect a single score, attenuate the published validity for your criterion and your range restriction, then compute the power that attenuated value gives you at the sample you can realistically reach. If the answer is below 80%, choose a different validation strategy rather than spending a year generating a null result you will have to argue about.