How many candidates before a score means anything?

Two questions that look like one

“How many do we need?” is asked in assessment programmes about two different things, and the answers move in different directions.

The group question. Your pass rate, your mean score, your time-to-hire — these are estimates of something about a population, and every one of them comes with an interval that narrows as the sample grows. At a 35% pass rate you need roughly 350 completed sittings for a 95% interval of ±5 percentage points, about 90 for ±10, and about 30 for ±17. Collect four times as many and the interval halves, because it scales with the square root of n.

The person question.How much a single candidate’s score can be trusted is not a sample-size question at all. It is set by the instrument’s reliability and the spread of scores, through the standard error of measurement, and it does not improve by one point when the thousandth candidate sits the test.

Conflating the two is how a team ends up with a beautifully precise cohort dashboard sitting on top of individual decisions the measurement cannot support. The second panel of the calculator exists to keep the two apart.

Why the interval here is a Wilson interval

The formula most people were taught — p ± 1.96 × √(p(1−p)/n), the Wald interval — is unreliable at exactly the sample sizes assessment teams actually have. Its true coverage drops well below the nominal 95% for small n and for rates near 0 or 1, it can return bounds outside 0–100%, and when nobody passes it collapses to zero width, reporting perfect certainty from the least informative data possible.

Agresti and Coull set out the case against it in The American Statistician in 1998, under the title Approximate is better than “exact” for interval estimation of binomial proportions. The alternative they favour is the score interval Edwin Wilson published in the Journal of the American Statistical Association in 1927, which is what this calculator uses: it stays inside the range, it keeps close to its nominal coverage at small n, and it costs one extra line of arithmetic.

This matters more than it sounds. A team reporting “a 35% pass rate” on 40 sittings with a Wald interval understates its own uncertainty, and the number then gets quoted onward without the interval at all.

The standard error of measurement, and the number it hides

SEM = SD × √(1 − reliability). With a score standard deviation of 12 and a reliability of 0.82 — which is a respectable figure for a well-built commercial instrument — the SEM is about 5.1 points. A 95% band around any single observed score is therefore about ±10 points wide.

Now put a cut score three points below a candidate’s result. On those numbers the probability that their true score is actually below the bar is around 28%. Nothing about that candidate is borderline in the report; the number reads as a pass. The uncertainty is invisible unless someone computes it, which is the entire argument for printing a band rather than a point.

The calculator also returns Kelley’s regressed true-score estimate — cohort mean + reliability × (observed score − cohort mean). Because measurement error inflates extreme scores in both directions, the best estimate of a true score always sits closer to the mean than the observed one. It is an uncomfortable output: it says the top of your shortlist is probably not as far ahead as the ranking implies, and the bottom is probably not as far behind.

When the band does straddle the bar, lengthening the test is the honest remedy and the Spearman–Brown prophecy formula prices it — going from α = 0.82 to α = 0.90 needs roughly twice the items, and reaching 0.95 needs about four times as many. That is a real cost, and it is the right thing to weigh against widening a borderline band instead.

The 0.70 rule is a misquotation

“Cronbach’s alpha above 0.70 is acceptable” is the most repeated sentence in assessment procurement, and it is a misreading of its own source. In Psychometric Theory (2nd ed., 1978, pp. 245–246) Nunnally recommended 0.70 for the early stages of research, where the interest is in correlations and group differences. For applied settings where important decisions turn on an individual’s score he was explicit that a reliability of 0.90 is the minimum that should be tolerated and 0.95 the desirable standard.

Lance, Butts and Michels traced how this and three other cutoffs became folklore in The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say? (Organizational Research Methods 9(2), 2006, 202–220). The pattern is always the same: a conditional recommendation gets cited, the condition falls away, and the number survives on its own.

Two practical consequences. First, ask a vendor which decision their reliability figure was computed for — an alpha of 0.78 is fine evidence for a cohort report and thin evidence for a rejection. Second, alpha is a lower bound on reliability only under assumptions most real tests violate; Klaas Sijtsma set out the case against relying on it in Psychometrikain 2009 (74(1), 107–120), and McDonald’s ω is the better figure to ask for when it is available.

What to do with the numbers this returns

1. Decide the precision before you look at the rate.“We need to know the pass rate to within five points” is a decision about how the number will be used. Choosing it after seeing the interval is how targets get set to whatever the data happened to support.
2. Report counts, not rates, under about 30.“9 of 26” is honest and legible. “34.6%” on the same data implies a precision that is not there.
3. Suppress cuts below 50 rather than caveating them. A footnote does not stop a thin subgroup figure being screenshotted into a board pack. If you cannot report a role, a site or a demographic group at n ≥ 50, do not break it out.
4. Print a band on every individual score you act on. If the band straddles the cut, the score has not made the decision and something else has to.
5. Route the borderline to a human, deliberately. Set the width from the SEM rather than from a round number, and write down what the human is supposed to look at.
6. Re-check after any change to the instrument. A new item set, a new rubric, a re-trained grader or a new delivery mode moves the distribution against a cut score that nobody re-examined. Treat it as a new instrument and a new sample.

Where these thresholds sit inside AssessAll

The same arithmetic decides what the platform will and will not say, and the numbers are constants in the codebase rather than positions in a brochure.

• Borderline results go to a person. On AssessAll Certified, a composite within 3 points of the nearest level cut is flagged borderline and escalated to human review rather than being resolved by the arithmetic. That is precisely the case the second panel above describes.
• A credential’s validity report is stamped DRAFT below n = 200. Under that many scored sittings the report says INSUFFICIENT SAMPLE and every band claim in it is labelled a design prior — a defensible starting point set by design, not a cut score derived from data.
• Item calibration will not run below 200 responses per item, and the adaptive engine stops a section when the standard error on the ability estimate reaches 0.30 — a precision rule, not an item-count rule.
• No benchmark is published at all. The internal gate requires at least 400 completed sittings on the single assessment being reported, from at least 10 distinct organisations, with no organisation contributing more than a quarter of the sample, and suppresses any reported cut below n = 50.

What that means in practice, plainly: AssessAll has not passed that gate on any instrument, so there is no AssessAll score benchmark, industry norm or national percentile — and any that appear here later will carry their sample size and their reliability on the same page. Where the platform reports a percentile inside a hiring drive it is a cohort percentile: the share of scored candidates in that same pipeline at or below this one. Not a national norm, and never presented as one.

Frequently asked questions

Sources

• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158), 209–212.
• Agresti, A., & Coull, B. A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. The American Statistician 52(2), 119–126.
• Nunnally, J. C. (1978). Psychometric Theory (2nd ed.), pp. 245–246. McGraw-Hill.
• Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: what did they really say? Organizational Research Methods 9(2), 202–220.
• Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika 74(1), 107–120.
• Spearman, C. (1910) and Brown, W. (1910), British Journal of Psychology 3 — the prophecy formula.
• AERA, APA & NCME (2014). Standards for Educational and Psychological Testing, chapter 2 (reliability and errors of measurement).