What is Item response theory?
Also called IRT, Latent trait theory
Item response theory models the probability of a given answer as a function of the candidate's underlying ability and the question's own parameters — usually difficulty, discrimination and, for multiple choice, guessing. Because ability and item difficulty are placed on the same scale, scores from different sets of questions remain comparable.
What the models are
The one-parameter model, closely associated with the Rasch model, describes each item by difficulty alone. The two-parameter model adds discrimination, so items differ in how sharply they separate people. The three-parameter model adds a lower asymptote for guessing, which matters for multiple-choice items where a candidate who knows nothing still has a floor.
Choosing among them is a modelling decision with consequences: more parameters fit better and need substantially more data to estimate stably.
What it buys, and what it costs
The payoff is invariance. Once items are calibrated onto a common scale, two candidates who saw different questions can be compared, new items can be added to a bank without re-standardising the whole test, and the next item can be selected adaptively to maximise information at the candidate's current estimate.
The cost is data and discipline. Calibration needs large samples per item, the model's assumptions — chiefly that a single dimension accounts for the responses, and that answering one item does not change the odds on another — need checking rather than assuming, and a bank calibrated on one population may not hold on another.
"Large samples per item" — the actual numbers, and why they are per item
The sentence above is the one every vendor writes and almost none quantifies, so here are figures anyone can check. John Michael Linacre's table in Rasch Measurement Transactions 7(4), 328 (1994) sets the floor at the level of a single item: "at least 8 correct responses and 8 incorrect responses are needed for reasonable confidence that an item calibration is within 1 logit of a stable value". To halve that tolerance to ±½ a logit at 95% confidence takes 64 to 144 respondents, with 100 recommended; for "definitive or high stakes" use the table recommends a minimum of 250.
Those are Rasch figures, and Rasch is the cheapest model in the family — one parameter per item. A two-parameter model estimates discrimination as well, and a three-parameter model adds guessing, so each step up multiplies what the same bank needs. Schroeders and Gnambs, writing a sample-size tutorial for item response theory in 2025, record the textbook conventions as "at least 250 or 500 respondents" or "a sufficient ratio of respondents to model parameters, such as 10:1 or 20:1" — and then spend the paper arguing that every one of those rules of thumb is unsafe, because requirements depend on item type, estimation method, dimensionality, trait distribution and test design, so "findings from published simulation studies may not generalize to the unique circumstances of a planned study".
The word that does the work in all of this is per item. A platform with 10,000 registered users has not calibrated anything; a platform where 250 people have answered each one of 400 items has, and the second number is 100,000 item responses. That is the arithmetic that decides whether an item bank can support adaptive delivery at all, and it is why an assessment catalogue can grow much faster than its psychometrics can.
It applies here too. AssessAll's internal research-readiness gate is 400 completed sittings on a single form, from at least ten organisations with no organisation above a quarter of the sample, and no AssessAll instrument has yet cleared it — which is why the site publishes no norm, no benchmark and no reliability coefficient. Ask any vendor quoting item parameters how many sittings sit behind each item, not how many users they have.
What a bank costs in responses
- Linacre (1994): ±1 logit at 95% confidence ≈ 30 respondents per item
- ±½ logit at 95% confidence ≈ 100 per item; definitive / high stakes ≈ 250 per item
- A 400-item bank at the high-stakes figure = 400 × 250 = 100,000 item responses
- At 40 items per sitting, that is 2,500 complete sittings — for a Rasch model, the cheapest of the three
A two- or three-parameter model estimates more per item and needs more again. The number to ask a vendor for is responses per item, not registered users — the two can differ by three orders of magnitude.
Not the same as classical test theory
Classical test theory works with total scores and treats item statistics as properties of a sample. Item response theory models each response individually and places people and items on one scale. Most commercial assessments still report classical statistics; the ones that support adaptive delivery cannot.
Sources
- Embretson, S. E. & Reise, S. P. (2000). Item Response Theory for Psychologists
- Linacre, J. M. (1994). Sample Size and Item Calibration Stability. Rasch Measurement Transactions 7(4), 328 — read at source 23 September 2026
- Schroeders, U. & Gnambs, T. (2025). Sample-Size Planning in Item-Response Theory: A Tutorial — open author copy read 23 September 2026
Related terms
More terms beginning with I
Check a selection process against the four-fifths rule
Free, no signup, computed in your browser — with the remedy, not just the verdict.
Last reviewed