Computerized adaptive testing (CAT) picks each next question from the candidate's answers so far, aiming every item near their estimated ability. Because a well-matched item carries more statistical information than one far too easy or hard, a CAT reaches a given precision in fewer questions — but only if it draws on a large, pre-calibrated item bank.
That last clause is where most adaptive-testing projects die: the algorithm is the cheap part, and the bank is the product.
The ratio two of the world's largest adaptive programmes actually run
The military's CAT-ASVAB was the first large-scale adaptive battery used in a high-stakes setting, and its development drew on data from over 400,000 test-takers across roughly 20 years. Its operational Forms 1 and 2 are documented in the ASVAB technical bulletins.
Scored items administered per subtest
- General Science, Arithmetic Reasoning, Word Knowledge, Mathematics Knowledge, Electronics Information, Mechanical Comprehension: 15 items each
- Paragraph Comprehension, Auto Information, Shop Information: 10 items each
Bank size behind those subtests (Form 1)
- Word Knowledge: 95 items
- Arithmetic Reasoning: 94 items
- Mathematics Knowledge: 84 items
- General Science: 72 items
- Mechanical Comprehension: 64 items
- Paragraph Comprehension: 50 items
A 15-item test standing on a 95-item bank. Roughly six times the content you will ever show one person.
PISA 2022 lands in the same territory from a different direction. The OECD technical report describes a three-stage adaptive mathematics design built from three item sets of 78 items each, generating 192 possible paths. Any one student answered 28 to 30 items, median 30. Again: around 30 items delivered, over 200 held in reserve.
What calibration costs, in people
Writing items is not calibrating them. Item response theory needs each item's difficulty and discrimination estimated from real responses on a common scale before the selection algorithm can use it.
The ASVAB record shows the bill. About 3,600 items were written — roughly 400 for each of nine content areas — and pretested on about 21,000 recruits, yielding roughly 300 responses per item. That pretest was only a screen. The 2,118 survivors were assembled into 43 booklets and administered to 137,000 applicants across 63 processing stations, leaving about 116,000 usable cases and roughly 2,700 responses per item for the calibration that counted.
Refreshing the bank cost the same order of magnitude: for Forms 3 and 4, 5,200 new items were authored over a 13-month contract, 2,770 went to tryout across 56 booklets with 151,811 examinees, and 2,598 survived screening for differential item functioning and factor structure.
Generative AI has collapsed the drafting step. In a 2026 Behavior Research Methods paper, the AI-GENIE pipeline had GPT-4o produce 320 candidate personality items and trimmed them to 181 using network psychometrics — but the empirical validation still rested on five US samples totalling 4,964 respondents. Item writing is now fast. Item calibration still costs a few hundred to a few thousand real answers per item, and no model gets you out of that.
Three problems adaptive testing creates for you
It makes your test predictable unless you deliberately degrade it
Pure maximum-information selection has an uncomfortable property, stated plainly in the ASVAB bulletin: for any ability estimate only one item is optimal, so candidates starting from the same assumed ability all get the same first item, two possible second items, and so on. The most precise adaptive test is also the most predictable one. Operational programmes therefore add exposure controls that intentionally pass over the best item some fraction of the time, buying security with a small loss of precision.
Regulators have noticed. US federal rules for adaptive tests used in adult education require that "the size of the item pool and the method of item selection must ensure negligible overlap in items" across a candidate's alternate administrations, and that scores from those administrations be equivalent in meaning (34 CFR 462.13). If your retake policy lets someone sit again next week off a thin bank, you are not retesting them. You are showing them the answers.
Nobody can recount the score
The ASVAB evaluation recorded a specific confusion among recruiters: it was hard to explain how a test with 16 items could produce a number-correct score of 35. Adaptive scores come from an ability estimate on a latent scale, not from counting ticks. Candidates and hiring managers both lose the ability to audit a result by eye, so your score report has to carry more explanatory weight than a fixed-form report ever did.
Strong candidates experience it as brutal
By design, a well-functioning CAT converges on items a candidate gets right about half the time. High-ability applicants in the ASVAB rollout reported the test felt genuinely difficult, and some recruiters concluded their candidates would fare better on the paper version. The measurement was fine; the experience needed explaining. That is a briefing problem, not a psychometric one, but it shows up in drop-off.
Variable length is a feature, and it has rules
The NCLEX nursing licensure exam is the cleanest public example of variable-length CAT. Under NCSBN's published design, the exam stops as soon as it is 95% certain the candidate's ability sits clearly above or clearly below the passing standard. The 2026 RN test plan sets the boundaries: a minimum of 85 items, a maximum of 150 within a five-hour window, of which 15 are unscored pretest items being calibrated for future use. Candidates near the cut score run long; clear passes and clear fails finish early.
Note what that design concedes. Even a mature CAT cannot decide every candidate efficiently — those at the boundary consume the maximum, and the programme publishes a fallback rule rather than pretending otherwise. PISA 2022 went further: rather than deliberately misrouting a share of students to keep its scale anchored, the OECD gave 25% of them a straight linear, non-adaptive form. The most sophisticated adaptive programme in educational measurement still runs a non-adaptive arm.
When a fixed form is the right answer
For most employers screening most roles, it is.
A fixed form is the better choice when your annual volume is in the hundreds rather than the tens of thousands, because you will never accumulate enough responses per item to calibrate a bank worth adapting from. It is better when the construct is a work sample or a judgement scenario rather than a unidimensional knowledge trait, since the IRT models underneath most CATs assume one dimension and local independence — the reason ASVAB's developers split Auto and Shop into separate pools and rewrote reading items to one question per passage. And it is better when you need an explainable result more than a short one.
The realistic middle path for a volume employer is not adaptive at all: several parallel fixed forms drawn at random from a shared bank, item statistics monitored for drift, and a cut score set by a documented method. That buys most of the exposure control at a fraction of the calibration cost. AssessAll is built for that pattern — AI-graded scenario responses, proctoring with integrity bands to catch the leakage exposure control is meant to prevent, and pay-as-you-go credits at ₹30 / US$0.50, so running three forms instead of one is a cost decision rather than a licensing negotiation. Where a genuine calibrated bank is warranted, that is custom assessment work with a calibration plan attached, not a feature toggle.
Standard setting still sits outside the algorithm
Adaptivity changes which items a candidate sees. It does not tell you where the line goes. The Standards for Educational and Psychological Testing still require you to document how a passing score was derived, and a CAT's precision advantage only matters if it is concentrated where the decision happens — which is why NCLEX frames its stopping rule around certainty relative to the passing standard, not certainty about the score in general.
Ask any adaptive-testing vendor three things: how many calibrated items sit in the bank for this construct, how many responses per item were used to calibrate them, and what the standard error of measurement is at your cut score. Vague answers mean you are buying a shorter test, not a better one.
The takeaway: adaptive testing earns its keep at scale, where a bank of several hundred calibrated items is affordable and the saved minutes multiply across tens of thousands of candidates. Below that, parallel fixed forms off a monitored bank give you the security and defensibility you actually needed, and the honest shorter test is the one you can explain.