What is Standard setting?
Standard setting is the formal process of deciding where a cut score belongs. It uses a documented method — Angoff, bookmark, contrasting groups — with a panel, a written definition of the borderline candidate, and a record of the judgements, so that the resulting threshold can be defended rather than merely asserted.
The methods, in one line each
Angoff: judges estimate, item by item, the probability that a borderline candidate answers correctly, and the estimates are summed. Bookmark: items are ordered by difficulty and judges place a bookmark where a borderline candidate would stop succeeding. Contrasting groups: people already classified as competent or not competent by independent evidence sit the test, and the cut is placed where the two distributions are best separated.
None of them discovers a true standard. Each of them documents a defensible judgement, which is the most that can honestly be claimed and considerably more than most organisations have.
One thing a standard-setting panel may not do
In the United States the panel’s output is the only lever available, because adjusting the result afterwards for some groups is prohibited by statute. 42 U.S.C. §2000e-2(l), added by section 106 of the Civil Rights Act of 1991 and read at source on 16 September 2026, states in full: “It shall be an unlawful employment practice for a respondent, in connection with the selection or referral of applicants or candidates for employment or promotion, to adjust the scores of, use different cutoff scores for, or otherwise alter the results of, employment related tests on the basis of race, color, religion, sex, or national origin.”
That forecloses the shortcut most often proposed when a threshold produces an uneven pass rate, which is to move the line for the affected group. It does not foreclose setting different standards for different jobs, validated for those jobs — that is ordinary practice and a different thing entirely.
What it leaves is the harder work, and standard setting is where that work happens: re-examine whether the construct the test measures is the one the job actually requires, look at individual items for differential functioning, reconsider how much weight the test carries in the decision, and check whether the threshold was ever justified against performance in the first place. This describes one jurisdiction and is not legal advice.
The record is the deliverable
What survives a challenge is not the number but the file: who was on the panel and why they were qualified, the borderline definition they worked to, the rounds of estimates, what data was shown between rounds, the final recommendation, and the business decision that accepted or adjusted it.
Set a review date at the same time. A standard set against a job that has since changed is a standard for a job nobody does.
Two defensible methods, one test, two different answers
The claim that a method documents a judgement rather than discovering a truth sounds like hedging until somebody runs two methods on the same test and publishes both numbers. Yousefi Afrashteh did exactly that in BMC Medical Education in 2021, on a 20-item medical performance test taken by 190 examinees, with a 12-judge Angoff panel and an 11-judge bookmark panel working from the same test and the same borderline definition.
The Angoff panel set the cut at 17.67 and 106 of the 190 examinees passed, a rate of 55.78%. The bookmark panel set it at 18.82 and 88 passed, a rate of 46.31%. A threshold 1.15 raw points higher moved eighteen people — 9.47 percentage points of the cohort, about one candidate in ten — from pass to fail, with no change to the test, the items or the candidates.
The paper does not conclude that the methods disagree systematically, and neither does this entry: a chi-square test on the two pass rates returns 1.76 with p = 0.20, which at this sample size is no detectable difference at all. The finding worth carrying is narrower and more uncomfortable. Method choice is a defensible, documented, expert judgement, and it is also a live parameter with a body count. Nobody in that cohort experienced it as a methodological preference.
Two reporting rules follow. Name the method in the file, because a cut score without its method is a number whose provenance cannot be reconstructed. And if a threshold sits within a point or two of where an alternative method would have put it, say so in the record, because that is the range within which the decision was a choice rather than a measurement. One process note about this source, on the site's own quote-what-you-cite rule: the paper's abstract prints the bookmark pass rate as 41.36% while its results section gives 46.31% for 88 of 190 examinees. The results figure is the one arithmetic supports and it is the one used here; the discrepancy is recorded rather than smoothed over.
How many candidates the choice of METHOD moves, on one published dataset
- One 20-item medical performance test, 190 examinees, two expert panels
- Angoff panel (12 judges) → cut score 17.67 → 106 pass = 55.78%
- Bookmark panel (11 judges) → cut score 18.82 → 88 pass = 46.31%
- Difference in the threshold: 1.15 raw points
- Difference in the cohort: 18 candidates, or 9.47 percentage points
- Candidates displaced per raw point of cut score: about 16 in this cohort
- Chi-square on the two pass rates: 1.76, p = 0.20 — not statistically significant
Read the last line with the ones above it, because it is what stops this being an argument that one method is wrong. At n = 190 the two pass rates are statistically indistinguishable; the paper claims no systematic difference and neither does this page. What is not indistinguishable is the eighteen people. A defensible process, run twice by qualified panels on the same test, produced two different answers for one candidate in ten — and it is the employer, not the method, who carries that.
Not the same as norming
Norming describes how a population performed. Standard setting decides what performance is good enough. Norming is descriptive and empirical; standard setting is a judgement about consequences and cannot be derived from data alone.
Sources
Read next
Related terms
More terms beginning with S
Check a selection process against the four-fifths rule
Free, no signup, computed in your browser — with the remedy, not just the verdict.
Last reviewed