What is Angoff method?

Why judgement rather than a percentile

Setting a pass mark at, say, the 60th percentile ties the standard to whoever happened to sit the test. If a weak cohort turns up, the bar falls with them. The Angoff method fixes the standard to the content instead: the question is not 'who is in the top 40%' but 'what would a minimally competent person be able to do'.

That makes it the appropriate family of method for certification, licensure and any decision where a fixed standard has to hold across cohorts and across years.

What a defensible Angoff study actually needs

A written definition of the borderline candidate, agreed before any estimating begins, is the single most important artefact — every judgement in the exercise is relative to it, and panels that skip it produce estimates that disagree for uninteresting reasons.

Then: a panel broad enough to represent the practice, at least two rounds with discussion between them, and — in the modified Angoff — real item statistics shown to judges before the second round, so that estimates can be reconciled with how candidates actually performed. Finally, the standard error of the panel's judgements should be reported: a cut score is itself a measurement and carries error like any other.

How broad is broad enough — the panel size, numbered

The paragraph above has asked for a panel broad enough to represent the practice without saying how many people that is, which is the question every organisation commissioning one actually has. A simulation study by Shulruf, Wilkinson, Weller, Jones and Poole in BMC Medical Education gives a figure: a panel of about fifteen judges provides a reliable and valid cut-score for a test consisting of 80 items or above. However, when the number of items is fewer than 15, it is recommended that no fewer than 20 judges are used. The relationship runs the way many people guess backwards — the fewer the items, the more judges you need, because there are fewer estimates per judge to average out.

The same study found that panels should include judges with mixed levels of expertise rather than only the most senior practitioners, unless there is clear evidence that the experts on the panel are not more stringent than the rest. Stacking a panel with the most expert people available is the intuitive move and it biases the standard upward, because an expert's sense of what is minimally competent drifts with their own ability.

It is worth reading that source with its own limitation attached, which the authors state themselves: simulations "do not describe natural phenomena as they do not use observed data", and the paper names the absence of data on examinees' true ability as a standing problem in standard setting. Fifteen is a defensible working figure from a simulation, not a measured constant.

The standard error this entry asks for, computed on its own example

The illustration below uses three judges, which is a teaching device rather than a panel, and running the error on it shows why. Each judge's own total across the four items is 2.65, 2.45 and 2.65 of 4. The mean is 2.583, which is the 64.6 per cent quoted below. The standard deviation across the three judges is 0.115, so the standard error of the mean cut score is 0.115 ÷ √3 = 0.067 of 4 points.

Turn that into a confidence interval and the three-judge panel recommends a cut score somewhere between 61.3 and 67.8 per cent — a band 6.5 percentage points wide. The note below warns against rounding 65 per cent up to 70 per cent for presentational comfort, and it is right to; the arithmetic adds that this panel cannot distinguish 65 per cent from 62 per cent either.

Now add judges and hold the same disagreement. Ten judges narrow the interval to 62.8–66.4 per cent, fifteen to 63.1–66.0, twenty to 63.3–65.8. The width falls with the square root of the panel size, so the return diminishes: the first ten judges halve the band and the next ten take about a fifth off what is left. That is the concrete reason fifteen is the recommendation and three is an illustration.

Two things this arithmetic does not cover, stated because a cut score is the most consequential number on any certification. It is the error between judges on one panel, not the error between panels — the replication literature asks whether a second panel of equally qualified experts would land in the same place, which is a larger and separately measured question. And it says nothing about whether the standard is right, only about how precisely this panel located it. A precise standard set against a badly written borderline definition is precisely wrong. See standard error of measurement for the candidate-side error that sits on top of this one, and standard setting for the wider family of methods.

A four-item illustration

Not the same as norm-referenced pass mark

An Angoff cut score is criterion-referenced: it is fixed to the content and does not move when the cohort changes. A norm-referenced pass mark takes a percentage of the cohort and therefore changes every time the applicant pool does.

Sources

• Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed.)
• AERA, APA & NCME, Standards for Educational and Psychological Testing (2014), Chapter 5 — Scores, Scales, Linking and Cut Scores
• Shulruf, B., Wilkinson, T., Weller, J., Jones, P. & Poole, P. (2016). Insights into the Angoff method: results from a simulation study. BMC Medical Education, 16, 134 — open access

Read next

• Angoff cut score calculator — run the arithmetic on your own panel

Related terms

More terms beginning with A