---
title: "0.74 to 0.98 Across One Programme's Forms: How to Check Whether Your Pass/Fail Decision Would Replicate"
description: "Score reliability does not tell you whether your pass/fail decision would survive a second sitting. Decision consistency does, professional standards require it when you classify candidates, and it can be estimated from a single administration."
canonical: https://www.assessall.com/blog/0-74-to-0-98-across-one-programmes-forms-how-to-check-whether-your-pass-fail
published: 2026-11-15
updated: 2026-11-15
author: "AssessAll Editorial"
source: AssessAll
---

# 0.74 to 0.98 Across One Programme's Forms: How to Check Whether Your Pass/Fail Decision Would Replicate

Score reliability does not tell you whether your pass/fail decision would survive a second sitting. Decision consistency does, professional standards require it when you classify candidates, and it can be estimated from a single administration.

Decision consistency is the probability that a candidate would receive the same pass/fail outcome on a second, equivalent form of your assessment. It is reported on a 0 to 1 scale, estimated from a single sitting, and it is not the same number as score reliability. Where you classify candidates, professional testing standards require it.

Most assessment reports stop at reliability. A coefficient alpha of 0.86 goes in the technical note, and everyone treats the question of precision as settled. But alpha describes how consistently the instrument orders people along a score scale. It does not describe how consistently your pass/fail line sorts them into two groups — and the second question is the one a rejected candidate, a works council, or a court will ask.

## Reliability describes the score. Decision consistency describes the decision.

The [*Standards for Educational and Psychological Testing*](https://www.testingstandards.net/) treat these as two separate reporting obligations. Standard 2.14 requires that "where cut scores are specified for selection or classification, the standard errors of measurement should be reported in the vicinity of each cut score." Standard 2.16 goes further and sits in a cluster of its own, titled Decision Consistency: "When a test or combination of measures is used to make classification decisions, estimates should be provided of the percentage of test takers who would be classified in the same way on two replications of the procedure."

That is a different statistic from alpha, and it is not optional language. The accreditation standards of the National Commission for Certifying Agencies say the same thing for certification bodies: under [Standard 20 on reliability](https://www.nasfaa.org/uploads/documents/2016_NCCA_Standards.pdf), programmes "must calculate and report estimates of score reliability, decision consistency, and standard errors of measurement," and those estimates "must be reasonable to support accurate pass/fail decisions."

If your assessment produces a hire/no-hire, pass/fail, or certify/do-not-certify output, you are making classification decisions, whether or not you call them that.

## What the number looks like when someone actually publishes it

Ohio's teacher-licensure testing programme reports a decision-consistency figure for every examination form with at least 100 test takers. In its [2020–21 technical report](https://www.oh.nesinc.com/Content/Docs/OAE_TestAdministration_TechnicalReport_2020-21.pdf), the published values run from **0.74** on one Early Childhood Special Education form to **0.98** on a Reading subtest form, estimated with the Livingston-Lewis method. Same programme, same standard-setting process, same quality controls — and a 24-point spread in how reliably the pass/fail decision replicates.

For a benchmark, Subkoviak's long-standing guidance, quoted in a [2024 review in *Practical Assessment, Research & Evaluation*](https://doi.org/10.7275/pare.2023), is blunt: "Tests used to make serious decisions should be sufficiently long to guarantee an agreement coefficient exceeding 0.85."

Measured against that, a good share of real operational forms fall short. The same review reports a hypothetical 10-item exam with alpha of 0.60 and a cut score of 5 returning a decision consistency of **0.75** — meaning roughly one candidate in four would land on the other side of the line on an equivalent form.

## Consistency is not accuracy, and the two get swapped

Two different questions hide behind "how good is the decision":

**Classification consistency**
- Would the same candidate get the same outcome on a different, parallel form?
- Compares one fallible measurement against another fallible measurement.
- This is what Standard 2.16 asks for.

**Classification accuracy**
- Does the observed outcome match the outcome implied by the candidate's true score?
- Compares one fallible measurement against a hypothetical error-free one.

Both are estimated from a single administration using a model, as the UCLA CRESST technical report on [estimating consistency and accuracy of classifications](https://cresst.org/wp-content/uploads/TECH475.pdf) sets out. They do not produce the same number. Ofqual's [study of AQA's GCSE and A-level grading](https://assets.publishing.service.gov.uk/media/5a755dbce5274a59fa717981/2011-03-16-aqa-classification-accuracy-and-consistency-in-gcse-and-a-levels.pdf) found accuracy running from 0.55 to 0.73 across AS units while consistency on the same units ran from 0.47 to 0.64 — and stated the relationship plainly: "Classification consistency is lower than classification accuracy."

That matters commercially, because accuracy is the flattering number. If a vendor quotes you one figure for decision quality, ask which of the two it is.

## How to compute it on your own assessment

1. **Name the decision, not the score.** One cut score, two categories. If your report issues four capability bands, you have three cut scores and need an estimate for each.
2. **Assemble four inputs.** Total scores for every candidate on a single form, a reliability estimate for that form, the minimum and maximum possible scores, and the cut score as actually applied.
3. **Choose a method and say which one you used.** The 2024 review compares the Livingston-Lewis approach, a Peng-Subkoviak normal-approximation method, and a simpler observed-data method built on confidence intervals around the cut.
4. **Expect the method to move the number.** On one licensure dataset with a cut at 63, the three methods returned 0.78, 0.84 and 0.80. Method choice is worth roughly six points here, so it belongs in the report, not in a footnote.
5. **Watch the short-test caveat.** Livingston-Lewis "has larger biases for exams with a small number of items" — relevant if your screening test is 15 or 20 items.
6. **Report it next to the SEM at the cut.** Standard 2.14 and Standard 2.16 are complementary, not alternatives.
7. **Re-run it per form and per subgroup.** The Ohio spread of 0.74 to 0.98 is across forms of the same test.

## Where the cut sits changes the answer

Decision consistency is lowest when the cut score falls at the peak of the score distribution, because that is where the most candidates sit within a hair of the line. Put the cut in a sparse region of the distribution and the index rises.

This is a genuine trap. Moving a cut score to flatter ground improves the statistic without improving the assessment, and if the original line came from a defensible standard-setting exercise, moving it for statistical convenience destroys the thing that made it defensible. The 2024 review makes the same point from the other direction: a low value at the distribution's peak may be entirely justifiable when the items were built to discriminate precisely among minimally qualified candidates. Report the number and explain it. Do not chase it.

## If the figure comes back low

- **Lengthen the test, or add items that discriminate near the cut.** Precision where the decision happens is what moves this index.
- **Reduce the number of categories.** A dichotomous pass/fail replicates far better than a five-band grade, which is visible in the Ofqual figures above.
- **Add a confirmatory second sitting for candidates in the band around the cut**, rather than re-testing everyone. On pay-as-you-go pricing like AssessAll's ₹30 / US$0.50 per credit, a targeted re-sit for the candidates sitting near the line is a per-candidate cost, not a procurement decision.
- **Report a band instead of a point** to the hiring manager, so a 68 and a 71 are not discussed as if they were different findings.
- **Check the grader, separately.** If scenarios are AI-graded — as AssessAll's are — a published human-agreement figure for the grader is not a decision-consistency estimate for the resulting pass/fail outcome. They are two different validations, and you need both.

What you must not do is round the borderline cases upward and call the problem solved. That converts a measurement question into an undocumented policy, which is exactly what Standard 2.16 exists to prevent.

## Before you publish a pass/fail result

- The decision, the cut score, and the number of categories are written down.
- A decision-consistency estimate exists for each cut score, with the method named.
- The SEM in the vicinity of the cut is reported alongside it.
- Estimates are reported per form, not pooled across forms.
- Consistency and accuracy are labelled distinctly, and you know which one you are quoting.
- Borderline-candidate handling is a documented rule decided before results, not a case-by-case judgement after.

Reliability tells you the instrument is behaving. Decision consistency tells you whether the decision you took on Tuesday would have been the same decision on Thursday — and that is the only one of the two a rejected candidate was ever affected by.

---

Source: https://www.assessall.com/blog/0-74-to-0-98-across-one-programmes-forms-how-to-check-whether-your-pass-fail