---
title: "Five Models Cleared the Agreement Standard, Then Scored Gibberish: Seven Checks Before an AI Grader Goes Live"
description: "Five essay-scoring models met the standard acceptance criteria - quadratic weighted kappa above .70 - then scored gibberish as zero in between 0% and 18.6% of cases. Seven checks that turn human-machine agreement into evidence an AI grader actually measures something."
canonical: https://www.assessall.com/blog/five-models-cleared-the-agreement-standard-then-scored-gibberish-seven-checks
published: 2026-10-24
updated: 2026-10-24
author: "AssessAll Editorial"
source: AssessAll
---

# Five Models Cleared the Agreement Standard, Then Scored Gibberish: Seven Checks Before an AI Grader Goes Live

Five essay-scoring models met the standard acceptance criteria - quadratic weighted kappa above .70 - then scored gibberish as zero in between 0% and 18.6% of cases. Seven checks that turn human-machine agreement into evidence an AI grader actually measures something.

Validating an AI grader means showing its scores relate to something outside the test. Agreement with human raters is a reliability check, not validity evidence. The published acceptance criteria are a quadratic weighted kappa of at least .70 against human scores, no more than .10 below human–human agreement, and a standardised mean difference within ±.15.

Those three numbers are the entry requirement. Most of the work happens after they are met.

## Where the agreement thresholds came from

The operative framework is Williamson, Xi and Breyer's [*A Framework for Evaluation and Use of Automated Scoring*](https://onlinelibrary.wiley.com/doi/10.1111/j.1745-3992.2011.00223.x) (*Educational Measurement: Issues and Practice*, 2012). ACT's 2021 technical brief [*Establishing Standards of Best Practice in Automated Scoring*](https://act.org/content/dam/act/unsecured/documents/R2100-auto-scoring-standards-2021-07.pdf) collects the criteria large testing programmes actually apply:

**Agreement with human scores**

- Quadratic weighted kappa between human and automated scores of at least .70
- Human–human QWK minus human–machine QWK no greater than .10
- Human–human minus human–machine exact agreement difference no greater than 5.125%

**Score distribution**

- Standardised mean difference between human and machine scores within ±.15
- Ratio of human to machine score standard deviations between 0.67 and 1.50

**Subgroup behaviour**

- Williamson et al. flag subgroup differences between machine and human scores at .10
- CTB's published practice flags subgroup SMD above .10 and population SMD above .12

Note what is absent: none of these criteria asks whether the score predicts anything. ACT's brief is explicit that a programme must "investigate more than just the consistency with human scores."

## Five models cleared the bar, then scored gibberish

Afrizal Doewes and Mykola Pechenizkiy tested that gap in [*On the Limitations of Human-Computer Agreement in Automated Essay Scoring*](https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_243.pdf) (EDM 2021). On the ASAP essay dataset, five of their nine models met the standard acceptance criteria, with QWK between .7736 and .8065.

They then fed the models responses any human rater would have zeroed:

- **Off-topic essays** — drawn from the other prompts in the dataset. The models scored them zero in between 0% and 5.8% of cases.
- **Gibberish** — the models scored it zero in between 0% and 18.6% of cases.
- **Paraphrased answers** — agreement between original and paraphrased versions held up, QWK .8109 to .9411. This is the one test the models passed.

Adding 350 off-topic essays to the training data raised off-topic accuracy to between 45.7% and 97%, and 100 to 200 gibberish examples raised gibberish accuracy to roughly 68% to 98.6%, with only small changes in QWK. The defect was fixable — it was simply invisible to the agreement statistics, because applicants in a calibration sample do not submit gibberish.

## Seven checks before an AI grader goes live

### 1. Measure the human baseline before you measure the machine

In ETS's evaluation of e-rater for the Praxis I Writing Test ([Ramineni, Trapani and Williamson, 2015](https://files.eric.ed.gov/fulltext/EJ1109680.pdf), over 68,000 operational responses across 35 items), human–human agreement was QWK .74 with 68% exact agreement. Human–e-rater agreement was QWK .78. The machine figure is only interpretable against the human one — and here the engine agreed with a human rater slightly more than two humans agreed with each other, which says as much about rater variance as about the engine.

### 2. Run every statistic at subgroup level

The same ETS study removed two of 35 prompts for exceeding the .15 SMD limit, and found one subgroup — Asian Pacific Islander test takers, n = 1,549 — above the .10 threshold at SMD .11, the engine scoring higher than humans. A later engine version eliminated it. The study's overall SMD of −.01 conceals that entirely. Note also SIOP's caution that subgroup mean differences "should not be relied upon to indicate psychometric bias" — they are a flag for investigation, not a verdict.

### 3. Feed it responses it should refuse to score

Build an adversarial set: off-topic answers, keyword-stuffed answers, empty-but-fluent answers, wrong-language responses, and text copied from the prompt. Score them by hand, then compare. This is where AI-graded scenario responses need an explicit refusal path — a "cannot be scored" output routed to human review rather than a number. On AssessAll that pairs with integrity banding, so an anomalous attempt surfaces as a flag rather than as a quiet mid-range score.

### 4. Do not accept a single agreement statistic

Doewes, Kurdhi and Saxena's [*Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring*](https://files.eric.ed.gov/fulltext/ED630859.pdf) (EDM 2023) shows why. QWK is sensitive to the rating scale, so the same pattern of agreement yields different kappas depending on how scores are combined. It suffers the kappa paradox: their worked example reaches 99.8% percent agreement with a QWK below .70. And moving the same number of agreements to different cells of the matrix shifted QWK from .665 to .853. Report exact agreement, the SD ratio and the SMD alongside kappa, and use Krippendorff's alpha or Fleiss' kappa for more than two raters.

### 5. Correlate the scores with something outside the test

This is the check that converts reliability into validity. ETS reported correlations with the Praxis I writing scaled score of .73 for human scores and .69 for e-rater. For employment use, [SIOP's January 2023 *Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection*](https://www.siop.org/wp-content/uploads/2024/06/Considerations-and-Recommendations-for-the-Validation-and-Use-of-AI-Based-Assessments-for-Employee-Selection-January-2023.pdf) is firmer: scores "should demonstrate relationships with future job performance or other job-relevant outcomes," the outcome should follow from a job analysis, and reproducing an existing score "does not establish that the AI assessment scores are job-relevant."

### 6. Decide whether the model scores alone or contributes

ETS did not recommend e-rater as a sole scorer for Praxis. It recommended a contributory model in which the engine replaces the second human rating only when it falls within 1.5 points of the first human score. Under that design, passing status was unchanged for about 96% of test takers, and only about 4% of cases needed a second human rating. The lesson is the numeric trigger: "human in the loop" should name a score gap or confidence threshold that routes a case to a person, not a general assurance that someone reviews things.

### 7. Pin the version and plan for drift

SIOP's statement asks organisations to document how often and under what conditions the assessment will change, and to "specify which versions of the algorithm produce comparable scores" — because frequent model revisions reduce the value of past validation work. ACT's brief cites CTB's practice of monitoring for a drop of more than 0.05 in perfect agreement between human–human and engine–human scoring as a drift trigger. Pick a monitoring statistic and threshold before go-live, not after a complaint.

## Where LLM-based graders currently sit

If the engine is a general-purpose language model rather than a purpose-trained scoring engine, the evidence base is thinner than it sounds. Hongli Li and colleagues synthesised 65 published and unpublished studies from January 2022 to August 2025 in [*Agreement Between Large Language Models and Human Raters in Essay Scoring*](https://arxiv.org/abs/2512.14561) (2026). Three findings matter operationally:

1. Reported QWK ranged from −.10 to .97, typically .30 to .80. Correlations ranged from .17 to .91.
2. The metrics varied so much across studies that no pooled estimate could be computed. There is no single number for "how well LLMs grade."
3. Agreement was highly context-dependent — stronger with more capable models, and improved materially by prompt design, including supplying the rubric and worked examples.

Point 2 is the practical one. A QWK from a public essay dataset is not an estimate of performance on your competency rubric, your candidate population or your item wording. It has to be re-established on your own items — work a custom assessment build should include rather than assume.

## The checklist

1. Record human–human agreement on a double-scored sample first.
2. Confirm QWK ≥ .70 and degradation ≤ .10 against that baseline.
3. Confirm SMD within ±.15 and SD ratio between 0.67 and 1.50.
4. Repeat 2 and 3 for every subgroup with sufficient n, flagging at .10.
5. Score an adversarial set by hand and compare.
6. Correlate scores with an external, job-analysis-derived outcome.
7. Set the numeric threshold that routes a response to a human.
8. Fix the model version, pick a drift statistic, and schedule the re-check.

## Takeaway

Agreement statistics tell you whether a machine reproduces your raters; they cannot tell you whether your raters measured the right thing, and they are silent on responses your calibration sample never contained. Establish the human baseline, test the edges deliberately, and anchor at least one claim to an outcome outside the assessment.

---

Source: https://www.assessall.com/blog/five-models-cleared-the-agreement-standard-then-scored-gibberish-seven-checks