Marking Consistency and Assessment Design Assessment for Teachers, Trainers and AssessorsTwo markers, one script, two grades. One of them is yours.

Methodology: Thirty-six original exercises across five formats: twelve marking exercises in which a piece of student work is marked out of twenty on a printed three-part rubric, ten keyed single-choice exercises, five select-every-that-applies exercises, five binary claims and four matching exercises. Two response instructions are declared. The marking strand is a performance task scored against a moderated panel mark. The other twenty-four exercises carry a knowledge instruction - what is most likely to be true, or most effective. The two are reported apart and never averaged, because agreeing with a panel and knowing why markers disagree are different things and a marker can be good at one and poor at the other. Construct statement: it measures how closely somebody's marking matches a moderated standard once their own severity and spread have been removed, how consistent they are with themselves on a script they mark twice, and what they know about writing questions that measure, about the rater effects that distort a mark, and about feedback a learner acts on. It does not measure subject knowledge in any discipline, teaching quality, classroom practice, or fitness to hold any teaching or assessing role, and it awards no qualification. Scoring is chance-corrected proximity, built in the order the method requires and not in the order that is convenient. First the twelve marks are ipsatised within the person - each mark expressed as a distance from that person's own mean in units of their own spread - and the panel key is ipsatised identically. Then the distance between the two ipsatised vectors is taken. Then that distance is corrected against an EMPIRICAL null: the mean distance produced by a respondent whose mark on each script is drawn from that script's own declared marginal distribution of marks, computed by simulation, and not by a uniform draw across the scale. A uniform null is wrong here and wrong in the direction that flatters: real markers cluster in the middle of a scale, so a respondent answering thirteen to everything lands close to the key by accident, and a scorer that does not correct for it produces the flat baseline where everybody is moderate. Ipsatisation is what makes the agreement figure independent of severity and of spread, and those two are then reported as their own numbers rather than being lost - a marker three marks below the panel throughout has a severity of minus three and an agreement figure that is unaffected by it, which is the whole point. Self-consistency is computed from the two seeded repeat scripts and is reported as a distance in marks, never as a pass or a fail. Every strand carries its standard error and reliability is reported as McDonald's omega estimated from item count and a stated assumed inter-item correlation; a strand whose omega will not carry a number gets a three-way placement and the refusal is printed. No percentile appears anywhere. Constructs and sources: rater severity, central tendency and the many-facet Rasch treatment of judge effects (Linacre 1989; Eckes 2011); halo and fluency effects on marking (Thorndike 1920; Klein and Hart 1968); contrast effects between consecutively marked scripts (Daly and Dickson-Markman 1982; Attali 2011); inter-rater reliability and the limits of percentage agreement (Shrout and Fleiss 1979); the effect of exemplars and standardisation on marker agreement (Bloxham, den-Outer, Hudson and Price 2016); reliability of essay marking in public examinations (Murphy 1979); item difficulty, discrimination and the uninformative ceiling item (Crocker and Algina 1986); assessment literacy as a distinct professional competence (Popham 2009; Stiggins 1991); feedback at task, process and self-regulation level and the harm of self-level feedback (Hattie and Timperley 2007); the grade crowding out the comment (Butler 1988); and ipsative within-person standardisation as a control for elevation and scatter in profile matching (Cronbach and Gleser 1953; Fisher 2004). All items and all student scripts are original works written for this instrument. No examination board's mark schemes, rubrics, examiner materials, scale names or report layouts are reproduced or implied, and no accreditation or affiliation is claimed.