Data Cleaning Skills Assessment with Executed ScriptsSay how many rows your rule will change. Then run it and find out.

Methodology: Forty-one original exercises across five formats: five executed cleaning programs run against real files, seven interval-estimate items, seven point estimates on a slider, five further single-choice items, nine select-every-that-applies items and eight true or false claims. Construct statement: it measures whether somebody can read a cleaning contract precisely, write a program that satisfies it on files it has not seen, say in advance how many rows a rule will touch and how sure they are of that number, avoid destroying information while cleaning, and record what was changed so somebody else can check it. It does not measure statistics, machine learning, database work, library knowledge, or speed. Declared response instruction: knowledge and demonstrated skill throughout. Execution: the five programs are run twice each - once on the file printed in the exercise and once on a second file the respondent has not seen, which carries the blank that is not empty, the negative in brackets, the date in the third format and the duplicate that is not quite a duplicate. Where the execution service cannot be reached, those five exercises are reported as not run and leave the denominator rather than scoring zero, and the report says so on the page. Scoring: the fourteen estimates are scored as calibration rather than as accuracy. An interval earns credit for containing the true value and loses it for width, so the widest interval - which cannot be wrong - scores near nothing, and a point estimate on a number that was stated in the exercise scores full. The report separates the two components a calibration score is made of: whether the intervals contain the truth as often as their width implies, and whether they change width when the information changes. A respondent can be well calibrated and useless, by giving every answer the same wide interval, and a respondent can be discriminating and over-confident. Those are different findings with different advice and a single percentage destroys the distinction. Construct grounding, drawn across sources rather than from one framework: the Brier score and its decomposition into reliability, resolution and uncertainty (Brier, 1950; Murphy, 1973); proper scoring rules for interval forecasts, which reward a narrow interval and penalise a miss (Winkler, 1972; Gneiting and Raftery, 2007); the overconfidence and interval-narrowness literature, in which stated ninety per cent ranges contain the truth around half the time (Alpert and Raiffa, 1982; Lichtenstein, Fischhoff and Phillips, 1982); the finding that calibration improves with feedback and with being asked for reasons the estimate could be wrong (Koriat, Lichtenstein and Fischhoff, 1980); the planning fallacy as the same effect on effort rather than quantity (Kahneman and Tversky, 1979; Buehler, Griffin and Ross, 1994); tidy-data structure and one row per observation (Wickham, 2014); the taxonomy of data-entry faults that survive validation (Broman and Woo, 2018); the distinction between a value that is missing and a value that is zero, and what imputing the second for the first destroys (Little and Rubin, Statistical Analysis with Missing Data); measured human error rates in data handling and the weakness of self-review (Panko, 1998; 2008); provenance and reproducibility as properties of a recorded transformation rather than of a script (Sandve and colleagues, Ten Simple Rules for Reproducible Computational Research, 2013); McDonald's omega in place of alpha (McDonald, 1999); and the standard error of measurement as the reason a band rather than a point is reported (AERA, APA and NCME Standards, 2014). All items are original works. No item, dataset, scale name or report section is taken from any commercial instrument, and no affiliation with or endorsement by any vendor is claimed or implied.