--- title: "Differential item functioning" description: "Differential item functioning occurs when two groups of equal ability on the attribute being measured have different chances of answering a particular question correctly. It is a property of the item rather than of the groups, and it is the statistical evidence used to identify questions that need review for bias." canonical: https://www.assessall.com/guides/glossary/d/differential-item-functioning updated: 2026-09-24 source: AssessAll --- # Differential item functioning _Also known as: DIF, Item bias analysis._ Differential item functioning occurs when two groups of equal ability on the attribute being measured have different chances of answering a particular question correctly. It is a property of the item rather than of the groups, and it is the statistical evidence used to identify questions that need review for bias. ## Why 'of equal ability' is the whole idea Two groups scoring differently overall is not DIF, and is not by itself evidence of a problem with the test. DIF asks a narrower question: among people who scored the same on the test as a whole, does this particular item still behave differently between groups? That conditioning is what makes DIF useful. It isolates something specific to the item — a culturally loaded example, a idiom, a diagram that assumes a familiar context — from real differences in the attribute being measured. ## Statistics find candidates; people make the decision The most widely used index is the Mantel-Haenszel statistic expressed on the delta scale. Educational Testing Service's classification, used in NAEP among many others, sorts items into three categories: A, negligible DIF, where the value is not significantly different from zero or is below 1.0 in absolute magnitude; C, large DIF, where it is both significantly greater than 1 and above 1.5 in absolute magnitude; and B for everything in between. A category C item is not automatically deleted. It goes to a review panel of test developers and subject-matter specialists, whose job is to decide whether the statistic reflects item bias or a genuine, defensible difference in the content. A pipeline that deletes items on the statistic alone will quietly remove legitimate content, and one that never reviews them will keep bad items indefinitely. ETS's own primer on the method, cited below, puts the limit more bluntly than most vendor pages will: "No statistic can determine whether or not a test question is biased. DIF helps us spot questions that may be unfair, but DIF is NOT synonymous with bias." That is the publisher of the most widely used DIF index saying its index does not answer the question it is usually quoted as answering. ## The thing DIF cannot see, and why it matters on a language-light test DIF works by matching candidates on the total test score and then asking whether one item still behaves differently. That matching criterion is the method's power and its blind spot at the same time, and the ETS primer acknowledges the circularity directly: the score used to establish that two candidates are of equal ability is produced by the very instrument under examination. The consequence is specific and worth stating plainly. If every item on a form is affected in the same direction and to a similar degree, the total score moves with them, the matching absorbs the effect, and no item stands out against its neighbours. A uniformly affected test can return a clean DIF run — nothing in category C — while still producing a difference in scores that has nothing to do with the attribute being measured. That is exactly the case a buyer should have in mind for formats sold on the strength of carrying no language, such as abstract or figural pattern reasoning. A clean DIF report on such a bank is evidence that no single item is an outlier among its siblings. It is not evidence that the format is fair, and the two claims are routinely conflated in this market. The honest reading of a DIF run is therefore narrow and still worth having: it finds the items that misbehave relative to the rest of the form, it hands them to people, and it says nothing about the form as a whole. Evidence about the whole form has to come from somewhere else — the norm group, differential prediction against an outcome, and the subgroup-difference literature for that measure. --- Source: https://www.assessall.com/guides/glossary/d/differential-item-functioning