What is Differential item functioning?
Also called DIF, Item bias analysis
Differential item functioning occurs when two groups of equal ability on the attribute being measured have different chances of answering a particular question correctly. It is a property of the item rather than of the groups, and it is the statistical evidence used to identify questions that need review for bias.
Why 'of equal ability' is the whole idea
Two groups scoring differently overall is not DIF, and is not by itself evidence of a problem with the test. DIF asks a narrower question: among people who scored the same on the test as a whole, does this particular item still behave differently between groups?
That conditioning is what makes DIF useful. It isolates something specific to the item — a culturally loaded example, a idiom, a diagram that assumes a familiar context — from real differences in the attribute being measured.
Statistics find candidates; people make the decision
The most widely used index is the Mantel-Haenszel statistic expressed on the delta scale. Educational Testing Service's classification, used in NAEP among many others, sorts items into three categories: A, negligible DIF, where the value is not significantly different from zero or is below 1.0 in absolute magnitude; C, large DIF, where it is both significantly greater than 1 and above 1.5 in absolute magnitude; and B for everything in between.
A category C item is not automatically deleted. It goes to a review panel of test developers and subject-matter specialists, whose job is to decide whether the statistic reflects item bias or a genuine, defensible difference in the content. A pipeline that deletes items on the statistic alone will quietly remove legitimate content, and one that never reviews them will keep bad items indefinitely.
Not the same as adverse impact
DIF is measured on a single item, conditioned on ability, at the level of the instrument. Adverse impact is measured on outcomes at the level of the hiring process. A test can be free of DIF and still produce adverse impact, and vice versa.
Adverse impact ratioSources
Related terms
Check a selection process against the four-fifths rule
Free, no signup, computed in your browser — with the remedy, not just the verdict.
Last reviewed