Adverse Impact vs Differential Item Functioning: One Tells You the Funnel Is Unfair, the Other Tells You Which Item Is

Adverse impact and differential item functioning are two different fairness checks, and they get used interchangeably. Adverse impact compares selection rates between intact groups — a decision-level statistic. Differential item functioning (DIF) compares performance on a single question between candidates already matched on measured ability. A gap in pass rates is impact. It is not evidence that any particular question is flawed.

That distinction governs what you are entitled to conclude from a fairness audit, and most audits conclude more than their statistics support.

The two numbers answer different questions

The adverse impact ratio is computed from outcomes. The Uniform Guidelines on Employee Selection Procedures set the familiar benchmark at 29 CFR 1607.4(D): "A selection rate for any race, sex, or ethnic group which is less than four-fifths (4/5) (or eighty percent) of the rate for the group with the highest rate will generally be regarded by the Federal enforcement agencies as evidence of adverse impact." It is a ratio of pass rates. It says nothing about why they differ.

DIF is an item-level statistic, and the matching is the entire point. In the ETS research report that set out the standard methods, Dorans and Holland draw the line precisely: "Impact refers to a difference in performance between two intact groups," whereas DIF "refers to a difference in item performance between two comparable groups of examinees" — groups "matched with respect to the construct being measured by the test."

Two groups can differ in measured ability for reasons that have nothing to do with your test: unequal schooling, unequal access to practice, unequal exposure to the language of the test. Where they do, item performance will differ too, and that difference is expected rather than suspicious. An NCME instructional module by Clauser and Mazor puts the consequence bluntly: "differences in performance, in and of themselves, are not evidence of bias."

A pass-rate gap can point the wrong way

Dorans and Holland include a worked example that belongs in every audit template. An item is answered correctly by 60% of one group and 50% of another. The impact statistic favours the first group. But when examinees are split by ability level, the second group performs better at every level. The authors' conclusion: "this item actually favors group B over group A, not vice versa as suggested by impact."

The aggregate reversed the sign. This is Simpson's paradox operating inside a hiring test, and it is why an item-level accusation built on raw pass rates is unsafe in both directions. You will clear items that genuinely function differently, and condemn items that are merely hard for a group that scored lower overall.

What a DIF flag actually is

DIF is not a yes/no verdict, and the operational rules are more conservative than most internal audits. ETS sorts each item into one of three categories using the Mantel-Haenszel delta statistic. As documented in Rebecca Zwick's 2012 review of those procedures:

• Category C — moderate to large DIF. The absolute value of MH D-DIF must be 1.5 or greater, and significantly greater than 1 at the 5% level.
• Category B — slight to moderate DIF. The Mantel-Haenszel chi-square exceeds 3.84 and the absolute MH D-DIF is at least 1.
• Category A — negligible or non-significant. Everything else.

Two consequences matter operationally.

A flag triggers review, not deletion

Michael Zieky's 2003 DIF Primer for ETS is explicit that "no statistic can determine whether or not a test question is biased" and that "DIF is NOT synonymous with bias." Flagged items go to human reviewers — including, in the ETS process, "external reviewers who are not affiliated with ETS" — who judge whether the difference is unfairly related to group membership. A 1988 ETS memo quoted by Zwick sets the bar for retaining one: Category C items "will NOT be used unless they are judged to be essential to meet test specifications." Clauser and Mazor frame the same point theoretically: DIF is "a necessary but not a sufficient condition for item bias."

You may not have the sample size

Zwick documents the ETS requirement: "at least 200 members in the smaller group and at least 500 in total are needed for DIF analyses performed at the test assembly phase," rising to 300 in the smaller group and 700 in total at the preliminary item-analysis stage. A campus drive of 400 candidates with 60 in the smaller group will not support a DIF analysis on any item, however carefully the statistic is computed.

What each check can and cannot do

The adverse impact ratio

• Works at any funnel stage, with modest numbers.
• Is the statistic regulators and plaintiffs actually use.
• Tells you nothing about which item, instruction or time limit produced the gap.
• Can be driven entirely by real ability differences in the applicant pool.

A DIF analysis

• Isolates individual items while holding measured ability constant.
• Catches problems invisible in the aggregate, including sign reversals.
• Needs several hundred candidates per group and a reasonably unidimensional test.
• Produces a flag that still requires expert judgement, not a conclusion.

When the pass-rate ratio is the right check

Three situations where impact, not DIF, is the number you need.

Your exposure is defined at the decision level. In Connecticut v. Teal, 54.17% of the identified Black candidates passed a written examination — approximately 68% of the passing rate for the identified white candidates. The employer's final promotion rate did not disadvantage Black candidates, and the Supreme Court held that this made no difference: "that 'bottom line' is no defense to this prima facie case." The principal focus of Title VII, the Court wrote, "is the protection of the individual employee, rather than the protection of the minority group as a whole." Impact at the component you administer is the exposure. A clean DIF run does not retire it.

The problem is not in the items. Differential speededness, mobile rendering, proctoring friction and instruction comprehension can all produce impact without any single item functioning differently. Dorans and Holland note that the matching variable itself "may be contaminated due to differential speededness" — in which case the DIF analysis is quietly measuring the wrong thing.

The bias is pervasive rather than item-specific. Where every item on a test tilts the same way, matching on total score conceals it, because the yardstick is tilted too. Clauser and Mazor state it directly: "Conditioning on total test score does not remove pervasive bias."

How to run both on a real funnel

1. Compute impact first, at every stage separately. Distinct ratios for the sift, each assessment and the interview. Teal is the reason you do not aggregate them into one bottom line.
2. Check whether a DIF analysis is even possible. Count candidates in the smaller group. Below roughly 200, you are doing exploratory work and the report should say so.
3. Pool across cycles before analysing. Item responses accumulate across drives in a way that pass-rate snapshots do not.
4. Match on a defensible score. Total score on the same construct, with flagged items purified out and the analysis re-run.
5. Send flags to people, not to a delete script. Reviewers who did not write the item, judging relevance to the job.
6. Keep the sensitivity review separate. Clauser and Mazor are clear that sensitivity reviews "are separate and distinct from DIF analyses" and that "neither can substitute for the other." One is editorial judgement before publication; the other is statistical evidence after administration.
7. Do not report either as validity evidence. "DIF analysis is not a substitute for validity studies," Dorans and Holland write. "In fact, the DIF analysis assumes that the criterion is valid and fair."

On AssessAll, item-level response data from AI-graded scenarios accumulates across drives, which is what makes a pooled DIF analysis feasible for employers who would never reach the sample threshold in a single cycle. Where flagged items need replacing, custom assessment services handle the rewrite and the re-review.

The takeaway

An adverse impact ratio tells you that your process produced an unequal outcome. A DIF analysis tells you which question behaved differently for equally able candidates. Reporting one as though it were the other is the most common error in assessment fairness work, and the only fix is to compute both — each at the level it actually describes.