What is Differential Item Functioning?
Lead consultant at Test Partnership, Ben Schwencke, explains differential item functioning.
Differential Item Functioning (DIF) is a psychometric phenomenon, whereby an Item’s Characteristic Curve (ICC) differs for members of specific subgroups. Often, this suggests that this specific item measures different psychological constructs depending on the subgroup, disadvantaging members of a specific subgroup. Consequently, members from the disadvantaged subgroup are likely to find the item with DIF to be more difficult, even after controlling for their actual level of ability.
Here is a hypothetical example of DIF. A cognitive ability test has been designed with a wide range of different cognitive tasks. One of these questions however, has additional instructions for completion which are presented only in English, whereas all other questions are language free. Consequently, English speakers find this question comparatively easier than non-English language speakers regardless of their overall ability, resulting in two ICCs (akin to the image below). Here you can see that the focal group has a lower chance of correctly answering the question than individuals from the reference group at the same level of ability.
In this instance, bias has been introduced into the assessment, as in addition to cognitive ability, this item also unintentionally measures English language competence, disadvantaging non-English speakers unfairly. To ensure that the assessment remains focused on its intended psychological construct, and to minimise any unfair bias against particular subgroups, best practice would involve amending or removing the offending item from the assessment.