Introduction to aiDIF

1. Overview

Classical differential item functioning (DIF) asks whether an item behaves differently across groups within a single scoring condition. When responses are scored by an AI system rather than by human raters, a second question appears: does moving from human to AI scoring change the group contrast?

aiDIF separates three quantities, using the robust scaling framework of Halpin (2024):

\(\mathrm{DIF}^H_i\), the group contrast for item \(i\) under human scoring;
\(\mathrm{DIF}^A_i\), the same contrast under AI scoring;
\(\mathrm{DASB}_i = \mathrm{DIF}^A_i - \mathrm{DIF}^H_i\), the Differential AI Scoring Bias, which is the change in that contrast.

An item can carry substantial DIF under both conditions and have a DASB of zero: the AI reproduced whatever the human raters were doing. Conversely, an item with no DIF under either condition in isolation can still show a nonzero DASB. The three quantities answer different questions.

2. Who should use aiDIF

aiDIF is for psychometricians, assessment researchers, and quantitative analysts who already have item calibrations from both scoring conditions and want an integrated workflow for robust DIF estimation, cross-scoring comparison, diagnostics, and visualisation.

It assumes comfort with IRT calibration output: slope and intercept estimates and their asymptotic covariance matrices. aiDIF operates after calibration. It never sees respondent-level item responses, which keeps it independent of any particular estimation package but means the quality of its input is your responsibility.

3. The example data

make_aidif_eg() returns item parameter estimates for six items in two groups under both scoring conditions, with a known structure:

Item 1 carries DIF under human scoring (+0.5 in the focal group).
Item 3 carries DASB: AI scoring adds +0.4 to the focal group only.
Latent impact is 0.5 SD.
A uniform +0.1 AI calibration offset affects all items in both groups.

Item 1 is the interesting contrast. It has real DIF, but the AI reproduces it exactly, so its DASB should be near zero.

eg <- make_aidif_eg()
str(eg, max.level = 2)
#> List of 2
#>  $ human:List of 3
#>   ..$ par.names:List of 2
#>   ..$ est      :List of 2
#>   ..$ var.cov  :List of 2
#>  $ ai   :List of 3
#>   ..$ par.names:List of 2
#>   ..$ est      :List of 2
#>   ..$ var.cov  :List of 2

4. Calibration and identification

This section matters more than its length suggests.

DASB is a difference of differences across four calibrations: two groups times two scoring conditions. If each of those calibrations carries its own arbitrary metric origin \(c_{gs}\), then the constants contribute

\[(c_{FA} - c_{FH} - c_{RA} + c_{RH})\]

to every item’s DASB. Raw DASB is therefore identified only up to an item-common additive offset, and that offset is perfectly confounded with a uniform DASB affecting all items equally.

There are two ways to proceed.

Common metric. If linking was established upstream, so that the four calibrations are demonstrably on the same metric, the offset is zero and raw DASB is interpretable as it stands. This is the default.

fit_common <- fit_aidif(eg$human, eg$ai, metric = "common")

Linked. Otherwise, estimate the offset from the data. aiDIF applies the same bi-square M-estimator it already uses for within-condition robust scaling to the DASB vector, and reports DASB relative to the estimated offset. This assumes most items have no scoring-condition-by-group effect — the same majority assumption that underlies robust DIF scaling generally.

fit_linked <- fit_aidif(eg$human, eg$ai, metric = "linked")
attr(fit_linked$scoring_bias, "offset")
#> [1] -0.005274666

Do not skip this choice. Subtracting intercepts from separately normalised latent scales without a defensible linking strategy produces numbers that look fine and mean nothing.

5. Independent and paired scoring designs

The variance of DASB depends on how the two conditions were produced.

Independent (the default) sums the four marginal variances. It is correct when the human and AI calibrations come from disjoint respondent samples.

Paired — the same responses scored twice — requires the cross-condition covariance within each group. Groups remain independent of each other, since they consist of disjoint respondents:

\[\mathrm{Var}(\mathrm{DASB}_i) = \sum_{g,s}\mathrm{Var}(d_{igs}) - 2\mathrm{Cov}(d_{iFH}, d_{iFA}) - 2\mathrm{Cov}(d_{iRH}, d_{iRA}).\]

That covariance is normally positive, so the independence variance is an upper bound for paired data. Using design = "independent" on paired data gives conservative p-values, not anti-conservative ones.

No single calibration produces the cross-condition covariance. From 0.3.0, paired_cross_cov() estimates it from the raw item scores by resampling examinees within group and refitting both calibrations on each resample (Section 5a). If you already hold the matrix, pass it directly:

fit_paired <- fit_aidif(eg$human, eg$ai,
                        design    = "paired",
                        cross_cov = my_cross_cov)

5a. Starting from item scores

The usual paired study has one data set of responses (for example, essays) and two score matrices: human ratings and AI ratings of the same responses, in the same row order. fit_aidif_responses() calibrates both conditions in mirt, estimates the cross-condition covariance by bootstrap, and runs the whole analysis. The example below uses simulate_aidif_responses(), which generates paired graded scores with a known DASB on item 3.

sim <- simulate_aidif_responses(n_per_group = 400, n_items = 6, n_cat = 3,
                                dasb_items = 3, dasb_mag = 0.6, seed = 21)
sim
#> Simulated paired human/AI scores
#>   Examinees: 400 reference, 400 focal   Items: 6   Categories: 3
#>   Exact human-AI agreement: 0.762
#>   True DASB items: item3

fit_resp <- fit_aidif_responses(sim$human, sim$ai, sim$group,
                                groups = c("reference", "focal"),
                                n_boot = 30, seed = 1)
fit_resp$cross_cov
#> Cross-condition covariance (paired design)
#>   Type: correlation   Bootstrap replicates used: 30 of 30
#>   group.1: human-AI correlation of matching parameters, median 0.620 (range 0.368 to 0.757)
#>   group.2: human-AI correlation of matching parameters, median 0.611 (range 0.240 to 0.914)
fit_resp$scoring_bias
#> Differential AI Scoring Bias
#>   Design: paired   Metric: linked   Adjustment: none
#>   Estimated item-common metric offset: -0.0544
#> 
#>          shift_g1 shift_g2    DASB     se   lower  upper       z  p_val
#> item1_d1   0.2240   0.4736  0.3039 0.3082 -0.3000 0.9079  0.9863 0.3240
#> item1_d2   0.1899   0.1640  0.0284 0.1743 -0.3132 0.3701  0.1631 0.8705
#> item2_d1   0.0879   0.3214  0.2879 0.2104 -0.1245 0.7003  1.3681 0.1713
#> item2_d2   0.3132  -0.1061 -0.3649 0.1932 -0.7436 0.0138 -1.8884 0.0590
#> item3_d1   0.4689   0.3832 -0.0314 0.3542 -0.7255 0.6628 -0.0885 0.9295
#> item3_d2  -0.0866   0.8311  0.9721 0.1940  0.5918 1.3523  5.0104 0.0000
#> item4_d1   0.2460  -0.0420 -0.2337 0.2099 -0.6450 0.1776 -1.1136 0.2654
#> item4_d2   0.0268  -0.0998 -0.0722 0.1331 -0.3330 0.1886 -0.5425 0.5874
#> item5_d1   0.0482  -0.0329 -0.0267 0.2909 -0.5968 0.5434 -0.0918 0.9268
#> item5_d2   0.0805   0.0561  0.0299 0.1080 -0.1819 0.2416  0.2767 0.7820
#> item6_d1  -0.0103  -0.0480  0.0168 0.3925 -0.7526 0.7861  0.0427 0.9659
#> item6_d2   0.1318   0.0720 -0.0054 0.1657 -0.3302 0.3194 -0.0326 0.9740

fit_aidif_responses() uses metric = "linked" by default. Each group’s trait is standardised in both calibrations, so an AI engine that flattens or steepens slopes shifts every item’s raw DASB whenever the groups differ in mean trait. In a small simulation (AI slopes 0.8 times human, impact 0.5 SD, no true DASB), raw DASB rejected at about 16% for a nominal 5%; the linked metric held 5%.

Thirty replicates keep this vignette fast; use 200 or more in practice. With se = "bootstrap" the DASB standard errors come from the same replicates, and under metric = "linked" they include the variability of the estimated offset. cores runs the replicates in parallel on any platform.

6. Fitting

mod <- fit_aidif(eg$human, eg$ai, metric = "linked", adjust = "BH")
print(mod)
#> AI-DIF Analysis
#> ----------------------------------------------
#> Human scoring  - robust scale est: -0.5776  (SE: 0.0747)
#>                - DIF items flagged: 3 / 6
#> AI scoring     - robust scale est: -0.5921  (SE: 0.0748)
#>                - DIF items flagged: 3 / 6
#> DASB test      - items with differential AI bias: 1 / 6 (BH-adjusted)
#>                - design: independent   metric: linked

7. Conventional DIF in each condition

The within-condition step is the robust scaling estimator of Halpin (2024), also implemented in the robustDIF package. compare_robustdif() runs both on the same estimates; they agree to machine precision.

compare_robustdif(eg$human)
#>   scale_by robustDIF_fun      aiDIF  robustDIF     abs_diff
#> 1   pooled        d_fun3 -0.5775923 -0.5775923 1.110223e-16
#> 2      ref        d_fun1 -0.6022755 -0.6022755 0.000000e+00
#> 3    focal        d_fun2 -0.5566159 -0.5566159 0.000000e+00

round(mod$dif_human, 4)
#>         delta     se       z  p_val
#> item1  0.5693 0.0759  7.4995 0.0000
#> item2  0.0366 0.1060  0.3448 0.7303
#> item3  0.2302 0.0623  3.6953 0.0002
#> item4  0.0163 0.0931  0.1756 0.8606
#> item5  0.2700 0.0693  3.8947 0.0001
#> item6 -0.1181 0.1232 -0.9584 0.3379
round(mod$dif_ai, 4)
#>         delta     se       z  p_val
#> item1  0.5756 0.0761  7.5596 0.0000
#> item2  0.0466 0.1046  0.4458 0.6557
#> item3  0.5499 0.0619  8.8820 0.0000
#> item4  0.0046 0.0926  0.0495 0.9605
#> item5  0.3308 0.0695  4.7559 0.0000
#> item6 -0.1455 0.1240 -1.1737 0.2405

8. DASB, intervals, and multiplicity

Six items means six tests. adjust applies a multiplicity correction across the items in this analysis; the family is the item set, not the union of the human-DIF, AI-DIF, and DASB tests.

mod$scoring_bias
#> Differential AI Scoring Bias
#>   Design: independent   Metric: linked   Adjustment: BH
#>   Estimated item-common metric offset: -0.0053
#>   Note: independence variance. Conservative if the same responses
#>         were scored under both conditions.
#> 
#>       shift_g1 shift_g2    DASB     se   lower  upper       z  p_val  p_adj
#> item1     0.13     0.12 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> item2     0.08     0.07 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> item3     0.11     0.54  0.4353 0.1534  0.1346 0.7359  2.8378 0.0045 0.0273
#> item4     0.12     0.09 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697
#> item5     0.07     0.13  0.0653 0.1278 -0.1853 0.3158  0.5106 0.6096 0.9697
#> item6     0.11     0.08 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697

Item 3 carries the planted scoring-condition-by-group effect, so its DASB should depart from zero. Item 1 shows that conventional DIF can persist across scoring conditions without the AI having introduced any additional differential functioning.

9. Describing the pattern of change

ai_effect_summary(mod)
#>       human_delta     ai_delta human_flag ai_flag       status
#> item1  0.56933231  0.575603316       TRUE    TRUE   stable_dif
#> item2  0.03656025  0.046648397      FALSE   FALSE stable_clean
#> item3  0.23015316  0.549884606       TRUE    TRUE   stable_dif
#> item4  0.01634948  0.004582135      FALSE   FALSE stable_clean
#> item5  0.26998185  0.330766728       TRUE    TRUE   stable_dif
#> item6 -0.11810882 -0.145530542      FALSE   FALSE stable_clean

These labels are descriptive. They compare significance flags between the two conditions, and a difference in significance is not itself a significant difference (Gelman and Stern 2006). The DASB table in the previous section is the test of whether the contrast changed; this table describes the pattern once you have that answer.

10. Anchor diagnostics

Items downweighted to near zero were effectively excluded from the robust scale estimate, which flags them as likely DIF-contaminated.

anchor_weights(mod)
#>       human_weight ai_weight
#> item1    0.0000000 0.0000000
#> item2    0.9455790 0.9100670
#> item3    0.0000000 0.0000000
#> item4    0.9861794 0.9988997
#> item5    0.0000000 0.0000000
#> item6    0.6093599 0.4496091

11. Visualisation

plot(mod, type = "dif_forest")

Forest plot of item-level DIF estimates with confidence intervals under human and AI scoring.

plot(mod, type = "dasb")

Bar chart of DASB estimates per item with confidence intervals.

plot(mod, type = "weights")

Dot plot of bi-square anchor weights per item in each scoring condition.

plot(mod, type = "rho")

Bi-square objective function against candidate location values, with the robust estimate marked.

12. Full report

summary(mod)
#> ============================================================== 
#>  AI Differential Item Functioning Analysis (aiDIF)
#> ============================================================== 
#> 
#> Items: 6   Scaling: pooled   Alpha: 0.05
#> DASB design: independent   Metric: linked   Adjustment: BH
#> 
#> --- Human scoring DIF ---------------------------------------
#>   Robust scale estimate: -0.5776  (SE: 0.0747)
#>   Iterations: 10   Multiple solutions: no
#> 
#>   Wald DIF tests:
#>         delta     se       z  p_val
#> item1  0.5693 0.0759  7.4995 0.0000
#> item2  0.0366 0.1060  0.3448 0.7303
#> item3  0.2302 0.0623  3.6953 0.0002
#> item4  0.0163 0.0931  0.1756 0.8606
#> item5  0.2700 0.0693  3.8947 0.0001
#> item6 -0.1181 0.1232 -0.9584 0.3379
#> 
#> --- AI scoring DIF ------------------------------------------
#>   Robust scale estimate: -0.5921  (SE: 0.0748)
#>   Iterations: 11   Multiple solutions: no
#> 
#>   Wald DIF tests:
#>         delta     se       z  p_val
#> item1  0.5756 0.0761  7.5596 0.0000
#> item2  0.0466 0.1046  0.4458 0.6557
#> item3  0.5499 0.0619  8.8820 0.0000
#> item4  0.0046 0.0926  0.0495 0.9605
#> item5  0.3308 0.0695  4.7559 0.0000
#> item6 -0.1455 0.1240 -1.1737 0.2405
#> 
#> --- Differential AI Scoring Bias (DASB) ----------------------
#>   H0: the human-to-AI scoring shift is the same in both groups
#>   (Positive DASB => AI scoring disadvantages the focal group)
#> 
#> Differential AI Scoring Bias
#>   Design: independent   Metric: linked   Adjustment: BH
#>   Estimated item-common metric offset: -0.0053
#>   Note: independence variance. Conservative if the same responses
#>         were scored under both conditions.
#> 
#>       shift_g1 shift_g2    DASB     se   lower  upper       z  p_val  p_adj
#> item1     0.13     0.12 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> item2     0.08     0.07 -0.0047 0.1243 -0.2484 0.2390 -0.0380 0.9697 0.9697
#> item3     0.11     0.54  0.4353 0.1534  0.1346 0.7359  2.8378 0.0045 0.0273
#> item4     0.12     0.09 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697
#> item5     0.07     0.13  0.0653 0.1278 -0.1853 0.3158  0.5106 0.6096 0.9697
#> item6     0.11     0.08 -0.0247 0.1248 -0.2694 0.2200 -0.1981 0.8430 0.9697
#> 
#> --- AI-effect classification ---------------------------------
#>   Descriptive only. It compares significance flags between the two
#>   conditions; a change in flag status is not itself evidence of a
#>   change in effect (Gelman & Stern, 2006). The DASB table above is
#>   the test of whether the group contrast actually changed.
#> 
#>       human_delta     ai_delta human_flag ai_flag       status
#> item1  0.56933231  0.575603316       TRUE    TRUE   stable_dif
#> item2  0.03656025  0.046648397      FALSE   FALSE stable_clean
#> item3  0.23015316  0.549884606       TRUE    TRUE   stable_dif
#> item4  0.01634948  0.004582135      FALSE   FALSE stable_clean
#> item5  0.26998185  0.330766728       TRUE    TRUE   stable_dif
#> item6 -0.11810882 -0.145530542      FALSE   FALSE stable_clean
#> 
#>   Status counts:
#> 
#> stable_clean   stable_dif 
#>            3            3

summary() returns an object, so components can be extracted rather than scraped from printed output:

s <- summary(mod, adjust = "holm")
s$dasb$p_adj
#> [1] 1.00000000 1.00000000 0.02725664 1.00000000 1.00000000 1.00000000

13. Using aiDIF after calibration with mirt

as_aidif() converts a fitted multiple-group model directly into the structure aiDIF expects, replacing the manual parameter and covariance extraction earlier versions required. mirt is in Suggests, so this section runs only when it is installed.

set.seed(2026)
n <- 600
J <- 8

a <- runif(J, 0.8, 1.6)
d <- rnorm(J, 0, 0.6)

sim_group <- function(n, a, d, mu = 0, d_shift = rep(0, length(d))) {
  th <- rnorm(n, mu, 1)
  p  <- t(vapply(th, function(t) plogis(a * t + d + d_shift), numeric(length(d))))
  matrix(rbinom(length(p), 1, p), nrow = n)
}

# Human scoring: item 1 has DIF in the focal group.
dif_h <- c(0.5, rep(0, J - 1))
# AI scoring adds a uniform +0.1 drift, plus DASB at item 3.
drift <- rep(0.1, J)
dasb  <- c(0, 0, 0.4, rep(0, J - 5), 0, 0)

human <- rbind(sim_group(n, a, d),
               sim_group(n, a, d + dif_h, mu = 0.5))
ai    <- rbind(sim_group(n, a, d + drift),
               sim_group(n, a, d + dif_h + drift + dasb, mu = 0.5))
grp   <- rep(c("reference", "focal"), each = n)

colnames(human) <- colnames(ai) <- paste0("item", seq_len(J))

# No invariance constraints: each group is calibrated on its own metric with
# the latent mean fixed at 0 and variance at 1. That is exactly the input
# robust scaling expects -- it estimates the linking constant itself, so
# constraining anchors here would pre-empt the method.
human_mod <- mirt::multipleGroup(as.data.frame(human), 1, group = grp,
                                 itemtype = "2PL", SE = TRUE, verbose = FALSE)
ai_mod    <- mirt::multipleGroup(as.data.frame(ai), 1, group = grp,
                                 itemtype = "2PL", SE = TRUE, verbose = FALSE)

human_in <- as_aidif(human_mod, groups = c("reference", "focal"))
ai_in    <- as_aidif(ai_mod,    groups = c("reference", "focal"))

fit_mirt <- fit_aidif(human_in, ai_in, metric = "linked", adjust = "BH")
print(fit_mirt)
#> AI-DIF Analysis
#> ----------------------------------------------
#> Human scoring  - robust scale est: 0.2777  (SE: 0.0812)
#>                - DIF items flagged: 2 / 8
#> AI scoring     - robust scale est: 0.3746  (SE: 0.1077)
#>                - DIF items flagged: 3 / 8
#> DASB test      - items with differential AI bias: 1 / 8 (BH-adjusted)
#>                - design: independent   metric: linked
fit_mirt$scoring_bias
#> Differential AI Scoring Bias
#>   Design: independent   Metric: linked   Adjustment: BH
#>   Estimated item-common metric offset: 0.0595
#>   Note: independence variance. Conservative if the same responses
#>         were scored under both conditions.
#> 
#>       shift_g1 shift_g2    DASB     se   lower  upper       z  p_val  p_adj
#> item1  -0.0545   0.2439  0.2389 0.2313 -0.2144 0.6922  1.0328 0.3017 0.7535
#> item2   0.3266   0.2694 -0.1168 0.2648 -0.6358 0.4023 -0.4409 0.6593 0.7535
#> item3  -0.0707   0.6101  0.6213 0.2122  0.2054 1.0371  2.9283 0.0034 0.0273
#> item4   0.1515   0.0991 -0.1119 0.1789 -0.4624 0.2387 -0.6254 0.5317 0.7535
#> item5  -0.0007   0.2136  0.1548 0.2201 -0.2765 0.5861  0.7033 0.4819 0.7535
#> item6   0.1598   0.1210 -0.0983 0.1653 -0.4222 0.2257 -0.5945 0.5522 0.7535
#> item7   0.2592   0.4245  0.1059 0.1953 -0.2768 0.4886  0.5422 0.5877 0.7535
#> item8   0.0966   0.1374 -0.0187 0.2083 -0.4270 0.3895 -0.0900 0.9283 0.9283

14. Simulating item-parameter inputs

simulate_aidif_data() does not simulate respondent-level item responses and refit IRT models. It directly generates item-parameter estimates and their asymptotic covariance matrices, consistent with a 2PL fitted to n_obs observations per group. It is intended for package demonstration and method benchmarking, not as a substitute for a full response-level simulation study. For the latter, use simulate_aidif_responses() with fit_aidif_responses() (Section 5a).

dat <- simulate_aidif_data(n_items = 8, n_obs = 600,
                           dif_items = c(1, 2), dif_mag = 0.5,
                           dasb_items = 5, dasb_mag = 0.4, seed = 123)
sim_mod <- fit_aidif(dat$human, dat$ai, metric = "linked", adjust = "BH")
print(sim_mod)
#> AI-DIF Analysis
#> ----------------------------------------------
#> Human scoring  - robust scale est: -0.2670  (SE: 0.0322)
#>                - DIF items flagged: 4 / 8
#> AI scoring     - robust scale est: 0.0536  (SE: 0.0363)
#>                - DIF items flagged: 5 / 8
#> DASB test      - items with differential AI bias: 1 / 8 (BH-adjusted)
#>                - design: independent   metric: linked

15. Limitations

The parameter-level functions work from calibrated item parameters and cannot detect problems introduced during calibration itself. fit_aidif_responses() calibrates with a standard two-group model; check its fit before interpreting DASB.
The linked metric assumes a majority of items are free of scoring-condition-by-group effects. If most items are affected, the estimated offset absorbs the signal and DASB estimates shrink toward zero.
If the AI engine changes item slopes and the groups differ in mean trait, raw DASB is shifted on every item in proportion to the item’s slope. The linked metric removes this when the shift is roughly common across items; with very heterogeneous slopes some residual bias can remain.
Variance under metric = "linked" is a first-order approximation that accounts for each item’s leverage on the estimated offset but treats the bi-square weights as fixed.
design = "paired" uses only the diagonal of the supplied cross-condition covariance. The bootstrap estimate from paired_cross_cov() needs every score category to be observed in every group and condition; collapse sparse categories first.
Only unidimensional models with a single slope per item are supported.
The AI-effect classification is descriptive, not inferential.