README

aiEvalR evaluates AI systems the way psychometricians evaluate tests — as measurement instruments with reliability, fairness, calibration, and error structure — rather than as black boxes scored on accuracy alone.

Modules

Module	What it does
Psychometric quality	The methodological core. Multi-facet generalizability theory (case × prompt × model × run) via `lme4`, with a G-study/D-study workflow. Prompts become “items,” AI outputs become “item responses.”
Reliability	Test-retest (ICC), internal consistency (alpha), bootstrap intervals.
Robustness	Stress testing, prompt sensitivity, adversarial stability.
Fairness	Group disparity + equalized odds; integrates with (does not duplicate) the `aiDIF` package for item-level IRT-based AI-scoring bias.
Calibration	ECE, Brier score, calibration curves, input-validated.
Hallucination	Aggregation of adjudicated verdicts; lexical-overlap baseline.
Dashboard	Descriptive multi-module profile (not a validated composite score).

Module

What it does

Psychometric quality

The methodological core. Multi-facet generalizability theory (case × prompt × model × run) via lme4, with a G-study/D-study workflow. Prompts become “items,” AI outputs become “item responses.”

Reliability

Test-retest (ICC), internal consistency (alpha), bootstrap intervals.

Robustness

Stress testing, prompt sensitivity, adversarial stability.

Fairness

Group disparity + equalized odds; integrates with (does not duplicate) the aiDIF package for item-level IRT-based AI-scoring bias.

Calibration

ECE, Brier score, calibration curves, input-validated.

Hallucination

Aggregation of adjudicated verdicts; lexical-overlap baseline.

Dashboard

Descriptive multi-module profile (not a validated composite score).

Quick examples

Generalizability (the core)

library(aiEvalR)

# long-format: one row per case x prompt x model x run
g <- ai_generalizability(data, score = "score", case = "case",
                          facets = c("prompt", "model", "run"))
print(g)  # G-study variance components

# D-study: project dependability for a specific measurement procedure
ai_dstudy(g, n_prompt = 3, n_model = 2, n_run = 1)

Fairness (Module 2) — integrates with aiDIF

# outcome-level group disparity (this package)
ai_group_disparity(outcome, group, predicted_class = decisions)

# item-level IRT-based AI-scoring bias -> dispatches to aiDIF if installed
ai_fairness(outcome, group, human_mle = h, ai_mle = a)

aiEvalR’s fairness functions and the causalfragility-lab/aiDIF package answer different questions: aiEvalR checks whether overall outcomes/decisions differ by group; aiDIF checks whether an AI scoring engine shifts specific item difficulties differently by group (via paired human/AI IRT calibration and a Differential AI Scoring Bias test). Use both, not one instead of the other. aiDIF is on CRAN (install.packages("aiDIF")).

Naming notes

Some functions carry deliberately precise names to avoid overclaiming what they measure:

Function	Rationale
`lexical_overlap()`	Measures token overlap (Jaccard); named to make clear it does NOT detect negation/contradiction or factual consistency.
`empirical_output_interval()`	Describes the observed spread of repeated outputs; named to avoid implying calibrated prediction-interval coverage guarantees.
`ai_group_disparity()`	A between-group difference is a disparity statistic, not automatically “bias” without further context.
`ai_dif_mh()`	A basic Mantel-Haenszel screening helper; named to distinguish it from the fuller IRT-based `aiDIF` package.

Function

Rationale

lexical_overlap()

Measures token overlap (Jaccard); named to make clear it does NOT detect negation/contradiction or factual consistency.

empirical_output_interval()

Describes the observed spread of repeated outputs; named to avoid implying calibrated prediction-interval coverage guarantees.

ai_group_disparity()

A between-group difference is a disparity statistic, not automatically “bias” without further context.

ai_dif_mh()

A basic Mantel-Haenszel screening helper; named to distinguish it from the fuller IRT-based aiDIF package.

Installation

devtools::install_github("causalfragility-lab/aiEvalR")

Citation

Hait, S. (2026). aiEvalR: Statistical and Psychometric Evaluation of AI Systems. R package version 0.1.0.

Generalizability-theory foundation:

Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The Dependability of Behavioral Measurements. Wiley.

Brennan, R. L. (2001). Generalizability Theory. Springer.

aiEvalR