Getting Started with aiEvalR

Why evaluate an AI system like a test?

Most AI evaluation reduces a system to a single accuracy-style score. But a language model used to score essays, answer questions, or make recommendations is a measurement instrument, and psychometrics has spent a century developing tools for exactly this: reliability, validity, fairness, calibration, and error structure. aiEvalR brings that toolkit to AI evaluation.

This vignette tours the package module by module with small runnable examples. Throughout, we simulate AI outputs where we need repeated model responses (there is no built-in “call an LLM” step), and use built-in R data where an ordinary regression suffices.

Reliability: is the system consistent?

Treat repeated calls to the same prompt the way classical test theory treats repeated administrations of a test. Here, rows are prompts and columns are repeated response occasions.

# 12 prompts, each answered on 3 separate occasions (e.g. repeated
# sampling). Scores are stable across occasions -> high reliability.
latent <- rnorm(12, mean = 5)
responses <- sapply(1:3, function(occasion) latent + rnorm(12, sd = 0.3))

rel <- ai_reliability(responses)
rel$test_retest$icc
#> [1] 0.8310085

An intraclass correlation near 1 means the system answers the same prompt consistently across occasions. A bootstrap interval expresses the uncertainty in that estimate:

boot <- ai_bootstrap_reliability(responses, n_boot = 200, seed = 1)
c(estimate = boot$estimate, lower = boot$ci_lower, upper = boot$ci_upper)
#>  estimate     lower     upper 
#> 0.8310085 0.4681429 0.9351861

Robustness: does the answer survive small changes?

If trivial rewordings of the same prompt swing the output, the system is brittle. stress_test() compares a baseline against perturbed versions.

baseline  <- rnorm(100, mean = 5, sd = 1)
# a mild perturbation and a severe one, applied to the same prompts
mild   <- baseline + rnorm(100, mean = 0.1, sd = 0.05)
severe <- baseline + rnorm(100, mean = 1.0, sd = 0.05)

st <- stress_test(baseline, list(mild = mild, severe = severe))
st$robustness_index
#> [1] 0.4790861

The robustness_index is a descriptive summary (higher = more robust); see ?stress_test for its limitations. prompt_sensitivity() separately quantifies how much output varies across paraphrases of the same underlying prompt.

Fairness: do outcomes differ by group?

ai_group_disparity() reports demographic parity and, given binary decisions, equalized-odds gaps. It is named “disparity,” not “bias,” deliberately: a group difference is not automatically evidence of an unfair process.

outcome <- rbinom(200, 1, 0.4)
group   <- sample(c("A", "B"), 200, replace = TRUE)

disp <- ai_group_disparity(outcome, group)
disp$demographic_parity_diff
#> [1] 0.06936243

For item-level, IRT-based differential AI scoring bias, aiEvalR integrates with the companion aiDIF package rather than reimplementing it – see ?ai_fairness.

Calibration: are stated confidences trustworthy?

If a system says “90% confident,” it should be right about 90% of the time. Expected Calibration Error and the Brier score quantify the gap.

# Well-calibrated: predicted probabilities match empirical frequencies
confidence <- runif(500)
outcome    <- rbinom(500, 1, confidence)

cal <- ai_calibration(outcome, confidence, n_bins = 10)
c(ECE = cal$ece, Brier = cal$brier)
#>        ECE      Brier 
#> 0.02963466 0.16165430

Hallucination: aggregating adjudicated labels

aiEvalR does not itself fact-check (that needs a retrieval/NLI component). It aggregates verdicts you supply, and offers a lexical baseline for overlap with a source.

# externally adjudicated: TRUE = claim unsupported
verdicts <- c(FALSE, FALSE, TRUE, FALSE, TRUE)
hallucination_rate(verdicts)
#> [1] 0.4

# lexical overlap is NOT factual consistency -- note the name
lexical_overlap("the treatment reduces mortality",
                "the treatment does not reduce mortality")
#> [1] 0.4285714

Notice the two opposite-meaning sentences score as highly similar: lexical_overlap() measures token overlap, not agreement, which is exactly why it is named that way.

An integrated dashboard

Once you have module-level scores rescaled to [0, 1], ai_dashboard() combines them into an overall profile.

ai_dashboard(
  reliability   = 0.94,
  fairness      = 0.82,
  robustness    = 0.70,
  calibration   = 0.88,
  hallucination = 0.81
)
#> AI Evaluation Dashboard
#> ========================
#> 
#> Reliability        █████████░ 0.94
#> Fairness           ████████░░ 0.82
#> Robustness         ███████░░░ 0.70
#> Calibration        █████████░ 0.88
#> Hallucination      ████████░░ 0.81
#> 
#> Overall: 0.83  (Grade: B)

The dashboard is a descriptive profile, not a validated composite index; the “grade” is a convenience, and you should choose weights deliberately for your context.

Where to next

The reliability and psychometric-quality modules are the package’s methodological core. For the full generalizability-theory workflow (G-studies and D-studies), see vignette("psychometric-core").