---
title: "Getting Started with aiEvalR"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started with aiEvalR}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
set.seed(2026)
```

```{r setup}
library(aiEvalR)
```

## Why evaluate an AI system like a test?

Most AI evaluation reduces a system to a single accuracy-style score. But
a language model used to score essays, answer questions, or make
recommendations is a *measurement instrument*, and psychometrics has
spent a century developing tools for exactly this: reliability, validity,
fairness, calibration, and error structure. `aiEvalR` brings that
toolkit to AI evaluation.

This vignette tours the package module by module with small runnable
examples. Throughout, we simulate AI outputs where we need repeated
model responses (there is no built-in "call an LLM" step), and use
built-in R data where an ordinary regression suffices.

## Reliability: is the system consistent?

Treat repeated calls to the same prompt the way classical test theory
treats repeated administrations of a test. Here, rows are prompts and
columns are repeated response occasions.

```{r reliability}
# 12 prompts, each answered on 3 separate occasions (e.g. repeated
# sampling). Scores are stable across occasions -> high reliability.
latent <- rnorm(12, mean = 5)
responses <- sapply(1:3, function(occasion) latent + rnorm(12, sd = 0.3))

rel <- ai_reliability(responses)
rel$test_retest$icc
```

An intraclass correlation near 1 means the system answers the same
prompt consistently across occasions. A bootstrap interval expresses the
uncertainty in that estimate:

```{r reliability-boot}
boot <- ai_bootstrap_reliability(responses, n_boot = 200, seed = 1)
c(estimate = boot$estimate, lower = boot$ci_lower, upper = boot$ci_upper)
```

## Robustness: does the answer survive small changes?

If trivial rewordings of the same prompt swing the output, the system is
brittle. `stress_test()` compares a baseline against perturbed versions.

```{r robustness}
baseline  <- rnorm(100, mean = 5, sd = 1)
# a mild perturbation and a severe one, applied to the same prompts
mild   <- baseline + rnorm(100, mean = 0.1, sd = 0.05)
severe <- baseline + rnorm(100, mean = 1.0, sd = 0.05)

st <- stress_test(baseline, list(mild = mild, severe = severe))
st$robustness_index
```

The `robustness_index` is a descriptive summary (higher = more robust);
see `?stress_test` for its limitations. `prompt_sensitivity()`
separately quantifies how much output varies across paraphrases of the
same underlying prompt.

## Fairness: do outcomes differ by group?

`ai_group_disparity()` reports demographic parity and, given binary
decisions, equalized-odds gaps. It is named "disparity," not "bias,"
deliberately: a group difference is not automatically evidence of an
unfair process.

```{r fairness}
outcome <- rbinom(200, 1, 0.4)
group   <- sample(c("A", "B"), 200, replace = TRUE)

disp <- ai_group_disparity(outcome, group)
disp$demographic_parity_diff
```

For item-level, IRT-based differential AI scoring bias, `aiEvalR`
integrates with the companion `aiDIF` package rather than reimplementing
it -- see `?ai_fairness`.

## Calibration: are stated confidences trustworthy?

If a system says "90% confident," it should be right about 90% of the
time. Expected Calibration Error and the Brier score quantify the gap.

```{r calibration}
# Well-calibrated: predicted probabilities match empirical frequencies
confidence <- runif(500)
outcome    <- rbinom(500, 1, confidence)

cal <- ai_calibration(outcome, confidence, n_bins = 10)
c(ECE = cal$ece, Brier = cal$brier)
```

## Hallucination: aggregating adjudicated labels

`aiEvalR` does not itself fact-check (that needs a retrieval/NLI
component). It aggregates verdicts you supply, and offers a lexical
baseline for overlap with a source.

```{r hallucination}
# externally adjudicated: TRUE = claim unsupported
verdicts <- c(FALSE, FALSE, TRUE, FALSE, TRUE)
hallucination_rate(verdicts)

# lexical overlap is NOT factual consistency -- note the name
lexical_overlap("the treatment reduces mortality",
                "the treatment does not reduce mortality")
```

Notice the two opposite-meaning sentences score as highly similar:
`lexical_overlap()` measures token overlap, not agreement, which is
exactly why it is named that way.

## An integrated dashboard

Once you have module-level scores rescaled to `[0, 1]`, `ai_dashboard()`
combines them into an overall profile.

```{r dashboard}
ai_dashboard(
  reliability   = 0.94,
  fairness      = 0.82,
  robustness    = 0.70,
  calibration   = 0.88,
  hallucination = 0.81
)
```

The dashboard is a descriptive profile, not a validated composite index;
the "grade" is a convenience, and you should choose weights deliberately
for your context.

## Where to next

The reliability and psychometric-quality modules are the package's
methodological core. For the full generalizability-theory workflow
(G-studies and D-studies), see
`vignette("psychometric-core")`.