LLMRpanel administers survey and experimental instruments to panels
of language model personas. Use it to pretest questions, pilot conjoint
designs, calculate sample sizes from pilot dispersion, or measure
responses from a configured model. panel_benchmark()
compares closed-item response shares with benchmark shares supplied by
the user. It records deviations by item and response, nonresponse, and
the number of closed items covered.
Every example below executes offline through the .runner
seam: a deterministic function stands in for the model, so the workflow
runs during package builds without credentials or charges. One chunk
near the end, gated by RUN_LIVE, shows the same call
against a live provider.
Instrument pretesting. Administer draft items and inspect unmatched replies and first-option sensitivity.
Design piloting. Run conjoint tasks and estimate response dispersion before planning human data collection.
Model measurement. Compare response distributions across personas, item orders, option orders, or model configurations.
library(LLMRpanel)
# A deterministic stand-in for the model: it always picks the first
# DISPLAYED option of a closed item (so option-order randomization shows up
# in the shares) and answers open items with a fixed sentence.
first_shown <- function(experiments, ...) {
experiments$response_text <- vapply(seq_len(nrow(experiments)), function(i) {
shown <- experiments$option_order[i]
if (is.na(shown)) "Because it reaches the most people."
else strsplit(shown, "|", fixed = TRUE)[[1]][1]
}, character(1))
experiments$success <- TRUE
experiments
}panel_from_margins() samples attribute values from the
supplied marginal distributions. set.seed() makes this draw
reproducible.
set.seed(110)
panel = panel_from_margins(
list(
age = c("18 to 34" = .30, "35 to 64" = .45, "65 plus" = .25),
party = c(left = .45, right = .45, independent = .10)
),
n = 12,
persona_template = "A {age} year old voter who leans {party}."
)
panel
#> <silicon_panel | 12 persona(s) | attributes: age, party>
#> e.g. A 18 to 34 year old voter who leans left.
instrument = panel_instrument(list(
item_likert("wk4", "A four day work week would benefit society."),
item_choice(
"fund",
"Which should the city fund first?",
c("public transit", "road repair")
),
item_open("why", "In one sentence, why?")
))
instrument
#> <panel_instrument | 3 item(s) | randomized: option_order>
#> [wk4] (likert) A four day work week would benefit society.
#> [fund] (choice) Which should the city fund first?
#> [why] (open) In one sentence, why?Margins are useful when targets are published as tables. When
microdata is available, panel_from_data() is the joint
distribution counterpart. It draws personas from observed rows and
therefore preserves relationships among attributes rather than sampling
each margin independently. LLMR::report() identifies
whether a panel came from supplied margins, microdata rows, or supplied
personas.
panel_administer() sends each item to each persona as an
independent request. It returns a panel_responses object
with response rows in $data and the panel, instrument,
benchmark record, and token usage in separate components. It randomizes
option order per response (a Likert scale is shown reversed for a random
half) and records item_position (the item’s fixed position
in the instrument) and option_order (what this respondent
saw) in $data. When comparing models, reset the seed before
each administration so both models face the same option-order
assignments.
cfg = LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0.8)
cfg_qwen = LLMR::llm_config("groq", "qwen/qwen3-32b", temperature = 0.8)
set.seed(110)
resp = panel_administer(panel, instrument, cfg, .runner = first_shown)
resp
#> <panel_responses | 12 persona(s) x 3 item(s) | 0 execution failure(s), 0 parse failure(s)>
#> NOT BENCHMARKED: no benchmark comparison has been run; see panel_benchmark().
resp$data
#> # A tibble: 36 × 14
#> persona_id item_id type item_position option_order response_text response_id
#> <int> <chr> <chr> <int> <chr> <chr> <chr>
#> 1 1 wk4 like… 1 strongly ag… strongly agr… <NA>
#> 2 1 fund choi… 2 road repair… road repair <NA>
#> 3 1 why open 3 <NA> Because it r… <NA>
#> 4 2 wk4 like… 1 strongly di… strongly dis… <NA>
#> 5 2 fund choi… 2 public tran… public trans… <NA>
#> 6 2 why open 3 <NA> Because it r… <NA>
#> 7 3 wk4 like… 1 strongly di… strongly dis… <NA>
#> 8 3 fund choi… 2 public tran… public trans… <NA>
#> 9 3 why open 3 <NA> Because it r… <NA>
#> 10 4 wk4 like… 1 strongly di… strongly dis… <NA>
#> # ℹ 26 more rows
#> # ℹ 7 more variables: success <lgl>, error_message <chr>, finish_reason <chr>,
#> # model <chr>, provider <chr>, response <chr>, score <dbl>
panel_bias_audit(resp)
#> # A tibble: 3 × 6
#> item_id n parse_failures execution_failures order_effect_p order_test_note
#> <chr> <int> <int> <int> <dbl> <chr>
#> 1 fund 12 0 0 NA sparse cells; …
#> 2 why 12 0 0 NA <NA>
#> 3 wk4 12 0 0 NA sparse cells; …
LLMR::diagnostics(resp)
#> # A tibble: 3 × 10
#> item_id n parse_failures execution_failures order_effect_p order_test_note
#> <chr> <int> <int> <int> <dbl> <chr>
#> 1 fund 12 0 0 NA sparse cells; …
#> 2 why 12 0 0 NA <NA>
#> 3 wk4 12 0 0 NA sparse cells; …
#> # ℹ 4 more variables: benchmark_state <chr>, items_covered <int>,
#> # items_total <int>, mean_abs_dev <dbl>
set.seed(110) # the same assignments for the second model
resp_qwen = panel_administer(panel, instrument, cfg_qwen,
.runner = first_shown)
panel_bias_audit(resp_qwen)
#> # A tibble: 3 × 6
#> item_id n parse_failures execution_failures order_effect_p order_test_note
#> <chr> <int> <int> <int> <dbl> <chr>
#> 1 fund 12 0 0 NA sparse cells; …
#> 2 why 12 0 0 NA <NA>
#> 3 wk4 12 0 0 NA sparse cells; …panel_benchmark() compares valid model response shares
with benchmark shares for matching item-response pairs. It also records
benchmark coverage and item-level nonresponse in
$benchmark. Before a benchmark is attached, response shares
describe the configured model under the supplied personas, not a human
population. bench_fund supplies shares for one closed
item.
bench_fund = data.frame(
item_id = rep("fund", 2),
response = c("public transit", "road repair"),
share = c(0.41, 0.59)
)
resp_partial = panel_benchmark(
resp,
bench_fund,
benchmark_name = "toy city survey"
)
resp_partial
#> <panel_responses | 12 persona(s) x 3 item(s) | 0 execution failure(s), 0 parse failure(s)>
#> PARTIALLY BENCHMARKED (1/2): vs 'toy city survey', mean abs. deviation 0.007 on covered items.bench_fund covers one of the instrument’s two closed
items. bench_all adds shares for wk4 and
covers both.
bench_all = rbind(
bench_fund,
data.frame(
item_id = rep("wk4", 5),
response = c(
"strongly disagree",
"disagree",
"neutral",
"agree",
"strongly agree"
),
share = c(.05, .20, .25, .35, .15)
)
)
resp = panel_benchmark(
resp,
bench_all,
benchmark_name = "toy city survey"
)
resp
#> <panel_responses | 12 persona(s) x 3 item(s) | 0 execution failure(s), 0 parse failure(s)>
#> BENCHMARKED against 'toy city survey' (2/2 items): mean abs. deviation 0.230 (max 0.450)
LLMR::report(resp)
#> BENCHMARKED (2/2 items). Against 'toy city survey': mean absolute deviation 0.230, max 0.450 (full table in $benchmark$table; nonresponse in $benchmark$nonresponse).
#> PANEL. 12 persona(s) drawn from supplied margins over: age, party.
#> RESPONSES. 36 total; 0 execution failure(s); 0 parse failure(s).
#> FIRST-OPTION SENSITIVITY (chi-squared p by item; small p = which option was listed first moved the answers):
#> fund n = 12 execution failures = 0 parse failures = 0 order p = n/a
#> why n = 12 execution failures = 0 parse failures = 0 order p = n/a
#> wk4 n = 12 execution failures = 0 parse failures = 0 order p = n/a
resp$benchmark$nonresponse
#> # A tibble: 2 × 4
#> item_id execution_failures execution_failure_rate nonresponse_rate
#> <chr> <int> <dbl> <dbl>
#> 1 fund 0 0 0
#> 2 wk4 0 0 0resp$benchmark$nonresponse gives the missing response
proportion for each closed item. The comparison shares use nonmissing
responses as their denominator.
conjoint_design() uses R’s random-number generator to
construct a classed design list. Its $profiles field
contains the initial profile table, and its $attributes
field contains the attribute universe. panel_administer()
draws the profiles each respondent sees. Set a seed before
administration to reproduce those respondent-level draws.
set.seed(110)
design = conjoint_design(
list(
price = c("low", "high"),
origin = c("domestic", "imported")
),
n_tasks = 4
)
design
#> <conjoint_design | 4 task(s) x 2 profile(s) | 2 attribute(s)>
design$profiles
#> # A tibble: 8 × 4
#> task profile price origin
#> <int> <int> <chr> <chr>
#> 1 1 1 high imported
#> 2 1 2 high domestic
#> 3 2 1 low imported
#> 4 2 2 low domestic
#> 5 3 1 low domestic
#> 6 3 2 high domestic
#> 7 4 1 low domestic
#> 8 4 2 high imported
design$attributes
#> $price
#> [1] "low" "high"
#>
#> $origin
#> [1] "domestic" "imported"conjoint_design() attempts to use distinct profiles
within each task in $profiles and warns when the attribute
space cannot supply them. conjoint_instrument() creates one
forced-choice item per task. Administration renders a fresh draw for
each respondent and records it with the response.
conjoint_amce() estimates from those recorded profiles
relative to the first level of each attribute and calculates standard
errors clustered by persona.
cj_instr = conjoint_instrument(design, "Which product would you buy?")
set.seed(110)
cj = panel_administer(panel, cj_instr, cfg, .runner = first_shown)
conjoint_amce(cj)
#> <conjoint_amce | 96 profile row(s) | 12 respondent(s) | 0 missing, 0 execution failure(s)>
#> # A tibble: 4 × 6
#> attribute level estimate std_error ci_lo ci_hi
#> <chr> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 price low 0 NA NA NA
#> 2 price high 0.0582 0.138 -0.244 0.361
#> 3 origin domestic 0 NA NA NA
#> 4 origin imported -0.135 0.0932 -0.340 0.0703conjoint_amce() returns a classed result with one row
for each observed attribute level. Baseline levels have estimate 0 and
missing standard errors. Other rows contain the estimated contrast and
95 percent interval. Run counts remain in separate columns.
The identical call against a live provider drops the
.runner argument. This chunk runs only when
RUN_LIVE is set to TRUE in the setup
chunk.
panel_administer() makes one request per persona-item
pair. Option randomization does not add requests. Provider prices and
prompt and response lengths determine cost. A versioned local model can
support later reruns when hosted endpoints change. The
$data field retains response_text,
response_id, success, model, and
provider, including when a reply cannot be matched to a
closed-item option. finish_reason is retained when the
runner supplies it.
LLMR supplies provider configuration and execution. LLMRcontent provides codebook-based text annotation and validation. LLMRagent provides agent experiments. LLMRpanel contains panel constructors, instruments, administration, and response summaries.