Timeout-aware service and AI latency benchmarks

The decision: does this backend meet a deadline?

A request that times out after two seconds does not have a two-second completion latency. Its completion time is unknown and exceeds two seconds. Dropping it describes only the requests that completed. Treating it as a completion at two seconds also distorts the distribution.

For service or AI inference benchmarks, retain one row per attempt with elapsed time, completion status, configured deadline, backend, workload, and run ID. Analyze comparable workload/load groups separately. Distinguish time to first token from end-to-end completion time; do not combine them into one outcome.

An administrative timeout can be a right-censored completion time when the underlying completion process and censoring assumptions are defensible. HTTP errors, refusals, invalid outputs, and cancelled retries are separate outcomes; they are not automatically censored successful completions. Report those counts and quality separately. Correlated requests in a shared load test also require run-level replication or an appropriate dependence model for uncertainty.

A reproducible demonstration

The following uses synthetic exponential times, not measured provider data. The known distribution lets us verify the inference against a closed-form MLE. It is an illustration, not a claim that service latency is exponential.

library(likelihood.contr)
library(likelihood.model)
set.seed(20261028)
deadline <- 2
latent <- rexp(1000, rate = 1)
requests <- data.frame(seconds = pmin(latent, deadline),
  status = ifelse(latent <= deadline, "exact", "right"))
model <- likelihood_contr("status",
  exact = contr_name("exp", "exact", "seconds"),
  right = contr_name("exp", "right", "seconds"))
rate <- sum(requests$status == "exact") / sum(requests$seconds)
fitted <- fit(model)(requests, c(rate = rate))
stopifnot(fitted$converged, abs(unname(coef(fitted)) - rate) < 1e-5)
data.frame(
  completed_only_p95 = unname(quantile(requests$seconds[requests$status == "exact"], .95)),
  model_p95 = qexp(.95, rate = coef(fitted)),
  simulation_truth_p95 = qexp(.95, rate = 1))
#>   completed_only_p95 model_p95 simulation_truth_p95
#> 1           1.716187  3.113602             2.995732

More than 5% of requests time out in this example. The observed data therefore do not determine an empirical p95 beyond the deadline. The fitted p95 is a model-dependent extrapolation. Longer observation windows, a suitable nonparametric survival analysis, and sensitivity to alternative tail models are more useful than presenting that extrapolation as measured fact.

For an SLO within the common observation window, count successful completions by that time. This directly answers the operational question under this design:

slo <- 1.5
met_slo <- requests$status == "exact" & requests$seconds <= slo
c(slo_seconds = slo, observed_fraction_meeting_slo = mean(met_slo),
  timeout_fraction = mean(requests$status == "right"))
#>                   slo_seconds observed_fraction_meeting_slo 
#>                         1.500                         0.767 
#>              timeout_fraction 
#>                         0.145

Compare backends at a common SLO and matched workload/load. A faster completed-only p95 can coexist with more timeouts. With unequal deadlines, a request censored before the SLO has an unknown SLO outcome; do not count it as either success or failure without reporting bounds or making assumptions.

When only histogram bins are retained

Telemetry often stores counts rather than raw times. Each disjoint bin (lo, hi] contributes count * log(F(hi) - F(lo)). An open final bin (deadline, Inf] contributes its count times the log survival probability. contr_freq() applies this principle to any additive contribution, retaining analytical derivatives when supplied. It allocates no expanded request table.

bins <- data.frame(lo = c(0, .5, 1, 2), hi = c(.5, 1, 2, Inf),
                   count = c(390, 240, 230, 140), status = "interval")
grouped <- likelihood_contr("status",
  interval = contr_freq(contr_name("exp", "interval", "lo", "hi"), "count"))
ll <- loglik(grouped)
# Explicit positive-domain parameterization; no automatic domain guessing.
solution <- optimize(function(log_rate) ll(bins, c(rate = exp(log_rate))),
                     c(-5, 5), maximum = TRUE)
exp(solution$maximum)
#> [1] 0.987393

Convert cumulative histogram counts to per-bin increments first. Keep labels, units, deadlines and run identities separate. Include timeout counts: a histogram of completed requests alone cannot recover missing censored attempts. Frequencies must be finite nonnegative integers; they are not survey weights.

The resulting likelihood differs from a multinomial likelihood by a constant that does not affect parameter estimates or likelihood ratios on the same bins. The generic fitting layer counts rows, not expanded observations: do not use its default BIC or row bootstrap for frequency data. Use the sum of frequencies for the sample size and resample counts using the actual sampling design. This example optimizes the likelihood directly for that reason.

Where this fits

Use survival for standard nonparametric and regression survival analysis, and fitdistrplus for standard parametric distribution fitting with censoring. likelihood.contr is useful when you need to compose exact measurements, censored observations, histogram frequencies, and custom observation mechanisms into one explicit likelihood. The choice of probability model remains yours.

Run demo("timeout-benchmark", package = "likelihood.contr") for the executable synthetic demonstration. This workflow is a starting point for a benchmark report, not a causal provider ranking or a guarantee of production SLOs.