Report small-sample LLM failure rates honestly

Report a small-sample failure result as a count with its denominator, selection method, and uncertainty. Two failures in ten reviewed answers are observations from that sample, not a stable failure rate for all users or future releases.

Start with the unit of analysis

Decide whether one observation means a claim, answer, conversation, or completed task. Ten claims from the same answer are not necessarily ten independent trials. Record excluded, missing, and unreviewed records so the denominator does not silently shrink.

Illustrate uncertainty without overstating it

The runnable example computes an illustrative Wilson interval with z = 1.96 for two failures among ten independent Bernoulli observations. It verifies that the observed fraction lies inside the interval. The method and assumptions are named because an interval does not repair a biased or correlated sample.

Publish the limitations with the number

1. State the observation period, task groups, and how examples were selected.
2. Show counts before rounding any rate.
3. Use an uncertainty method appropriate to the sample and decision, with specialist statistical review when necessary.
4. Keep high-impact failures visible as cases; an overall interval is not an acceptable-risk determination.

Do not overread a zero

No observed failures can be useful evidence, but it does not prove zero risk. A test containing only ordinary questions says little about rare policy exceptions. Expand coverage according to the failure consequences and the tasks you have not observed, rather than publishing a universal safety claim from a small clean sample.

Run the worked example

Save the synthetic Node.js example and run node report-small-sample-llm-failures.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.

// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';

const wilson = (failures, n, z = 1.96) => {
  assert.ok(n > 0 && failures >= 0 && failures <= n);
  const p = failures / n;
  const denominator = 1 + z * z / n;
  const center = (p + z * z / (2 * n)) / denominator;
  const half = z * Math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator;
  return [center - half, center + half];
};
const bounds = wilson(2, 10);
assert.ok(bounds[0] < 0.2 && bounds[1] > 0.2);
assert.ok(wilson(0, 10)[1] > 0);
console.log(JSON.stringify({ failures: 2, reviewed: 10, fraction: 0.2, illustrativeWilsonBounds: bounds }));

References and further reading

• NIST statistical handbook: confidence intervals for proportions
• DriftRail integration documentation

Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.