Report event coverage as separate counts for expected requests, deliberately sampled requests, successfully observed requests, and reviewed answers. A failure fraction among reviewed answers cannot automatically describe every request your application served.
Sampling and loss answer different questions
In the fixture, the application serves 100 requests, deliberately retains 20, receives 18 retained records, and reviews 10 answers. Two records are missing from the intended sample. Four reviewed failures describe 4 of 10 reviewed answers, not 4 of 100 production requests.
| Count | Meaning |
|---|---|
| 100 expected | Requests known to the application counter |
| 20 selected | Requests the policy intended to retain |
| 18 observed | Selected records actually available |
| 10 reviewed | Answers with the required review label |
Reconcile before publishing a rate
- Choose a stable request-level denominator and handle retries separately.
- Compare selected IDs with observed IDs to distinguish intentional exclusion from delivery loss.
- Record review selection and unlabeled records instead of silently treating missing labels as passes.
- State the sampling policy, period, and task mix next to any quality summary.
When extrapolation is unsafe
A sample enriched for errors or expensive requests is useful for diagnosis but is not a simple random sample. Nonuniform selection, clustered requests, and missing records can bias estimates. Document an appropriate estimation method before extrapolating; the supplied arithmetic deliberately makes no population-quality claim.
Run the worked example
Save the synthetic Node.js example and run node count-missing-and-sampled-ai-events.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.
// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';
const counts = { expected: 100, selected: 20, observed: 18, reviewed: 10, failures: 4 };
assert.ok(counts.reviewed <= counts.observed && counts.observed <= counts.selected);
const missingSelected = counts.selected - counts.observed;
const reviewedFailureFraction = counts.failures / counts.reviewed;
assert.equal(missingSelected, 2);
assert.equal(reviewedFailureFraction, 0.4);
assert.notEqual(reviewedFailureFraction, counts.failures / counts.expected);
console.log(JSON.stringify({ missingSelected, reviewedFailureFraction, populationFailureRate: 'not estimated' }));
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.