Set latency objectives for the user-visible completion milestone of each workflow. An interactive chat answer and an asynchronous extraction job can use different budgets without either being judged by a universal “good LLM latency” number.
Name the milestone and the clock
For chat, distinguish first visible output from a complete useful answer. For extraction, distinguish job acceptance from validated output ready for downstream use. A fast acknowledgment followed by a stalled job is not a fast extraction. Include queuing, tools, retries, and required review in the milestone the user actually needs.
Create objectives that can be measured
- Define each journey, its outcome categories, and the chosen completion event.
- Collect latency distributions for representative successful requests and report failed or incomplete counts separately.
- Choose an objective based on user needs and operating constraints, then state the observation window and percentile method.
- Revisit the budget when workflow steps or user expectations change, not merely when a model becomes faster.
Read the two workflow samples
The runnable example uses a documented nearest-rank calculation on tiny synthetic arrays and compares each workflow with an illustrative budget. The batch values are larger but evaluated against a different milestone and budget. None of these numbers is an industry benchmark or a recommended production target.
Keep the rest of the service visible
Small samples and successful-only timing can hide slow failures and abandonment. Separate waiting for a human from model execution when that distinction helps operations, while still reporting the complete user journey. Verify clocks, cancellations, and long-running job status before relying on a latency dashboard.
Run the worked example
Save the synthetic Node.js example and run node set-chat-and-batch-latency-budgets.mjs with Node.js 22 or newer. The assertions check the illustrated records and calculations; they do not test a deployed provider, SDK, or access control. No credentials or network connection are needed.
// Synthetic teaching fixture. No credentials, network calls, or production verification.
import assert from 'node:assert/strict';
const nearestRank = (values, p) => {
assert.ok(values.length > 0 && p > 0 && p <= 1);
const sorted = [...values].sort((a, b) => a - b);
return sorted[Math.ceil(p * sorted.length) - 1];
};
const chat = { values: [300, 600, 800, 1200], illustrativeBudget: 1000 };
const batch = { values: [4000, 6000, 9000, 11000], illustrativeBudget: 10000 };
assert.equal(nearestRank(chat.values, 0.75), 800);
assert.equal(nearestRank(batch.values, 0.75), 9000);
assert.ok(nearestRank(chat.values, 0.75) <= chat.illustrativeBudget);
assert.ok(nearestRank(batch.values, 0.75) <= batch.illustrativeBudget);
console.log('Different completion journeys use distinct illustrative budgets; no production target is implied.');
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.