LLM latency is the time a user waits for an AI operation. Distinguish time to first token from time to a complete useful answer, and include retrieval, tool calls, queuing, and retries in the full workflow.
Useful measurements
Time to first token describes when streaming begins. Total response time describes when generation completes. End-to-end task latency includes the surrounding application work. Report distributions and failure rates rather than an average alone.
Example: fast first token, slow task
An assistant starts streaming immediately but waits for two sequential tools before it can answer. Optimizing model token generation may barely change the user’s wait. Trace each stage and identify the actual dependency before adding concurrency.
Choose an objective for the workflow
Interactive chat, batch extraction, and a background review have different expectations. Define a useful completion milestone and a time budget for each. There is no single industry latency benchmark that establishes success for all applications.
Verify optimization tradeoffs
Check complete answers after reducing output length, moving to a different model, or enabling caching. Include timeout handling, cancellations, and slow cohorts in the comparison.
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.