Monitor a production LLM by linking each user request to its model calls, retrieval, tools, and final outcome. Start with the failures your team must investigate, then collect the smallest set of events that explains them.
1. Instrument one complete journey
- Give the application request a stable identifier and propagate it through model and tool calls.
- Record model and prompt versions, durations, usage, status, and sanitized failure details.
- Exercise a timeout, rejected input, failed tool call, and successful request. Confirm each is distinguishable.
2. Separate system health from answer quality
Track request failures, latency distributions, usage, and retry volume for system health. Evaluate task completion, grounded claims, policy decisions, and reviewed user feedback for quality. A fast HTTP 200 response can still contain the wrong answer.
3. Set alerts from your service objectives
Use a baseline from representative traffic. Choose an observation window, minimum sample size, acceptable error budget, and responsible owner. Do not copy a universal hallucination or latency threshold from another industry. Test that an alert reaches the person who can act.
4. Check retention and access
A monitoring record may itself contain personal or confidential information. Test redaction before transmission, limit access by project, and verify deletion and retention behavior. Keep identifiers that support an investigation without retaining unnecessary prompt content.
5. Verify the change
Replay a reviewed sample after a release and compare the same task groups. Record the denominator behind each failure rate, missing events, classifier uncertainty, and any change in sampling. Follow the integration documentation for supported SDK methods instead of copying an unverified snippet.
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.