LLM observability is the practice of recording and analyzing how a language model application behaves in production. It connects requests, retrieval, model responses, tool calls, costs, and quality signals so a team can explain a failure and decide what to change.
What should you record?
Start with a request identifier, timestamp, application release, model identifier, prompt version, duration, usage, and outcome. Link retrieval and tool operations to the same request. Keep only the content you are authorized to retain; a trace can be useful with redacted messages and references instead of raw personal data.
| Question | Evidence to collect | What it cannot prove alone |
|---|---|---|
| Why was the answer slow? | Retrieval, model, and tool timings | Whether the answer was correct |
| Why did costs rise? | Model, token usage, retries, and request volume | Whether higher cost improved outcomes |
| Was the answer grounded? | Retrieved sources and claim review | Whether the retrieved sources were true |
| Did the task succeed? | Application outcome and reviewed feedback | Whether every successful task was safe |
Example: a wrong support answer
A customer asks about a refund. The final answer is fluent, but it cites an expired policy. A useful trace identifies the retrieved document version, the prompt that consumed it, and whether a cache returned the old response. Without those links, the team may change the model when the retrieval index needs updating.
Build a small investigation loop
- Instrument a representative user journey and confirm failed requests are captured.
- Define correctness and safety checks using labeled examples from that journey.
- Compare results by model, prompt version, and audience rather than relying on an overall average.
- Assign an owner for alerts and record the mitigation and its measured effect.
Monitoring is evidence, not a guarantee
A classifier flag is a signal for investigation, not proof of an error. Sampled traces can miss failures, and logging can introduce privacy risk. Choose sampling, access, and retention deliberately. For instrument names and attribute support, check the OpenTelemetry conventions and the instrumentation version you actually run.
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.