Model drift is a change in an AI system's observed behavior or performance over time. In an LLM application, the cause can include a model update, new prompts, changed retrieval data, tool behavior, or a different mix of user requests.
Distinguish behavior change from traffic change
A higher refusal rate may mean a model became more restrictive, or simply that more users submitted prohibited requests. Compare like-for-like tasks and inspect model, prompt, retrieval, and policy versions before assigning a cause.
Build a useful baseline
- Choose task groups and outcomes that matter to users.
- Record the observation period, sample size, and missing or sampled events.
- Keep a repeatable evaluation set alongside live traffic measurements.
- Annotate releases and upstream changes so comparisons are interpretable.
Example: refund answers change
A support assistant starts rejecting valid returns. Compare the same policy questions against the previous retrieval snapshot. If the model is unchanged but the policy document is different, investigate ingestion and document versioning before replacing the model.
Alert on meaningful changes
Set a minimum sample size and observation window that suit the workflow. Review the size and consequences of the change alongside statistical evidence. Small groups, correlated requests, and repeated comparisons can produce misleading alerts. Confirm suspected drift with reviewed examples.
Record the response
Save the investigation, responsible owner, mitigation, and subsequent evaluation result. A rollback or prompt change is useful only if it improves the intended outcome without creating new failures.
References and further reading
Maintained by DriftRail. AI assisted the October 2026 content and template revision. Examples are illustrative; they are not measured product benchmarks. Suggest a correction.