AI Model Drift Explained

Your AI worked perfectly at launch. Three months later, users are complaining about quality. Nothing in your code changed. This is model drift, and it affects every production AI system eventually.

Types of Drift

Data Drift — The distribution of inputs your model receives in production shifts away from the distribution it was trained or evaluated on. Users start asking different questions, the product expands to new markets, or seasonal patterns change the query mix. The model has not changed, but the world it operates in has.

Concept Drift — The relationship between inputs and correct outputs changes over time. What constituted a good answer six months ago may not be a good answer today — regulations change, products update, company policies evolve. This is particularly common in customer support and knowledge base applications.

Provider-Side Model Updates — If you use API-based models, the provider may update the underlying model without notice. OpenAI and Anthropic regularly update their models, sometimes changing behaviour in subtle ways. A prompt that worked perfectly with one model version may degrade with the next.

Detecting Drift

Baseline Evaluation — Establish quality benchmarks at deployment time using a representative evaluation dataset. Run these benchmarks regularly (daily or weekly) and track scores over time. Any consistent downward trend signals drift.

Input Distribution Monitoring — Track statistical properties of your input data (query length, topic distribution, language mix) and alert when the distribution shifts significantly from baseline.

User Feedback Trends — Track user satisfaction signals over time. A gradual increase in negative feedback or regeneration requests is often the first human signal of drift.

Output Distribution Monitoring — Track model output characteristics (response length, confidence scores, format compliance) and alert on distributional shifts.

Responding to Drift

When drift is detected, the response depends on the cause. Data drift requires updating your evaluation dataset and potentially retraining or adjusting prompts. Concept drift requires updating your ground truth and knowledge base. Provider-side changes require re-evaluating your prompts against the new model version. In all cases, the key is detecting drift early — before it compounds into a visible quality crisis.