What to Monitor in Production LLM Systems

Most teams monitor their AI systems the same way they monitor web applications: uptime, response codes, and latency. That is necessary but nowhere near sufficient. An LLM can return 200 OK while hallucinating, leaking data, or costing 10x what it should. Production AI monitoring requires a fundamentally different approach.

Quality Metrics

Automated Evaluation Scores — Run a subset of production queries through an evaluation pipeline that scores output quality against ground truth or rubric-based criteria. Track the distribution over time and alert on degradation.

User Feedback Signals — Thumbs up/down, corrections, regeneration requests, and abandonment rates. These are lagging indicators but they catch quality issues that automated evaluation misses.

Output Distribution Monitoring — Track the statistical distribution of output characteristics: length, confidence scores, token count, and format compliance. Sudden shifts indicate something has changed — model update, data pipeline issue, or prompt template error.

Operational Metrics

Latency Percentiles — Track P50, P95, and P99 separately. A system with 500ms P50 but 8s P99 has a problem that averages hide. Track time-to-first-token separately from total generation time for streaming applications.

Cost Per Request — Calculate and track the actual cost of each request including input tokens, output tokens, and any supporting calls (embeddings, retrieval). Alert on per-request cost anomalies.

Error and Retry Rates — Model errors, timeouts, rate limit hits, and fallback activations. Track at the pipeline level, not just the model call level.

Security Signals

Prompt Injection Attempts — Log and classify inputs that match injection patterns. Track frequency and sophistication over time.

PII in Outputs — Automated scanning of model outputs for personally identifiable information that should not be there.

Unusual Access Patterns — Spike in requests from a single user, unusual query patterns, or attempts to probe system prompt boundaries.

The Monitoring Stack

We recommend a two-layer approach: an AI-specific observability tool (Langfuse, Langsmith, or custom) for quality and semantic monitoring, integrated with your existing infrastructure monitoring (Datadog, Grafana, CloudWatch) for operational metrics.