How to monitor AI after launch: evaluation, hallucination detection, and audit trails

Three weeks after launch, the RAG customer service system started giving wrong answers. Not a system failure, degradation. The knowledge base got updated with a new return policy the previous month, but embeddings for the old policy still sat in the index, never cleaned out. The model confidently cited rules that had become obsolete weeks earlier. Customers didn't notice, because the handful of test questions from launch day still answered correctly.

This is the central problem: most teams' evaluation stops at launch.

When launches end, monitoring begins

Post-launch LLM monitoring means continuously measuring response quality, hallucination rate, and auditability after your system enters production, then triggering alerts and rollbacks when metrics degrade. This is an operational function, not a one-time gate. It differs from pre-launch evaluation. Before launch, you prove the model works now. After launch, you prove it stays working.

The model itself rarely changes. Everything surrounding it does: knowledge bases updated, users asking harder questions, upstream APIs versioned, an engineer tweaking a line of the prompt. None of this throws an error. It just makes answers gradually worse. Without monitoring, the first signal is customer complaints. By then, you've shipped hundreds of wrong answers.

LLM evaluation and monitoring: build a test set that fails

The core of post-launch LLM evaluation isn't scoring. It's the test set. Build a golden set: 50 to 200 questions covering high-frequency normal questions, known edge cases, and real production mistakes. Mark the right answer or acceptance criteria for each.

Run this set automatically once a day and check the pass rate. Raw scores alone mean nothing. Look for the trend: if today's 92% and last week's 96%, that's a warning signal even if customers haven't complained. Scoring requires manual work, but use LLM-as-judge as a first pass. Let another model compare answers against your criteria and flag suspicious failures for human review. This narrows daily review from hundreds of questions to roughly a dozen.

One thing often gets skipped: keep the test set alive. Feed every real customer complaint and manual fix back as a new test case. Otherwise your evaluation stays locked to a three-month-old model of the world.

Hallucination detection: don't wait for customers to catch it

Hallucination detection must happen before the response goes out, not afterward. In RAG scenarios, the most practical approach is retrieval consistency checking. For every key fact the model states, check whether it appears in the original text snippets you retrieved. If not, mark it as low confidence, either downgrading to 'I'm not certain; please contact a specialist' or escalating to human review.

Also run spot checks on hallucinations offline, tracking three metrics: citation coverage (the percentage of sentences backed by retrieved text), refusal rate (whether it says 'I don't know' when appropriate), and contradiction rate (whether answers conflict with the knowledge base). Refusal rate often gets missed. A system that never refuses is unreliable.

Audit trails: every response must be replayable

When something goes wrong, the first question is always 'why did it answer that way?' If you can't answer it, you're left guessing and restarting. Log a structured record for every generation: input, retrieved chunks, prompt version, model version, output, confidence score, timestamp. Financial and healthcare customers need another layer: de-identification, retention periods, access controls to align with audit requirements.

This isn't just a compliance requirement. It's a debugging tool. In the return-policy case above, audit trails identified 'old embeddings not cleaned' in 20 minutes instead of two days spent trying to reproduce the issue.

This three-tier monitoring checklist covers what most systems need:

Wire monitoring into operations, not just a dashboard

The most common failure: everything above becomes a quarterly report that sits unread. Monitoring needs an outlet. Alerts go to the on-call channel. Degradation triggers prompt or index rollback. Customer complaints become new test cases with one click. It's a closed loop, not a dashboard.

Monitoring belongs in the launch scope, not as an add-on. Engineers set it up on-site, connecting the test set, hallucination thresholds, and audit logs to the customer's existing on-call stack. Handoff includes training on alert interpretation. A polished demo doesn't mean the work is done. Shipping a system that still answers correctly six months later, and where every response is traceable when things break, that's delivery that matters.