How to evaluate an AI customer support system before launch

A successful demo is not a launch test

A demo usually contains clear questions and prepared knowledge. Production contains incomplete messages, conflicting policies, account-specific context, angry customers and integrations that fail at inconvenient times.

The evaluation set must represent that variation. Include common questions, rare high-risk requests, missing information and examples the system should refuse or escalate.

Define what a correct answer means

Correctness may require more than matching a reference sentence. The answer may need a current policy version, the right product context, a source citation, an approved tone and a specific escalation route.

Score these dimensions separately. A fluent answer with the wrong policy should fail even if it sounds helpful.

• Factual support from an approved source
• Correct policy and version
• No prohibited promise or action
• Appropriate request for missing context
• Correct escalation when confidence or authority is insufficient

Test retrieval and generation separately

If the wrong source is retrieved, changing the response prompt will not solve the underlying problem. Record which documents were selected and whether they contain enough information to answer.

Then evaluate whether the generated response uses those sources correctly. This separation makes failures actionable for content owners and engineers.

Design escalation as a product feature

Escalation should carry the original request, selected sources, attempted classification and the reason a person is needed. Sending a generic alert forces the support agent to repeat the same research.

Test queue ownership, response time and what happens when the assigned person is unavailable. A safe refusal without a working handoff is still a broken customer experience.

Include privacy and authorization scenarios

Verify that users cannot retrieve another customer's information, expose hidden instructions or persuade the system to execute an unauthorized action. Minimize the customer fields included in prompts and log access to sensitive tools.

Define retention for conversations, source documents, evaluations and provider logs before using production data.

Release in stages

Begin with internal search or agent-assist, where a person reviews every response. Compare resolution time, correction rate, escalation quality and unresolved topics.

Expand only the scenarios that meet an agreed threshold. Keep a feature switch, manual fallback and a routine for rerunning evaluations when knowledge, models or prompts change.