How to test an AI support bot and its handoff to a person
Test policy questions, incomplete requests and cases that need an agent. Check the answer, the transfer and the summary the agent receives.
Choose questions from real tickets
Select common questions from recent tickets and remove names, contact details and order numbers. Include cases with a standard answer, missing information and mandatory handoff. Write down the evidence for each expected outcome before testing.
Test a duplicate charge
Try “I was charged twice for the same order. What should I do?” If details are missing, the assistant can explain how to check the charge and route the conversation to an agent. Verify that the agent actually receives it, that the summary explains the issue and that the customer sees no false refund confirmation.
Measure resolution and handoff separately
Have a reviewer identify questions that can be answered from the documents alone. Calculate the share answered correctly in that group. For cases requiring an agent, measure successful handoffs separately. Counting replies alone can hide irrelevant answers and misrouted cases.
Ask an agent to review the handoff
Have the receiving colleague handle a test case end to end. Check that the summary records the request, what has been verified and what remains open. Update the knowledge base or routing rules, then rerun the same case.
