How to Test Your AI Chatbot Before Launch: A Pre-Launch QA Checklist

The moment a chatbot finishes crawling your site, it's tempting to grab the embed code and ship it. The setup really is that fast. But a chatbot that's trained doesn't automatically mean a chatbot that's ready to face your actual customers, and the gap between those two states is usually invisible until a visitor finds it for you in a public, slightly embarrassing way.

This checklist is the twenty minutes worth spending between a successful crawl and a public launch. None of it requires technical skill. It just requires asking the bot the questions a real visitor would ask, on purpose, before they do, and knowing what to look at when something comes back wrong.

Why a working crawl isn't the same as a ready chatbot

A crawl finishing without errors tells you the tool could read your pages. It doesn't tell you whether your pages actually contain the answers customers are looking for, whether the bot picked the right page when two pages said similar things, or whether your pricing structure is confusing enough that even a well-trained bot states it incorrectly. Those are content problems, not technical ones, and the only way to find them is to talk to the bot the way a stranger would, before a stranger actually does.

Step one: read your own site like a stranger would

Before testing the bot, skim your own pricing, hours, service area, and policy pages as if you'd never seen them. Note anything vague, outdated, or split across two pages that contradict each other slightly, like an old promotion still listed on one page while a newer price lives on another. A chatbot trained on contradictory source pages will confidently repeat the contradiction back to a customer, and it has no way to know which version is current unless you fix it at the source first.

Step two: ask it the questions you already get by phone and email

You already have the best test questions available: the ones your team answers manually every week. Pull five to ten real questions from recent emails, call logs, or whatever channel customers actually use, and type them into the bot verbatim, typos and all. This catches two things a generic test never will: whether the phrasing customers actually use matches anything on your site, and whether your most common question has a confident, correct answer or quietly triggers a hedge.

Step three: try to break it on purpose

Ask it something your site genuinely doesn't cover, and confirm it says so honestly instead of guessing. Ask it a question with a trick premise, like referencing a service or price you don't actually offer, and see whether it corrects the assumption or plays along. Ask the same question two different ways and compare the answers for consistency. None of this needs to be adversarial in spirit. You're just confirming the bot admits uncertainty instead of inventing an answer that sounds plausible, since a confident wrong answer does more damage to a customer relationship than an honest admission that it isn't sure.

Step four: check the handoff and the unanswered-questions list

Find out where a frustrated visitor actually lands if the bot can't help: does it offer a real contact method, a booking link, or a human handoff, or does it just repeat the same hedge in a loop. Also check whether the tool logs questions it couldn't answer. That list is where you'll find your real content gaps after launch, but it's worth glancing at even before launch, since a handful of test questions from step two will usually already be sitting in it, which confirms the logging itself actually works.

Step five: test it on an actual phone

A surprising share of chatbot traffic is mobile, and a widget that looks fine on a laptop can cover a call-to-action button, overlap a cookie banner, or need three taps to even open on a small screen. Open your own site on your phone, not a browser's mobile preview mode, and walk through opening the widget, typing a question, and reading the answer the way a visitor actually would. A test that only ever happens on a desktop monitor misses the device most of your traffic is probably arriving on.

What to do if it fails any of these

A bad answer during testing is cheap. The same bad answer live, in front of a paying customer, costs you a sale or a support ticket. If a test question comes back wrong, fix the source page first, not just the symptom, since the same gap will resurface in a different phrasing eventually if the underlying page never changes. Re-run the specific question once the page updates and the knowledge base refreshes, confirm the fix actually landed, and only then treat the chatbot as ready for real traffic instead of a work in progress wearing a live badge.