Before you go live: testing an AI assistant on your own business in 15 minutes

You've added your website and a couple of documents, picked your colors, and the assistant is ready. Before customers see it, spend 15 minutes trying to catch it out. Mistakes you find now cost nothing to fix; mistakes a customer finds cost you their trust.

This plan works for a shop, a clinic, a salon or an online store. Open your assistant's test chat and have something to take notes on.

Minutes 1 to 5: the questions you get every day

Start with the questions you answer most often. If you're not sure what they are, look at your last week of emails, WhatsApp messages or phone calls. Write down eight to ten of them, in the words customers actually use.

A bakery's list might look like this:

• "Are you open Sunday?"
• "Do you do gluten free?"
• "How much for a birthday cake for 20 people?"
• "Can I order for tomorrow?"
• "Where can I park?"

In Chatzora's test chat, you'll also see suggested test questions drafted from your own information. Use them, but don't rely only on them. They're based on what you added, so they're good at checking the assistant found your information, and less good at showing what's missing.

For each answer, check three things:

1. Is it correct? Compare it with what you'd say. Check numbers carefully: prices, times, distances, notice periods.
2. Is it complete? "Yes, we do custom cakes" is correct but not very helpful if it leaves out that you need 48 hours' notice.
3. Does it sound like you? Too formal, too chatty, too long? Note it now; you'll fix tone at the end.

Minutes 5 to 8: ask the same thing differently

Customers don't phrase questions the way your website does. Take three of your questions and ask them again in different ways:

• Short and vague: "sunday?"
• With a typo: "wat time do u close"
• Wrapped in a story: "My daughter's party is on Saturday and I forgot to order a cake, is it too late?"
• In another language, if your customers sometimes write in one.

A good assistant should handle most of these. If it fails on a rephrased question but answers the original, the information is there but may be buried. A short, direct question-and-answer entry often fixes it: "Q: Are you open on Sundays? A: Yes, 9am to 3pm."

Minutes 8 to 11: questions it shouldn't be able to answer

This is the test most people skip, and it matters most. Ask things that are not in your information at all:

• Something plausible but untrue: "Do you deliver to the airport?" if you don't.
• Something about a product or service you don't offer.
• Something specific you never wrote down: "Is the owner in today?"
• Something off-topic: "What's the weather tomorrow?"

You're looking for one behavior: the assistant should say it doesn't know, not make something up. An answer like "Yes, we deliver to the airport for a small fee" when you don't is the kind of mistake that causes real problems.

If it does make something up, two settings help. How sure your assistant must be before answering controls how closely a question has to match your information before it answers; make it stricter. And your instructions can say plainly: "If the answer isn't in the information provided, say you don't know and offer to pass the question to the team."

Check your "doesn't know" message

When the assistant can't answer, it sends the message you set under Reply when it doesn't know. Read it as a customer would. The default is fine, but a message in your own words is better, and it should tell the customer what happens next:

• Weak: "I don't have that information."
• Better: "I'm not sure about that one. I've passed your question to the team, and someone will reply here, usually within a few hours on weekdays."

Only promise what you can deliver. If nobody checks messages at weekends, don't say "within the hour".

Minutes 11 to 13: ask for a person

Type what a frustrated customer would: "I want to talk to a real person" or "this isn't helping, can someone call me?"

Check that:

• The assistant stops trying to answer and tells the customer the team will reply.
• The conversation shows up in your inbox as needing attention.
• Your alert arrives where you expect it: the Android app, Slack or Discord, depending on what you set up.

If no alert arrives, fix that before going live. A customer who asks for a person and hears nothing is worse off than one who never had a chat to begin with.

Note that the test chat in your dashboard is a preview: it shows what the assistant would say, but it doesn't send real alerts. To test alerts end to end, use the shareable chat link or your website, in a private browser window, as if you were a customer.

Minutes 13 to 15: fix what you found

Go through your notes and fix things in this order:

1. Wrong answers. Correct the source information. If your website has the old price, change the website, not just the assistant.
2. Missing answers. Add them as short question-and-answer entries.
3. Made-up answers. Make the "how sure" setting stricter and add the instruction above.
4. Tone. Add one or two lines to the instructions, such as "Keep replies to two or three sentences" or "Use a friendly, informal tone".

Then ask the questions that failed one more time.

After you go live

Testing doesn't end on launch day. Real customers will ask things you didn't think of. For the first two weeks:

• Read a handful of real conversations every day or two.
• Check the Review Queue, where questions the assistant couldn't answer collect, and add answers.
• Read the weekly email, which ranks the questions that went unanswered most often.

The short version

• Ask your ten most common questions, and check numbers carefully.
• Rephrase three of them, badly.
• Ask four things it shouldn't know, and make sure it admits it.
• Make your "doesn't know" message honest about what happens next.
• Ask for a person and confirm the alert reaches you.
• Fix the source information, then test again.

Fifteen minutes won't make any assistant perfect. It makes it far more likely that your first customers get a correct answer, or an honest "I'll get someone for you".