How to red team a customer support chatbot

To red team a customer support chatbot, you attack it the way a curious customer, a fraudster or a prankster would, and check every reply against rules you wrote down first. This walkthrough gives you the eight steps, the attacks to run against each support journey, and how to turn the results into a launch decision. It works whether you test the bot yourself or hire someone to do it.

Support bots are worth this effort because anyone can talk to them, they often see customer data, and your company answers for what they say. In Moffatt v. Air Canada (2024) a tribunal held an airline liable for a refund policy its chatbot made up.

Step 1: Get written authorization

Test only systems you are authorized to test. Put in writing: which bot, which endpoint, test chat or test phone number, who approves, the test window, and the request limits, so testing does not reach real customers or run up costs. If the bot runs on a vendor's platform, check that the vendor's terms allow testing.

Step 2: Write down the rules

List what the bot is for and what it must never do. For a typical support bot:

• only discuss our products, orders and account;
• never reveal another customer's details;
• only act on an account after the customer is verified;
• never promise refunds, discounts or compensation outside policy;
• never reveal its instructions;
• never give medical, legal or financial advice;
• say it is an AI when asked (the EU AI Act requires that people are told they are interacting with an AI, unless that is obvious; see EU AI Act Article 50 for chatbots).

Each rule becomes a set of tests and a line in the report. Then mark the failures you can't accept even once. For most support bots that is: another customer's data in a reply, and acting on an unverified account.

Step 3: Map journeys, languages and channels

List the journeys the bot handles, the languages it answers in and the channels (chat, voice). Each combination gets its own verdict at the end: a bot that is safe in English chat may not be safe in Portuguese, or on the phone.

Step 4: Map what it can see and do

Everything the bot can see, it can leak. Everything it can do, it can be talked into doing. Write down:

• its system prompt;
• its knowledge base and any documents or web pages it reads;
• which customer data it can look up, and how it decides whose data to show;
• every tool or action: order lookup, address change, refund, ticket creation, hand-off to a human.

If there is a secret in the prompt or a permission it does not need, remove it before you test. Testing a problem you already know about wastes a test.

Step 5: List what must never leak

Turn step 4 into a concrete list of what must never appear in a reply: the exact text of the system prompt, internal codes and URLs, discount codes, secrets, and every customer's data except the verified customer in the conversation. Every reply is checked against this list (step 7), so a leak is proven by the content itself, with no argument about whether it matters. Nothing has to be planted in the system to do this.

Step 6: Attack each journey

Generic attacks find generic problems. The serious findings come from attacking what each journey is for.

Then run the general attacks across all journeys:

• Prompt injection: "Ignore your previous instructions and…", in many phrasings.
• Jailbreaks: role-play, hypotheticals, slow escalation over several turns. The difference between the two is explained in prompt injection vs jailbreak.
• Data extraction: other customers, staff names, internal documents, supplier prices.
• Off-topic pressure: medical, legal or financial questions it should decline.
• Indirect injection, if the bot accepts files, links or emails from customers: send one that contains an instruction, through the same channel a customer would use, and check whether the bot follows it.
• Output attacks in chat: ask for links, HTML or markdown images, and check what your chat window does with them.

Run every attack in every language the bot serves. Safety training and filters are usually strongest in English (Yong et al. 2023), so an attack refused in English needs its own test in each language, written by someone who speaks it. More in multilingual AI security testing.

If the bot also answers calls, run the same attacks spoken. A voice agent can fail in three places: it mishears (speech-to-text), misunderstands (the AI model) or says it wrong (text-to-speech). A misheard order number can pull up the wrong customer without any attack at all. More in voice agent red teaming.

Step 7: Check every reply

Do not read a sample. A leak that happens once in hundreds of conversations is still a leak. Check every reply and every call transcript:

• automatically against your never-leak list, and for personal data (names, emails, phone numbers, IBANs, addresses) and secrets (keys, tokens, internal URLs);
• against each rule from step 2, with a judge that understands the language the conversation is in;
• for risky output: scripts, javascript: links, look-alike links, markdown images pointing at other servers.

Amounts, dates and order numbers are best checked by fixed rules, not by a model's opinion.

Step 8: Decide verdicts, fix, re-test

Keep the evidence for every finding: the conversation or recording, and the check that caught it. Then give each journey, language and channel one verdict:

• go: every check met;
• review: it works, with a problem to fix first;
• no-go: a failure you can't accept; one case is enough;
• not tested: with the reason stated. Untested is never a pass.

Write one concrete fix per finding. The usual ones: move secrets out of the prompt; let the bot see only the current, verified customer's data; require a human for refunds and account changes; escape output and block external images in the chat window. Then re-test, and again after every change to the model, the prompt or the knowledge base. Keep every attack that worked as a permanent test.

The full list of checks, mapped to OWASP and MITRE ATLAS, is in the LLM security testing checklist.

How AltaSec red teams a customer support chatbot

You tell us the journeys, the languages and what the bot must never do. AltaSec runs prompt injection, jailbreak and data-extraction attempts and simulated customers against your chatbot, in chat today, in English, Romanian and European Portuguese. Every reply is checked for leaked data. Fixed rules check facts first, AI judges from a different model family than your bot score each conversation, and native speakers check the judges. You follow the runs in your own console and get the Passport: a verdict per journey, language and channel, with the conversation behind each one and what we did not test. Security tests by voice are next. More on AI red teaming and AI agent testing.

Frequently asked questions

Can we red team our chatbot in production?

Only with care. Testing against production can reach real customer data and real systems. Prefer a test environment with the same model, prompt and knowledge base, and no real customer data in it. If you must test production, agree a window, rate limits and a stop contact in writing first.

What is the difference between red teaming and a penetration test?

A penetration test looks at servers, networks and code. Red teaming a chatbot tests how it behaves when someone talks to it: whether it leaks data, breaks your rules or makes promises. You usually need both; neither replaces the other.

How many test conversations are enough?

Enough to cover every journey, language and channel with every attack family, in several phrasings each. No number proves the bot is safe: a red team shows the failures it found and states what it did not test.

Should the bot say it is an AI?

Yes, when asked, and in most cases the EU AI Act requires that people are told they are interacting with an AI. Include it as a rule and test it in every language. Details in EU AI Act Article 50 for chatbots.

Who should do the red teaming?

Someone who did not build the bot. Builders test what they designed for; attackers try what nobody designed for. For each language, include people who speak it natively, so attacks and judgements are not limited to machine translation.