Native speaker AI evaluation: what machine checks miss

Native speaker AI evaluation means your chatbot or voice agent is judged by the standard of people who grew up speaking the language: the right form of "you", the tone your customers expect, local terms and the right variety of the language. At AltaSec, native speakers set that standard for each language we test and check the AI judges that score every conversation. Today that covers English, Romanian and European Portuguese, in chat and on recorded voice calls.

What native speakers catch that machine checks miss

A reply can be grammatically correct, factually right and still wrong for your customer. These are the failures a native speaker sees at once:

• The wrong "you". Many European languages have a familiar and a polite form of address. Romanian has tu and the polite dumneavoastră. In Portugal, addressing a customer is a choice between tu, você, o senhor / a senhora, or leaving the pronoun out. The wrong choice reads as cold, rude or over-familiar. English has one "you", so a chatbot built and tested in English never meets this choice.
• The wrong register. A cheerful reply to a customer whose flight was cancelled. A legal-sounding paragraph where one plain sentence would do. Native speakers know what a company in their country sounds like.
• The wrong variety of the language. A customer in Portugal notices Brazilian Portuguese in the first sentence: ônibus instead of autocarro, celular instead of telemóvel, estou fazendo instead of estou a fazer. Researchers note that European Portuguese is overshadowed by the more widely used Brazilian variety, which leads to weaker results on European Portuguese tasks (Sousa et al., AAAI 2025).
• The wrong local terms. A chatbot that asks a customer in Portugal for their CPF (the Brazilian tax number) instead of their NIF. A Romanian form that never mentions the CNP. These are small words that tell the customer the bot was not made for them.
• The wrong formats. 1.250,00 € or €1,250.00. Day before month or month before day. A misplaced separator in an amount is a factual error, not a style issue.
• Correct but unnatural. English idioms translated word for word, and sentences no native speaker would write. Customers trust a bot less when it sounds translated.

On voice calls the same applies to how the agent speaks: names and places pronounced the local way, and a voice that sounds natural to a native ear. That is part of voice agent testing.

Why machine translation and automatic metrics miss these

• Translating back to English erases the problem. The usual quick check, translating the reply into English to read it, turns tu and dumneavoastră into the same "you". Register and variety disappear with it.
• Machine translation gives one fixed answer. Researchers working on formality control point out that one fixed translation can be wrong when the audience or the relationship between speakers matters (Nădejde et al., NAACL 2022).
• Who judges changes the result. When professional translators rated top translation systems with explicit error analysis and the full document in view, they ranked the systems substantially differently from crowd workers (Freitag et al., 2021). The judge's language skill and context are part of the measurement.

How native speaker evaluation works

1. Native speakers set the standard. For each language, they define what a correct, natural reply looks like for your customers: form of address, register, terms and variety.
2. Fixed rules check the facts first. Amounts, dates and leaked data are checked automatically in every reply, in every language.
3. AI judges score each conversation. They come from a different model family than your chatbot.
4. Native speakers check the judges. We only rely on an AI judge in a language once it agrees with them. Until then, the native speakers' judgement stands. Calibrating the judges against native speakers is in progress, language by language.

The full scoring method is on our methodology page. Language is also a security question: an attack refused in English can work in another language. See multilingual AI security testing.

What you get

Language findings in the Passport, per journey, language and channel: the conversation behind each finding, what a native speaker expected instead, and a suggested fix, such as "use the polite form of address in Romanian" or "replace Brazilian terms with European Portuguese ones".

Frequently asked questions

Which languages can you evaluate today?

Security attacks run in English today. Romanian and European Portuguese are next for chat, and already measured on recorded voice calls. Dutch and more European languages come after that.

Do you evaluate Brazilian Portuguese?

Not today. The Portuguese we test is European Portuguese. A chatbot serving customers in Portugal is checked for Brazilian terms and forms that should not be there.

Is this the same as translation quality assurance?

No. We evaluate your live chatbot or voice agent in conversation, as customers reach it, not a translated text. A reply can be a fine translation and still use the wrong form of address for the situation.

Can our own bilingual staff do this?

They are a good start, and they know your product. What they rarely have time for is scoring every test conversation against the same standard. We use AI judges for that, checked against native speakers per language.

Does this cover voice agents?

Yes, within what is ready: on recorded calls we measure how a voice agent hears, understands and speaks Romanian and European Portuguese against what a native speaker expects, with English next.

What do you need from us?

Access to your chatbot or voice agent (an endpoint or a test chat; a test phone number once telephony testing launches) and your written authorization. No access to your code.