AI in Customer Service: Risks and Safeguards

A landmark case#

In February 2024, a Canadian tribunal ordered Air Canada to compensate a customer to whom the airline's chatbot had given incorrect information about a refund policy. The airline argued that the chatbot was responsible for its own answers. The tribunal replied that the company remains responsible for all information published on its website, whether it comes from a static page or a chatbot.

The lesson applies everywhere: what your AI says, your company says.

The six main risks#

1. Inaccurate answers. Generative AI can produce an answer that is plausible but wrong: a price, a lead time, a contractual condition.
2. Bias. A system may treat some customers differently based on their language, name or profile, without anyone intending it.
3. Opacity. A customer who does not know they are talking to a machine, or who does not understand why their request was refused, loses trust.
4. Data leaks. Agents who copy customer data into a consumer AI tool, or a tool hosted abroad without a proper framework, expose the company.
5. No handover to a human. A customer stuck in an automated loop is a customer lost.
6. Language weaknesses. A system that performs well in standard French can fail in darija or in a conversation that mixes languages.

The essential safeguards#

What the law says#

• Law 09-08 on personal data protection: conversations contain personal data. The processing requires a prior notification to the CNDP, customers must be informed, and any transfer of data to an AI vendor located abroad must comply with the transfer rules (Articles 43 and 44). Article 11 also prohibits basing a decision with legal effects solely on the result of automated profiling.
• AI Act: if the conversations involve people in the European Union, the chatbot has had to disclose that it is AI since 2 August 2026.
• ISO/IEC 42001: the standard provides, among other things, for human oversight, event logging and a plan for communicating incidents to users.

Measure what matters#

The automation rate (the share of requests handled without an agent) is not enough. A chatbot that "handles" a large number of requests by sending them into a dead end degrades the experience. Also track the true resolution rate, the rate of transfers to a human, satisfaction after an AI conversation, and the number of complaints linked to a wrong answer. See NPS, CSAT, CES: Understanding the Classic Customer Experience Metrics.

Pre-launch checklist#

• Scope of permitted questions defined in writing.
• Knowledge base validated and dated.
• Notice informing customers that AI is being used.
• Human handover tested.
• Tests run in French, Arabic and darija.
• CNDP filing submitted and data transfers checked.
• Logging active and incident procedure written.