AI receptionists seem like a dangerous idea after a patient revealed it failed to understand her request

AI receptionists are supposed to make calling the doctor easier. But a growing number of patients are finding that the opposite can be true – particularly when a machine struggles with the way they speak, understand instructions, or communicate their needs.

A recent case in South Yorkshire brings that problem into sharp focus. Judith Butterfield, a 71-year-old woman recovering from a stroke, told the BBC that an AI receptionist used by her GP surgery repeatedly failed to understand her speech when she tried to book an appointment. Her stroke had left her speech fragmented and affected movement on the right side of her body. After several unsuccessful attempts, she eventually registered with another practice.

The surgery has since decommissioned the system. But the incident raises a much bigger question: how much of healthcare can safely be handed to an AI that expects patients to communicate in a way it understands?

A machine can handle the queue, but what happens when a patient does not fit the script?

The AI receptionist, known as EMMA and developed by QuantumLoopAI, was introduced to help GP practices handle telephone demand and reduce waiting times. Healthwatch Rotherham has since reported complaints from patients who struggled to communicate with the system, including people with regional accents and speech impairments.

Butterfield’s experience shows why a minor technical weakness can become something more serious in healthcare. The automated service reportedly asked her not to use speakerphone, yet she said she needed to hold the phone in one hand because her other hand had been affected by her stroke. When the system failed to understand her name, there was no human instinct to recognize that something was wrong and simply ask her to repeat it differently.

The issue is broader than one receptionist system. Healthwatch England has collected similar accounts from patients encountering AI when booking appointments, requesting medication and communicating their needs. Some people with speech or learning difficulties said automated systems could not understand them, while others reported problems getting their requests correctly handled.

Healthcare cannot afford an AI that works only for easy conversations

AI does have an obvious appeal for overstretched GP practices. A system that can answer calls immediately could reduce queues and free reception staff for other work. QuantumLoopAI says EMMA supports multiple languages and accents and allows callers to request a human at any time. The company also said practices can configure priority routing so certain patients go directly to reception staff.

But that safeguard only works if patients know it exists and can successfully navigate the system long enough to use it. Healthwatch has warned that people with communication needs, disabilities and lower digital confidence can be particularly vulnerable to being excluded by poorly designed healthcare technology. The problem is not that AI occasionally gets a word wrong. It is that the person on the other end may be trying to access healthcare in the first place.

As GP practices continue experimenting with automated reception and triage, the lesson from South Yorkshire is difficult to ignore: an AI receptionist should make it easier to reach a human, not become another obstacle standing between a patient and care.