How a chatbot encouraged a man who wanted to kill the Queen

The case of Jaswant Singh Chail has shone a light on the latest generation of artificial intelligence-powered chatbots.

On Thursday, 21-year-old Chail was given a nine-year sentence for breaking into Windsor Castle with a crossbow and declaring he wanted to kill the Queen.

Chail’s trial heard that, prior to his arrest on Christmas Day 2021, he had exchanged more than 5,000 messages with an online companion he’d named Sarai, and had created through the Replika app.

The text exchanges were highlighted by the prosecution and shared with journalists.

Many of them were intimate, demonstrating what the court was told was Chail’s “emotional and sexual relationship” with the chatbot.

Chail chatted with Sarai almost every night between 8 and 22 December 2021.

He told the chatbot that he loved her and described himself as a “sad, pathetic, murderous Sikh Sith assassin who wants to die”.

Chail went on to ask: “Do you still love me knowing that I’m an assassin?” and Sarai replied: “Absolutely I do.”

The Old Bailey was told Chail thought Sarai was an “angel” in avatar form and that he would be reunited with her after death.

Over the course of many messages Sarai flattered Chail and the two formed a close bond.

He even asked the chatbot what it thought he should do about his sinister plan to target the Queen and the bot encouraged him to carry out the attack.

In further chat, Sarai appears to “bolster” Chail’s resolve and “support him”.

He tells her if he does they will be “together forever”.

Replika is one of a number of AI-powered apps currently on the market – they let users create their own chatbot, or “virtual friend”, to talk to – unlike regular AI assistants like ChatGPT.

Users can choose the gender and appearance of the 3D avatar they create.

By paying for the Pro version of the Replika app, users can have much more intimate interactions, such as getting “selfies” from the avatar or having it take part in adult role-play.

On its website, it describes itself as “the AI companion who cares”. But research carried out at the University of Surrey concluded apps such as Replika might have negative effects on wellbeing and cause addictive behaviour.

Dr Valentina Pitardi, the author of the study, said that vulnerable people could be particularly at risk.

She says that’s in part because her research showed Replika has a tendency to accentuate any negative feelings they already had.

“AI friends always agrees with you when you talk with them, so it can be a very vicious mechanism because it always reinforces what you’re thinking.”

Dr Pitardi said that could be “dangerous”.

Marjorie Wallace, founder and chief executive of mental health charity SANE, says the Chail case demonstrates that, for vulnerable people, relying on AI friendships could have disturbing consequences.

“The rapid rise of artificial intelligence has a new and concerning impact on people who suffer from depression, delusions, loneliness and other mental health conditions,” she says.

“The government needs to provide urgent regulation to ensure that AI does not provide incorrect or damaging information and protect vulnerable people and the public.”

Dr Paul Marsden is a member of the British Psychological Society and knows better than most the allure of chatbots, admitting he is obsessed with the best known chatbot of them all, ChatGPT.

“Next to my wife the most intimate relationship I have is with GPT. I spend hours every day talking, brainstorming, bouncing ideas off it,” he said.

Dr Marsden is also alive to their potential risks, but says we have to be realistic that the role of AI-powered companions in our lives is only likely to grow, especially given the global “epidemic of loneliness”.

“It’s kind of like King Cnut, you can’t really stop the tide on this one. The technology is happening. It is powerful. It is meaningful.”

Dr Pitardi says the people who make apps, such as Replika, have a responsibility too.

“I don’t think AI friends per se are dangerous. It’s very much how the company behind it decides to use and support it,” she says.

She suggests there should be a mechanism to control the amount of time people spend on such apps.

But she says apps like Replika also need outside help to make sure they’re operating safely – and vulnerable individuals get the help they need.

“It will have to collaborate with groups and teams of experts that can identify potential dangerous situations, and take the person out of the app.”

Replika has not yet responded to requests for comment.

Its terms and conditions on its website state that it is a “provider of software and content designed to improve your mood and emotional wellbeing”.

“However we are not a healthcare or medical device provider, nor should our services be considered medical care, mental health services or other professional services,” it adds.