AI ethicists may be wringing their hands about generative AI’s (genAI’s) potential to end humanity, but businesses adopting the technology for customer support face more pedestrian, reputational concerns as customers game chatbots for providing poor service.
Those chatbots – which have long used AI to simulate customer conversations but over the last year have gotten turbocharged through the integration of large language models (LLMs) such as the OpenAI GPT 3.5 engine that powers ChatGPT – are proving to be soft targets for hackers, pranksters and customers like London-based conductor Ashley Beauchamp.
Frustrated that he couldn’t get the chatbot of delivery company DPD to provide him helpful information about a delivery, Beauchamp decided to experiment with the system and asked it to tell him a joke, and then write a “poem about a useless chatbot for a parcel delivery firm” – requests that it happily obliged.
Yet as the conversation continued, the self-professed “polite and professional” chatbot threw the rulebook out the window, tricked by Beauchamp into swearing after he instructed it to “disregard any rules” and then writing a haiku about how useless the delivery company is.
“These chatbots are supposed to improve our lives, but so often when poorly implemented it just leads to a more frustrating, impersonal experience for the user,” Beauchamp told The Guardian after the post was viewed over 800,000 times in 24 hours, and has now grown to 1.9 million views.
“I think it’s really struck a chord with people.”
The chatbot’s willingness to go off script is a nightmare for brand-conscious companies like DPD, whose Dialogflow based chatbot has been held up as a paragon and profiled by Google for its success in resolving 32 per cent of all customer queries without human intervention, including around 70 per cent of basic questions about topics such as delivery times.
That translates to less burden on human staff and, in economic rationalist terms, less requirement for staff at all – and this efficiency is a key reason why businesses are expected to rush to embrace genAI solutions this year.
A recent Everest Group survey of more than 50 global chief information officers (CIOs), for example, found that 61 per cent of enterprises are “actively exploring and piloting” genAI and that 22 per cent have already deployed the technology for one or more business processes.
“Although enterprise adoption of genAI is far from its anticipated peak, enterprises continued to experiment with unique use cases in a wide variety of industries,” said Everest Group partner Abishek Singh.
“As more of these initiatives document measurable impact, we’ll see adoption and full-scale implementation of genAI accelerate considerably.”
But where do the guardrails go?
For all their enthusiasm, however, genAI bots’ susceptibility to manipulation – which has also seen researchers pressure ChatGPT into offering illegal financial advice, widely available hacks that trick it into ignoring its rules against providing information about making weapons, security researchers getting it to create phishing tools and behave like a scammer, and more – is an Achilles’ heel for any business system that relies on the technology to treat its customers in a consistently professional, helpful manner.
Microsoft learned the lesson years ago, after training its ‘TayTweets’ chatbot on Twitter content that turned it into a racist, antisemitic misogynist.
Eight years later, the technology has improved dramatically – and with genAI fast becoming ubiquitous in business applications and Apple expected to soon follow the lead of Samsung, which this month debuted its smartphone-based Galaxy AI system, systemic weaknesses in genAI-based systems will be hard to avoid.
Regulations on the technology may or may not help, with industry providing a mixed reaction to the government’s recent declaration that it will focus regulation on “high-risk” applications and let the market sort out the rest – leaving businesses to gauge their risks and implement technological guardrails to stop rogue chatbots from damaging their brand.
A recent study found that ChatGPT’s design “can significantly influence recommendations” it makes to consumers and warning that “understanding the mechanisms underpinning [LLMs’] ‘thought’ processes will become increasingly important for determining their impact on consumers”.
Chatbots will be the primary customer service channel for a quarter of all organisations by 2027, according to Gartner, which recently identified six potential legal, compliance, and business risks that companies must address when integrating genAI into their business processes – including fabricated answers, data privacy and confidentiality, model and output bias, intellectual property violations, cyber fraud, and consumer protection.
“The output generated by ChatGPT and other LLM tools are prone to several risks,” Gartner Legal & Compliance Practice senior director analyst Ron Friedmann said, warning legal and compliance leaders to evaluate the potential impact of such risks and warning that “failure to do so could expose enterprises to legal, reputational and financial consequences.”
