Training warm and friendly AI systems could also mean they become sycophantic and promote conspiracy theories, a new study has warned.
Large language models such as ChatGPT are being actively encouraged to be friendly to their users – in part because users have expressed a desire for such warm responses. When OpenAI tweaked its systems so that they would be less flattering, for instance, users rebelled and the company had to roll back its updates.
That has led the companies that develop the systems to specifically focus on making their tools more warm. OpenAI says it explicitly makes its tools “helpful, honest and harmless”, while Anthropic aims to make its systems “empathetic” and “engaging”.
Other AI companies are specifically making their tools to behave like friends to their users. Sites such as Replika and Character.ai sell their chatbots as friendly or even potential romantic partners.
But those changes could also mean that the systems are more willing to promote inaccuracies, be overly sycophantic and even indulge in conspiracy theories, according to a new study from Oxford University.
In the test, researchers trained a series of large language models to be used in chatbots, similar to the way they are implemented in ChatGPT and Anthropic’s Claude.
They found that those who had been trained to be warmed in their responses were 30 per cent less accurate, and 40 per cent more likely to agree with users in their false beliefs.
The researchers also found that the chatbots were particularly keen to validate users’ false beliefs when they expressed feelings of sadness. An increasing number of people are turning to such systems to fill the role of counsellors and therapists, leading to increased concern.
”As these systems are deployed at an unprecedented scale and take on intimate roles in people’s lives, this trade-off warrants attention from developers, policymakers and users alike,” the researchers warn in the new paper, published in the journal Nature.
