Fretting about AI taking your job is so passé. At best, it will upend society as we know it. At worst, it could kill and dismember us to make paperclips. Gainful employment will be the least of your concern if humanity got turned into stationery. At least, that’s the premise of the thought experiment haunting Silicon Valley since 2014, when philosopher Nick Bostrom published Superintelligence: Paths, Dangers, Strategies.
Elon Musk is a fan. “Worth reading,” he posted on Twitter, now rechristened X after Musk bought the platform. “We need to be super careful with AI. Potentially more dangerous than nukes.” Not so dangerous Musk hasn’t invested heavily in AI with xAI, which birthed his chatbot Grok. He also donated $1 million to the Future of Humanity Institute (FHI), an interdisciplinary Oxford University hub run by Bostrom.
Musk clearly enjoyed Bostrom’s follow-up book, Deep Utopia: Life and Meaning in a Solved World (2024), endorsing it with a hearty “yeah”. By then, Bostrom’s philosophical career had become more complicated. In 2023, he apologised for using a racist slur in an old email. The next year Oxford closed the FHI, citing funding issues, and Bostrom resigned from the university. Yet he remains an influential figure in AI circles. Bill Gates lists Superintelligence as a must-read. Sam Altman, founder of ChatGPT creator OpenAI, said it was “the best thing I’ve seen on this topic”. The topic being the problem of an AI that — oops — accidentally wipes us out.
That’s the paperclip maximiser. Bostrom theorised a superintelligent AI given a seemingly benign task, say, overseeing a paperclip factory, could buy a one way ticket to our extinction. Forget to inform the superintelligence to prioritise human life, and it could decide maximising production would be best served by turning us into paperclips.
“It’s a cartoon example,” Bostrom tells me. “But it’s a stand-in for the type of future that might result if human values don’t shape developments, [and] instead they get determined by some super intelligence that seizes power and happens to have goals that just don’t involve humans or human flourishing.”
Rules of engagement
Bostrom published Superintelligence, the philosopher has been amazed at the progress made in an AI race he was originally spitballing about. “It’s all happening now,” he says. “I’m quite impressed by the speed of developments that we’ve seen in the past several years. It looks like we are on the track towards AGI.”
AGI being Artificial General Intelligence — an AI that could meet or even surpass human intelligence. Not to be confused with GenAI, or Generative AI, the kind of machine learning that powers chatbots such as ChatGPT and Grok. Large language models (LLMs) trained to answer questions, remember details and serve as tools and act as digital assistants. Everyone from private companies to nuclear superpowers wants to be the first to develop AGI. No one wants to be second. But while Bostrom is the tech bro thinking man’s crumpet, he’s also their Cassandra, making dire warnings that, unheeded, could spell doom.
Bostrom once said superintelligence was a greater risk to humanity than climate change. A decade later, the climate crisis fully underway, does he stand by that? “There remains always the possibility that human civilisation might destroy itself in some other way, such that we don’t even get the chance to try our luck with super intelligence,” he says.
But AGI is, in his opinion, coming — like it or not. Personally, he’s open to the radical potential it would bring. “Completely reorganising society could be a positive thing,” says Bostrom.
He corrals his risks of AGI into four areas. The first, alignment, covers the paperclip theory. “An alignment failure might result in features that are very suboptimal from our point of view,” suggests Bostrom. “As we build increasingly capable AI systems, how to develop scalable methods for AI control such that we can make sure that these AIs don’t harm us, that they remain aligned with the intentions of their creators.” Bostrom feels that currently AI development labs are thinking critically about this problem.
Secondly, governance, which pertains to how said creators come to a diplomatic agreement not to use AGI for evil. “How we can ensure we humans use these powerful AI tools primarily for beneficial purposes,” says Bostrom. “As opposed to, for example, to wage war against each other or to oppress each other.” Here we’re already on shaky ground. AI systems have already been shown to have inherent biases against minorities, while technology such as AI-enabled drones have been enthusiastically adopted on the battlefield. Tech advances can always be used for good and ill alike.
Aliens and machine morality
While these first two get the most attention, there are two more ethical quandaries that Bostrom wants to consider. “It’s the challenge of making sure we respect the moral status of digital minds,” he explains. “This might sound strange, but as we are building increasingly sophisticated and complex AIs, it’s possible that some of those will have various degrees and forms of moral status.”
If, like Bostrom theorises, AIs one day outnumber “biological minds”, we will need rules of respectful engagement. Perhaps memes about slurs for AIs (“clanker”, the derogatory term for droids in Star Wars, is my favourite) could one day be considered genuinely hurtful.
Fourthly, how would we stop superintelligent AIs from oppressing each other? Bostrom postulates that an alien race could bring its own superintelligence into the mix. “If any of those types of beings exist, an important desideratum when we create our own superintelligence is to make sure that it will get along with these other super beings.”
As for the current conversation around the AI we already have — and whether it will take all our jobs — Bostrom is more sanguine. “The goal is full unemployment,” he says. “We’d need to find different bases for our self-worth and dignity and different ways of filling our lives and days, aside from having to work for a living.”
Children and retirees have a nice time of it, Bostrom points out. “Indeed, it used to be the ideal of the traditional British aristocracy that it was very desirable not to have to work for a living,” he says. “That it is almost demeaning to have to sell your time in order to make money.” A life of leisure and creative pursuit is the goal, sure — but funding that lifestyle without being landed gentry is tricky. Our oppressed human underclass won’t vanish overnight.
But Bostrom, who identifies as a “fretful optimist”, says he is excited for the future. “Ultimately, much greater space for human flourishing could be unlocked through advanced AI. If things go well, people will look back on 2025 and shudder in horror at the lives we lived.”
