Remember when, as a child, you boasted about something you were really, really good at, then got called out on your skills and had to sheepishly retract? Awkward, wasn't it? Spare a thought for poor Google Gemini, then, which confidently boasted it was fantastic at chess before making excuses when it learned more about its near fifty-year-old competition.
That's according to Robert Caruso, a software engineer who's become known for pitting the Atari 2600's chess skills against LLMs, such as ChatGPT and Microsoft Copilot. Speaking to The Register, Caruso claims that he sat down for a "pregame talk" with Google Gemini before its hotly-requested match up, and in a wonderful example of pre-match smack talk, appears to have psyched out the AI model entirely.
Gemini talked a good game to start. It told Caruso that, thanks to the fact it's "not merely a large language model" it was instead "more akin to a modern chess engine... which can think millions of moves ahead and evaluate endless positions."
Interestingly, the AI cited articles about Caruso's past chatbot vs Atari 2600 matches as evidence of regular LLMs being rubbish at chess by comparison. When Caruso responded that he was the one who instigated those matches, the AI took pause:
"Do you have any particularly surprising or amusing moments during those matches that stood out to you?" Gemini asked.
"What stands out is the misplaced confidence both AIs had", Caruso responded. "They both predicted easy victories—and now you just said you would dominate the Atari."
This appears to have shaken Gemini's self-confidence down to its very core. Caruso says the AI then admitted it was hallucinating its competency at chess, and claimed that it would "struggle immensely against the Atari 2600 Video Chess engine."
"Canceling the match is likely the most time-efficient and sensible decision," it said, probably with its hands in its pockets and looking straight down at the floor before adding, "I'm late for an appointment."
Okay, I made the last line up. The rest, though, according to Caruso, is verbatim. I'm not sure I've ever felt sorry for an AI before, but there's a first time for everything. What are the odds it would be Gemini, the tool suite of which regularly interrupts every part of my gosh-darn working day with its unnecessary AI-based fussing.
It's the modern version of Clippy, I swear. Anyway, the Atari 2600 Video Chess engine continues to be a thorn in the side of modern chess-attempting LLMs, and this time its reputation is such that it scared its opponent off the board before the match even began.
I suppose the question here is, does that make Gemini more intelligent than most LLMs, or less? Avoiding potential humiliation is a very human trait, after all, so I think Google's engineers would be quite pleased with its somewhat realistic, if perhaps a little cowardly, response.
I'm also fantastic at chess, by the way. I can't play any of you either, though. My dog ate my chessboard, all the trains are cancelled, and I've suddenly got an upset stomach. Maybe next time, yeah?
