Some day we might all be walking around wearing smart glasses, and now that I’ve been testing the Galaxy XR, I have a pretty good sense of how they might work. This mixed reality headset has been a mixed bag so far (full review to come), but it has a secret weapon no other device can match right now. And it actually changes the way I think about AI and AR and how they’re colliding — in a good way.
I’m talking about Gemini Live, which is the AI assistant that’s built right into the Galaxy XR. All you have to do is long-press the top right button on the headset (or say “Hey, Google) and you can start chatting with Gemini. Even better, you can share your screen with Gemini so it can “see” whatever you see. And it’s a potential game changer for how this category evolves.
An always-ready AI Assistant
For example, I was looking through Google Photos on the Galaxy XR and I wanted to know what a particular flower was. I could ask on the fly, “What kind of flower is that?” and I got an answer almost immediately. It turns out that it was a purple Clematis viticella, and I could ask follow up questions on how to care for it and more.
Gemini can see your real-world view through the Galaxy XR’s pass-through as well. For example, while standing in my sun room next to our wet bar, I looked that the liquor bottles on my shelf and asked what I could make out of the ingredients. Gemini gave me the recipe for a chocolate martini in a separate window that I could use to follow along.
Since the Galaxy XR has Netflix on board, I wanted to see if Gemini could help me decide if the No. 1 movie was worth watching. So I asked Gemini to pull up reviews for “The Woman in Cabin 10” on Rotten Tomatoes. Except that I didn’t even need to say the name of the movie, because Gemini could see what’s on my screen.
Your gaming copilot
Gemini really impressed me through the Galaxy XR when I used it as a gaming co-piilot. While playing “Diablo Immortal” through the headset I asked for advice on where to go, as well as which attacks would help me defeat a boss in a battle. Gemini knew at a glance what game I was playing, the character I was playing as (a monk) and my special skills.
Gemini does have some limitations through Galaxy XR. It can’t see DRM-protected content. So, for example, while watching an NFL game through YouTube TV, I couldn’t get any answers when I asked “how many rushing yards does he have” while looking at Saquon Barkley. Instead, I had to ask Gemini specifically about the player to get any info.
I also wish Gemini could do a bit more in terms of performing actions on your behalf, While you can use it to open apps and look up info on pretty much any topic, it would be nice if I could say “Hey, Google, start a screen recording.”
Speaking of which, it would be nice if you could use Gemini screen sharing and record your Galaxy XR screen at the same time. Right now you can't. However, I suspect that Gemini will get more capable over time through the Galaxy XR via software updates.
Are we giving AI too much access?
Some people might be freaked out about having an AI assistant watch their every move through the Galaxy XR, but at least it’s easy to pause screen sharing and Gemini altogether. And you could always opt for voice-only mode if you don’t want the assistant to have eyes on your virtual or physical world.
And I also like that you can choose to have live captions turned on for Gemini, which could come in handy for those who are hard of hearing. Or if you want to play your game at full blast and still see the advice from your AI buddy pop up on screen.
Over time, I can see Gemini Live having a big impact on smart glasses, and Google has already announced that they will be working with partners like Gentle Monster and Warby Parker. But for now I think the Galaxy XR is a pretty compelling example of where AI on your face is going.
