Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
In my office there are bookshelves where I keep my magic tricks. Next to them are panoramic windows overlooking Paris – except, not really. It’s the mixed reality overlays that greet me when I get dressed See Apple Pro Headset, mapped to my actual windows. A virtual clock hanging on the wall.
The bookshelves are real. Paris windows and wall clocks are not. But Siri, whom I conjure as a glowing orb in front of me, sees them all. When I ask what’s in front of me, Siri’s text box describes it all.
My first little moments while wearing the Vision Pro and testing A comprehensive view of Siri AI In the developer preview for VisionOS 27 Surprising but very familiar. I’ve had moments like this Samsung Galaxy XR The headset, which already has a Gemini Live mode that can recognize your room, default apps and open windows. I did too I wear many smart glasses I asked the camera-aware AI to tell me about the world as I walked around.
As I expectedApple acquired The new supercharged Siri It’s coming to a range of devices this fall, from watches to iPhones to iPads, and is expanding its visual intelligence capabilities beyond the iPhone. However, the Vision Pro’s Siri is the coolest, and perhaps the most ambitious. It can see everything in your field of vision on demand, like a sensory companion.
Not everything is perfect in this early preview build, which was revealed just days ago in Apple’s WWDC conferencebut it already works well enough to show what could come next. Yes, glasses that can do that seem quite possible now. In the meantime, the auxiliary functions of Vision Pro will also become interesting.
Siri knows my bookshelf, my virtual window in Paris, and my virtual clock.
Siri’s glowing orb is ready to see from the start
Unlike Visual intelligence On iOS and iPadOS, powered through the Camera app, Siri from Vision Pro can see things when you simply say, “Hey, Siri,” and ask a question.
Siri appears as a glowing 3D ball, appearing in my room like a spirit. You can drag and drop it anywhere, but it’s cool to see the glowing marble light effects on my desk or room using VisionOS’s spatial graphics engine.
I was able to say, “Hey Siri, what’s in front of me?” This stimulated the visual intelligence, emitting a new chirping sound and then performing a quick visual scan around the area my eyes were looking at, using eye tracking. He casts a wide net: he can see my bookshelf full of books in front of me and all the little action figures and toys I put there. She read me some book titles, like Uzumaki and Wonderbook.
The AI’s feedback comes after the camera takes a still image of what’s in front of you, whether real or virtual. It’s similar to the other camera-based AI snap response rhythm on Samsung’s Meta or Galaxy XR glasses.
But it doesn’t have a direct mode like those glasses and headphones. It’s a matter of one shot per order.
My cluttered bookshelf was recognized by Siri. Siri’s orbit glows near real-world objects, too, and can be pinched and pulled.
Identify the real and the virtual
On my desk, Siri could see a red VR headset (Virtual boy) and a Steam surfaceBoth are real. Next, I tried to look at the virtual windows of Paris, the wall clock, and the bookshelf full of magic, and recognize them as well.
In this early beta, Siri tended to stick to a particular show for a period of time and analyze that capture unless I closed Siri or moved the orb to try again. Or, I guess, I’m just getting used to it.
But I’m already wondering how I can use this visual awareness to get through work quickly. I asked Siri for a Notes app summary of my recent playwriting, which I pulled up just by asking (it summarized the main points), and I asked it about the browser windows I had open on my MacBook via the virtual screen. (This tells me now, as I work on this story in Vision Pro via my Mac, I have a Google Doc window open where I write about Siri’s recognition of real and virtual objects.)
Just curious what Siri thinks of the Virtual Boy and Steam Deck.
Convert my own photos into 3D panoramic wallpapers
I also got to try out the new panoramic photo conversion effect in VisionOS 27, which lets any panoramic photo in your library be transformed into a 3D wraparound background that you can use while you work.
The results aren’t fully 3D like Apple’s own immersive environments, and don’t move or have ambient sound effects. They’re like very large 3D windows, with some of my desk still visible around the edges.
It looks nice, although not all of my panoramas have converted (it’s very early days, I’m sure it will work). I took a photo of my mom in her backyard when I visited during the pandemic, and now she can greet me like I’m about to get there again. I would be happy if Vision Pro is added Captures perfectly immersive Gaussian noise From multiple images, as I said Done on Meta Quest.
I can’t help but think about how Apple’s expected smart glasses will use Siri for vision for assistive purposes. But I’m getting ahead of myself. Right now, you need the $3,499 Vision Pro to make that happen. But when Apple has AR glasses like Google and the Xreal Aura Project Are they smaller and more affordable? When can other apps link to the Siri framework? It will be great to see what happens next. I feel like I’m getting a glimpse of things to come.
