Apple’s new AI photo editing tools mostly work, for better and worse

The world’s most popular camera just got its first set of serious AI-powered photo editing features, and I don’t think any of us are ready for it.

As far as AI photo editing is concerned, the new features in iOS 27 are pretty weak compared to what you can do on Google’s Pixel phones, for example. But for iPhones, it represents a turning point in what the native Photos app allows you to do with your photos. namely memories. I mean I don’t know anymore.

These new features are part of the iOS 27 developer beta at the moment, so keep in mind that Apple may still make tweaks to them before releasing them to the general public. There are three, or maybe two and a half, new AI editing features in this update. The new cleaning tool is half, because it was there before but It was so bad it didn’t really matter. This is the tool that lets you remove spoilers from the background of your photos, and it’s gotten a major upgrade this year. There’s also an Extend feature, which lets you extend the edges of your image using AI to draw in some reasonable-looking padding. And there’s spatial reframing, which simulates the effect of moving the camera around the scene to allow you to recreate an existing image. It is the most ambitious and perhaps the most problematic of the three.

But first things first: cleaning. It’s actually good now. Instead of using only on-device models to remove objects and fill in details, more powerful models can now be used in the cloud. This is what Google has been doing for years, which is why the company’s Magic Editor tools are so much better than the version Apple introduced last year. The cleaning process on the entire machine wasn’t very good at drawing convincing details to replace what was removed. It left strange artifacts and was generally more trouble than it was worth. Cleaning 2.0? It does the job.

Using AI to remove objects from images is the generative editing tool I don’t worry about using. I would use it to clear snot out of my child’s nose or bring out a stranger from the background. This new version in iOS does all that without a problem, and I think it will be popular with iPhone owners.

To go to the next level of complexity and what the image is, there is expansion. Think of it like cutting, but in reverse. It allows you to widen the edges of your frame, which you may want to do if your composition is too tight for your subject and you want to give it more room to breathe, for example. Extend lets you do this, but only to a certain extent. He seems to avoid making adjustments to people, and will sometimes tell you that an image can only be stretched in a certain direction. It will just add a little padding as well, reducing the type of tricks it can be used for. I appreciate that. Like Clean Up, it does its job convincingly. He seems to tend to search for symmetry, which usually works. She added part of a rally car that was out of frame in my original photo, and added a side mirror to match the one already in the photo.

He doesn’t seem keen on making things to put in your photos, for example, Samsung’s early efforts. But I noticed the addition of a potted plant on a side table; It seems reasonably convincing, however I Be aware that it is not a real plant. I would feel weird about it if I put that picture on Instagram.

Expanding works with your image in 2D space; Spatial reframing adds a third dimension. It builds on an existing feature that makes your photos look 3D, allowing you to reframe the photo as if you had physically moved the camera and changed your perspective of the scene. You can’t go also Away from it – only as far as you could move your arm when the photo was originally taken. But the idea is that you can fix your framing if you’re not quite perfect when you take the shot.

See, this appeals to my type A nature. Sometimes I love everything about the photo I took, except wouldn’t it be better if I moved to the left to avoid framing something distracting in front of my subject? You can’t always capture these things in real time. These are the minor adjustments that spatial recasting is designed for.

It sounds reasonable, but there is room to create existential chaos, even with minor modifications. I tried to reframe a photo I took at a tech talk with Apple executives after the WWDC keynote. Since I was sitting to the side, one of the executives on stage was obscured in the original shot. I changed the frame and the AI ​​created a man sitting next to Craig Federighi.

Predicting what should come next in two-dimensional space seems like an easier problem than predicting what should happen in three-dimensional space. The results for spatial reframing compared to extension reflect this. They’re weirder. The further you move away from a subject, the less room you have for “recomposition,” and the more realistic the objects generated by the AI ​​become. But you end up with a photo that’s only slightly different from the actual photo you took, and at that point, what do we do?

In photos where the subject is closer to the camera, things get weird. The effect of “recomposition” is more dramatic, and the AI ​​must work harder to fill in the gaps. You can change perspective in a portrait, but that means the AI ​​has more detail it needs to fill in on your face, which makes the valley a little weird very quickly. Even within a limited adjustment range, it can make faces look a little skewed and “off.” He is more likely to invent things that did not exist. Sure, that It seems It’s nice to be able to rescue an image from a composition that wasn’t quite right. In practice, I don’t think I like it.

Conveniently, photos edited with these AI tools will get Synth ID labels that indicate they have been edited using AI. Instagram picked up this information when you uploaded some photos, but it doesn’t display that information unless you click on the “AI Info” menu for that photo. Stickers aren’t an ironclad solution right now, and I think the biggest risk is the rapid erosion of the idea that we can usually trust a photo someone takes and posts from their phone. Apple is certainly not the boldest player here. But even introducing a little doubt about where the houseplants on a side table came from, or whether someone was actually standing in the spot where they appear to have taken the photo, can lead to a lot of trouble over time.

Photography by Alison Johnson/The Verge