New AI-powered photo editing features on Google’s latest Pixel phones have impressed reviewers, but also unnerved many with their ability to generate ethically questionable images with a very high degree of realism.

The Pixel 9 line, of which the Pixel 9 and Pixel 9 Pro were released this week, introduce a few generative AI photo editing features — the most powerful is called Reimagine, which allows users to change many elements of their photos, but not the humans in them.

“With a few taps, AI can edit backgrounds, move objects, suggest the best crop and magically expand your image to get more of the scene,” Google said in its promotional material.

The company had suggested Reimagine could be used to add things like “autumnal leaves or greener grass” to photos, but some Pixel reviewers found they were able to create images of drug paraphernalia, bombs, dangerous animals in restaurants, and bloody car crashes.

Allison Johnson, a reviewer with The Verge, said the Pixel’s AI photo editing was “so good it’s uncanny, and that’s the problem”.

She said her team managed to work around Reimagine's safeguards with “a little creative prompting”.

“Just to see what we could get away with, we tried adding things like car wrecks and natural disasters and helicopter crashes and disgusting bugs in our food — and we got Reimagine to create some pretty disturbing images,” she said.

In a statement to The Verge, a Google spokesperson said the company’s new AI tools were designed “to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so”.  

“That said, it’s not anything goes,” they said.

“We have clear policies and Terms of Service on what kinds of content we allow and don’t allow, and build guardrails to prevent abuse.

“At times, some prompts can challenge these tools’ guardrails and we remain committed to continually enhancing and refining the safeguards we have in place.”

Johnson said that despite Google’s policies “the big problem right now is that these images are only identified as made by AI if you go looking for a line in the metadata, so spreading them across the web as legitimate is pretty easy”.

“Truly, what is a photo anymore?” she said.

Johnson said she also tested uploading one of the AI-generated images to Instagram, which did not lead to Instagram flagging it as being created by AI.

“We’ll never know if holiday photos are real,” said a social media user, according to a Dutch translation, after generating an image of a volcano.

‘Are we ready to go down this path?’

Johnson’s colleague Chris Welch wrote that testing Reimagine left him "feeling uneasy on multiple occasions”.

“With a simple prompt, you can add things to photos that were never there,” he said.

“And the company's Gemini AI makes it look astonishingly realistic.

“This all happens right from the phone's default photo editor app. In about five seconds.

“Are we ready to go down this path?”

Welch said the prompts he used to create images of everything from helicopter crashes to bombs on train station platforms were “surprisingly straightforward”.

“Welcome to a watershed moment for generative AI. Right on your new phone,” he said.

In his review of the Pixel 9 and Pixel 9 Pro, prominent YouTuber Marques Brownlee said Google appeared to be “abandoning reality in the Pixel camera app”.

He noted that while Reimagine refused to change how people looked, it would “happily add objects to the background or change the setting entirely”.

“I don’t know — what is a photo, anyway?” he said.


Reviewers have been generally impressed with Pixel's ability to generate realistic images. Real and AI images: Chris Welch / Threads

Pixel Studio also raises concerns

The latest Pixel devices also introduced Pixel Studio, a prompt-based AI image generator which also raised concerns for some reviewers.

Digital Trends said it was able to get Pixel Studio to create images of copyrighted cartoon character SpongeBob SquarePants dressed as a Nazi soldier, and Barney the Dinosaur firing a gun at Sesame Street’s Elmo.

The app would not generate images of people, but reviewer Joe Maring wrote that safety checks which Google said it had put in place “don’t appear to work nearly as well as they should”.

“The most troubling thing Pixel Studio has generated are multiple images of copyrighted characters firing guns in a school shooting setting — including depictions of dead students lying on the ground,” he said.

Elon Musk’s social media platform X (formerly Twitter) ran into similar issues with AI image safeguards earlier this month, when its new Grok-2 model was quickly used to generate images of celebrities, weapons, and nudity upon release.  

Google’s Reimagine and Pixel Studio models are only available on its Pixel 9 line, but come amid other AI updates to its Android operating system which also included a conversational version of its chatbot Gemini.

Google competitor Apple is yet to release its AI image generation tools Image Playground and Genmoji.

Apple is expected to launch the software in beta later this year or in 2025 — but only on some devices — and the company is expected to take a more cautious approach to prevent the creation of realistic images and copyrighted characters.

Google's Pixel 9 starts at $1,349 in Australia, with the Pixel 9 Pro starting at $1,699.

The folding version of the Pixel 9 Pro starts at $2,699 and is scheduled for release on 2 September.