I built marshmallow castles in Google’s new AI world generator

Google DeepMind is opening up access to Project Genie, its AI tool for creating interactive game worlds from text prompts or images.

Starting Thursday, Google AI Ultra subscribers in the US can try out the experimental search prototype, which is powered by a combination of Google’s latest global model Genie 3and the Nano Banana Pro and Gemini image generation models.

The move comes five months after the Genie 3 research preview, and is part of a broader campaign to collect user feedback and training data as DeepMind races to develop more capable global models.

Global models are artificial intelligence systems that generate an internal representation of the environment, and can be used to predict future outcomes and plan actions. Many AI leaders, including those at DeepMind, believe global models are a critical step to achieving artificial general intelligence (AGI). But in the near term, labs like DeepMind envision a go-to-market plan that starts with video games and other forms of entertainment and branches out into training embodied agents (aka bots) in simulations.

DeepMind’s release of Project Genie comes as the global modeling race begins to heat up. Fei-Fei Li Global Laboratories released it late last year The first commercial product called marble. Runway, the AI ​​video production startup, has done so as well It launched a global model recently. And former chief meta-scientist Yann LeCun’s AMI Startup Labs It will also focus on developing world models.

“I think it’s exciting to be in a place where we can have more people accessing it and giving us feedback,” Shlomi Fruchter, director of research at DeepMind, told TechCrunch via video interview, smiling from ear to ear in obvious excitement about the release of Project Genie.

The DeepMind researchers TechCrunch spoke to were frank about the experimental nature of the tool. They can be inconsistent, sometimes generating impressively playable worlds, and other times producing confusing results that miss the mark. Here’s how it works.

You begin by “drawing the world” by providing text prompts for both the environment and the main character, which you will later be able to maneuver through the world from either a first-person or third-person perspective. Nano Banana Pro creates an image based on prompts that you can, in theory, edit before Genie uses the image as a starting point for an interactive world. The adjustments worked for the most part, but the model would occasionally stumble and give you purple hair when you called for green.

You can also use real-life images as a baseline for the model to build a world on, which, again, is hit or miss. (More on that later.)

Once you’re satisfied with the image, Project Genie will take a few seconds to create an explorable world. You can also remix existing worlds into new interpretations by building on their prompts, or exploring curated worlds in the gallery or via the randomizer for inspiration. You can then download videos of the world you just explored.

DeepMind only gives 60 seconds of global generation and mobility at the moment, partly due to budget and computing constraints. Because Genie 3 is Autoregressive modelit requires a lot of dedicated computing — which puts a tight cap on how much DeepMind can provide to users.

“The reason we limited it to 60 seconds is because we wanted to offer it to more users,” Fruchter said. “Basically, when you use it, there’s a segment somewhere that’s just for you and assigned to your session.”

He added that extending it beyond 60 seconds would reduce the additional value of the test.

“The environments are interesting, but at some point, because of their level of interactivity and the dynamism of the environment, they are somewhat limited. However, we see that as a limitation that we hope to improve.”

Whimsy works, realism doesn’t

When I used the model, the safety barriers were already up and running. I couldn’t produce anything resembling nudity, nor could I create worlds that even remotely inspired Disney or other copyrighted material. (In December, Disney hit Google with a cease and desistaccusing the company’s AI models of violating copyright by training on Disney characters, intellectual property, and creating unauthorized content, among other things.) I couldn’t even convince Jenny to create worlds of mermaids exploring underwater fantasy lands or ice queens in their winter castles.

However, the demo was very impressive. The first world I built was an attempt to live out a little fantasy from my childhood, where I could explore a castle in the clouds made of marshmallows with a river of chocolate sauce and trees made of candy. (Yes, I was a fat kid.) I asked the model to do it claymation style, and she introduced me to a strange world I would have eaten as a kid, where pastel-and-white castle turrets and towers looked puffy and delicious enough to tear off a piece and dunk in a chocolate moat. (Video above).

However, Project Genie still has some kinks to work out.

The models excelled at creating worlds based on artistic prompts, such as the use of watercolor, cartoon style, or classic anime aesthetics. But they tend to fall flat when it comes to realistic or cinematic worlds, often appearing like a video game rather than real people in a real setting.

She also didn’t always respond well when given real pictures to work with. When I gave him a picture of my office and asked him to create a world based on the photo exactly as it was, he gave me a world that contained some of the same furniture as my office—a wooden desk, plants, and a gray sofa—arranged differently. It looked sterile and digital rather than lifelike.

When I fed it a picture of my desk with a stuffed toy, Project Genie animated the toy as it moved through space, and even had other objects sometimes interact as it moved next to it.

This interaction is something DeepMind is working to improve. There were several occasions where my characters ran through walls or other solid objects.

When DeepMind initially released Genie 3, researchers highlighted how the model’s autoregressive architecture meant it could remember what it had created, so I wanted to test this by going back to parts of the environment it had already created to see if it would be the same. For the most part, the model worked. In one case, I created a cat exploring another desk, and only once when I returned to the right side of the desk did the model create a second mug.

The part I found most frustrating is the way you navigate space using the arrows to look around, the space bar to jump or climb, and the WASD keys to move. I’m not a gamer, so this didn’t occur to me naturally, but the keys were often unresponsive, or sent you in the wrong direction. Trying to walk from one side of the room to the doorway on the other often became a chaotic winding exercise, like trying to steer a shopping cart with a broken wheel.

Fruchter assured me that his team was aware of these flaws, and reminded me again that Project Genie is an experimental prototype. In the future, he said, the team hopes to enhance realism and improve interaction capabilities, including giving users more control over actions and environments.

“We don’t think of (Project Genie) as a comprehensive product that people can come back to every day, but we think there’s already a glimpse of something interesting and unique that can’t be done otherwise,” he said.