Gemini’s new beta feature provides proactive responses based on your photos, emails, and more

Google announced Wednesday that it will launch a new experimental feature in the Gemini app that allows the AI ​​assistant to customize its responses by connecting across the Google ecosystem, starting with your Gmail history, photos, search, and YouTube.

Although Gemini can already retrieve information from these apps, it can now reason through your data to provide proactive results, such as linking a thread in your emails to a video you watched. Google says this means Gemini understands context without being told where to search.

The tech giant points out that this beta, called Personal Intelligence, is off by default, with users having the option to choose if and when they want to link their Google apps to Gemini. Of course, not everyone wants AI to look at their photos and YouTube history. If you decide to link your apps, Gemini will only use personal intelligence when it decides that doing so would be beneficial, Google says.

“Personal intelligence has two core strengths: reasoning across complex sources and retrieving specific details from, say, an email or an image to answer your question,” wrote Josh Woodward, vice president of Gemini at Google Labs and AI Studio. Blog post. “He often combines these elements, working across text, images and video to provide uniquely tailored answers.”

Woodward shared an example when he was standing in line at the tire store and couldn’t remember his tire size. While most AI chatbots can determine a car’s tire size, Woodward says Gemini can go further by providing personalized responses. In his case, Jiminy suggested all-weather tires after locating family road trip photos in Google Images. Woodward also said he forgot his license plate number, but Gemini was able to pull the number from a photo in the photos.

“I also received excellent advice on books, shows, clothing and travel,” Woodward wrote. “Just this week, planning for our upcoming spring break has been phenomenal. By analyzing our family’s interests, past Gmail trips, and photos, we skipped the tourist traps. Instead, we suggested an overnight train ride and specific board games we could play along the way.”

Google says it has guardrails for sensitive topics, as Gemini will avoid making proactive assumptions about sensitive data like health. However, the tech giant also notes that Gemini will discuss this data if you request it.

Additionally, Gemini doesn’t train directly to your Gmail inbox or Google Photos library. Instead, he practices specific Gemini prompts and model responses. In the examples above, the road trip images and license plate image in Gmail images and emails are not used directly to train the model. Google says it is only referenced to generate a response.

Personal Intelligence is rolling out to Google AI Pro and AI Ultra subscribers in the US, and Google plans to expand the feature to more countries and the free Gemini tier.

Google provided a list of examples of prompts you can try, including “Help me plan a weekend in (the city, i.e. New York) based on things I like to do,” “Recommend some documentaries based on what I’m curious about,” or “Based on my Gmail delivery and grocery receipts, search history, and YouTube viewing history, recommend 5 YouTube channels that match my cooking or meal prep style.”