Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
The year 2026 began with a terrifying example of the potential for misuse of generative AI. Grok, Elon Musk’s xAI tool, It was used Photos of people being undressed or undressed are being shared on X (formerly Twitter) at an alarming rate. Groke made 3 million sexual images over 11 days in January, of which nearly 23,000 contained images of children. According to a study From the Center for Countering Digital Hate.
Now, competitors like OpenAI and Google are beefing up their security measures to avoid becoming the next “Grok.”
Safety advocates and researchers have long been concerned about AI’s ability to create offensive and illegal content. Creating and sharing non-consensual intimate images, sometimes referred to as revenge porn, was a big problem before AI. Generative AI makes it faster, easier and cheaper for anyone to target and harm people.
On January 14, two weeks after the scandal occurred, X’s Safety account confirmed it in a message mail It will temporarily stop Grok’s ability to edit photos on the social media app. Grok’s image creation capabilities are still available to paying subscribers to its standalone app and website. X did not respond to multiple requests for comment.
Most major companies have safeguards in place to prevent the kind of large-scale breaches we saw possible with Grok. But cybersecurity is never a solid metal wall of protection; It is a brick wall that is constantly undergoing repairs. Here’s how OpenAI and Google tried to strengthen their safety protections to circumvent Grok-like failures.
Read more: AI Slop is destroying the Internet. These are the people fighting to save her
OpenAI fixes vulnerabilities in image generation
At a basic level, all AI companies have policies that prohibit the creation of illegal images, such as child sexual abuse material, also known as CSAM. Many tech companies have guardrails to completely prevent the creation of intimate images. The Grok is the exception, with “warm” modes for photos and video.
However, anyone intending to create non-consensual intimate images could try to trick AI models into doing so.
Researchers from Mindgard, a cybersecurity company focused on artificial intelligence, A vulnerability has been found in ChatGPT Which allowed people to circumvent its guardrails and take intimate photos. They used a tactic called “hostile induction,” where testers try to create vulnerabilities in the AI with specially crafted instructions. In this case, the software was tricking the chatbot’s memory with personalized prompts, then copying the nude pattern onto photos of known people.
Mindgard alerted OpenAI to its findings in early February, and the ChatGPT developer confirmed on February 10 — before Mindgard announced its report — that it had fixed the issue.
“We are grateful to the researchers who shared their results with us,” an OpenAI spokesperson told CNET and Mindgard. “We moved quickly to fix the flaw that allowed the model to generate these images. We appreciate this kind of collaboration and remain focused on strengthening safeguards to keep users safe.”
This process is how cybersecurity often works. Researchers outside of the Red Team, such as the Mindgard program, test vulnerabilities or workarounds, simulating strategies that bad actors might use. When they identify vulnerabilities, they alert the software provider so fixes can be deployed.
“Assuming that motivated users won’t try to bypass safeguards is a strategic miscalculation. Attackers are repeating this. Guardrails must assume continuity,” Mindgaard wrote in a blog post.
While tech companies brag about how their AI can be used for any purpose, they also need to make a strong promise that they can prevent AI from being used to enable abuse. For AI image generation, this means having a robust set of prompts that will be rejected and returned to users.
When OpenAI launched its Sora 2 video model, it promised to be more conservative in content moderation for this very reason. But it’s important to ensure that stewardship practices are effective consistently, not just at product launch. It makes testing AI health an ongoing process for cybersecurity researchers and AI developers alike.
Watch this: Artificial intelligence is indistinguishable from reality. How do we detect fake videos?
Google is upgrading its search reports
For its part, Google is taking the necessary steps to ensure that offensive images do not spread easily. The tech giant has streamlined its process for requesting the removal of explicit images from Google search. You can tap the three dots in the upper right corner of the photo, tap “Report” and then tell Google you want the photo removed because it “suggests a sexual image of me.” The new changes also allow you to select multiple images at once and track your reports more easily.
“We hope this new takedown process will reduce the burden faced by victims of explicit, non-consensual images,” the company said in a statement. Blog post.
When asked about any other steps the company is taking to prevent misuse of artificial intelligence, Google pointed to CNET Amnesty International Prohibited Use Policy. Google’s policy, like that of many other tech companies, prohibits the use of AI for illegal or abusive activities, such as creating intimate images.
There are laws aimed at helping victims when these images are shared online, such as the Take It Down Act of 2025. But the scope of this law is limited, which is why many advocacy groups, such as the National Center on Sexual Exploitation, oppose this law. Pressure for better rules.
There is no guarantee that these changes will prevent anyone from using AI for harassment and abuse. That’s why it’s so important that developers remain vigilant to ensure we’re all protected – and act quickly when reports and issues arise.
(Disclosure: Ziff Davis, the parent company of CNET, in 2025 filed a lawsuit against OpenAI, alleging that it infringed Ziff Davis’s copyrights in training and operating its AI systems.)