Meta now alerts parents if their teen discusses suicide or self-harm using its AI-powered chatbot

Meta announced Thursday that it will now notify parents if their teen discusses suicide or self-harm using the company’s Meta AI software. Meta says it’s also working on the ability to contact emergency services if someone’s conversations indicate they may be at risk of harming themselves.

These changes arrive as Meta and other tech companies face scrutiny from regulators and parents over how AI-powered chatbots operate. Responding to users in crisesespecially teens – a question of liability that is increasingly affecting how AI companies design and market their products.

Meta says it has built a custom AI system to identify conversations in which a teen is clearly referring to self-harm.

“We understand how annoying these alerts can be for parents,” Mita wrote in a blog post. “That’s why, as we continue to improve our detection, all reported chats will be manually reviewed by our AI before an alert is sent. If a teen’s intent is ambiguous, we will err on the side of caution and alert parents. While this means we may occasionally notify parents when there is no real cause for concern, we feel this is the right starting point, and we will continue to monitor to help make sure we’re in the right place.”

These alerts are now available to parents who use Instagram Parental supervision in the US, UK, Australia and Canada, and will be rolled out globally by the end of the year, Meta says.

This update depends on Alerts that Meta actually sends For parents when their teen repeatedly searches for the terms suicide or self-harm on Instagram. It also depends on which feature Allows parents to see topics Their teen discussed with Meta AI over the past week.

Mita also declared that “Limited contentThe setting — which lets parents put their teens into a more restricted Instagram experience — now applies to Meta AI as well. Meta AI is already trained to avoid sexual, romantic, or alcohol-related discussions with teens, and the Limited Content setting expands those safeguards by having the chatbot reject a broader range of prompts. Meta didn’t specify what these additional prompts include, but TechCrunch asked the company for more information.

Additionally, Meta says it will contact emergency services if someone’s conversation with Meta AI, whether the user is an adult or teen, indicates that someone is at risk of suicide. It’s worth noting that Meta already takes this step when someone posts something on Facebook or Instagram that indicates they’re at risk, so this same action extends to conversations with its chatbot.