Patreon stops asking AI bots not to scratch, and starts banning them

Patreona membership platform for creators, is cracking down on AI that scrapes its content for training purposes. On Thursday, the company shared that it is working with internet infrastructure provider Cloudflare to directly block access to AI bots designed to train its AI models on creators’ works without permission.

The company says the enhanced measures were necessary because the process of AI scraping has become more complex since it first put measures in place to deter AI crawlers in 2023. Additionally, Patreon’s long-standing paywall has locked much of creators’ content out of the reach of crawlers. But recently, the company has introduced new discovery tools like a redesigned Home Feed and its features Jokes that look like tweetswhich may expose more content to crawlers.

These changes come as more publishers and online content creators are learning how AI can absorb their work to make their AI models smarter. To combat this, Cloudflare now offers tools that allow website publishers To restrict AI botsincluding A A marketplace that allows websites to charge AI bots for scrapingit’s called “pay per crawl”. Earlier this month, It changed its policies So that “diversified” crawlers, i.e. those that index and train website content, are blocked by default on any pages hosting ads.

Patreon says it is expanding its existing work with Cloudflare to use the company’s AI Crawl Control technology to modernize its AI policies and enforcement tools. The difference here is that instead of simply requiring AI crawlers not to scrape content using robots.txt files — a standard method of providing bots with instructions on how to use their site — Patreon is now actively banning AI training bots.

“Consent should not depend on whether the person doing the scraping has chosen to act or not,” A Patreon blog post He explains, referring to the stricter measures.

When testing the features, weekly attempts by individual AI training crawlers to access Patreon dropped from “thousands of attempts to zero,” the post noted. This suggests that the AI-powered scrapers were ignoring Patreon’s robots.txt file and deleting the site anyway, despite its requests.

However, the company said it will allow bots to index pages and organize information that can be used to bring users back to Patreon.

“As AI agents grow in power and popularity, creators deserve to have a meaningful say in how AI companies use their work,” Patreon head of product Drew Rooney commented in the announcement. “On most websites, creators have to accept AI training in their work just to reach and grow an audience. Patreon has a different vision: Creators should be able to grow their audience and control how their work is used.”