Creative Commons announces initial support for “pay-to-crawl” AI systems

After it was announced earlier this year framework For an Open AI Ecosystem, the non-profit Creative Commons I preferred “pay for crawling” technology – a system to automate compensation for website content when it is accessed by machines, such as AI-powered web crawlers.

Creative Commons (CC) is known for leading the licensing movement that allows creators to share their work while retaining copyright. In July, the organization Announce A plan to provide a legal and technical framework for sharing data sets between companies that control the data and AI providers who want to train on it.

Now, the nonprofit tentatively supports pay-for-crawl systems, saying it is “cautiously supportive.”

“Implemented responsibly, pay-for-crawl can be a way for websites to continue creating and sharing their content, manage alternative uses, and keep content publicly available where it might not be shared or might hide behind a more restrictive paywall.” Said blog post.

led by Companies like Cloudflarethe idea behind pay to crawl is to charge AI bots a fee every time they scan a site to collect its content for model training and updates.

In the past, websites freely allowed web crawlers to index their content for inclusion in search engines like Google. They benefited from this ranking by seeing their sites listed in search results, attracting visitors and clicks. But with AI technology, the dynamic has changed. After a consumer gets their answer via an AI-powered chatbot, they are less likely to click through to the source.

This transformation has already taken place Devastating for publishers By killing search traffic, it shows no sign of stopping.

On the other hand, a pay-for-crawl system can help publishers recover from the hit that AI has taken on their bottom line. Additionally, it can work best for smaller web publishers who don’t have the ability to negotiate one-time content deals with AI providers. Large deals have been concluded between companies such as OpenAI and Condé Nast, Axel Springer and others; And also between Confusion and Janet; Amazon and the New York Times; and Meta and various media publishers, within Others.

CC offered several caveats about its support of pay-for-crawling, noting that such systems could concentrate power on the web. It would also potentially deny access to content for “researchers, non-profit organizations, cultural heritage institutions, educators and other actors working for the public good.”

It proposed a series of principles for responsible pay-for-crawling, including not making pay-for-crawling a default setting for all websites and avoiding blanket web rules. Additionally, she said pay-for-crawl systems should allow for restriction, not just blocking, and should keep access in the public interest. It should also be open, interoperable, and built with standardized components.

Cloudflare isn’t the only company investing in the pay-for-crawl space.

Microsoft is also building AI market for publishers, Smaller startups like ProRata.ai and TollBit are starting to do this as well. Another group is called RSL Collective Announce Its own specification for a new standard called Really Simple License (RSL) that would limit what parts of a website can be accessed by web crawlers but would stop short of actually blocking crawlers. Cloudflare, Akamai, and quickly It has since been adopted by RSL, which is… Supported By Yahoo, Ziff Davis, O’Reilly Media, and others.

CC was also among them Announce Its support for RSL, Next to CC signalsits broader project to develop the technology and tools needed for the age of artificial intelligence.