Cloudflare Launches A Bot-Scraping Paywall

Cloud-infrastructure provider Cloudflare is debuting a new marketplace that will allow publishers to paywall their sites, forcing AI firms to drop some cash each time they want to crawl them.

Why It Hits: Querying chatbots is killing the publishing industry, which relies on people actually visiting their sites in order to make money. Many publishers have responded by either suing AI firms or striking licensing agreements. But by utilizing the new tool from Cloudflare, publishers may no longer have to choose between the extremes.

Behind the Code: Cloudflare is instituting a new digital transaction called “Pay Per Crawl.”

• Currently in private beta, Pay Per Crawl allows publishers to set a rate that AI firms must pay each time they “crawl” — analyze their sites for answers or information.
• Publishers also have the ability to block all AI crawling or allow AI sites to crawl free of charge.
• The tool will also give publishers data on why these AI firms are crawling their sites — i.e., for training data, search responses, etc.
• To use the marketplace, both publishers and AI firms need to have a Cloudflare account. Both parties can negotiate how much they’d like to either buy or sell a crawl, with Cloudflare acting as the middleman.

Checkout: Up until a couple of years ago, the economy of search revolved around securing referrals from Google. For every 14 times Google scraped a site for search results, websites received a referral. Now, websites can expect only one referral for every 1,700 times that OpenAI crawls them, or one for every 73,000 times that Anthropic crawls them. That’s not sustainable for publishers. That’s why Cloudflare, which serves 20% of web traffic, is stepping in.

Prediction: With the rise of AI agents that visit websites on behalf of users, Cloudflare hopes it can help create a digital economy where those agents will need a budget to curate information across the internet. So, expect that the higher the budget, the better the results will be.