Cloudflare Draws a Line: AI Crawlers Must Pay Publishers by September

The News

Cloudflare, the infrastructure firm that sits in front of a large slice of the world's websites, has set a hard deadline of 15 September 2026 for artificial intelligence companies to stop treating publisher content as a free resource. From that date, so-called mixed-use crawlers, the ones that gather pages both for search indexing and for AI training or agent tasks, will be blocked by default from ad-supported pages unless a site owner chooses otherwise.

The change lands on new Cloudflare customers, on new sites added by existing customers, and on the company's entire base of free users. Chief executive Matthew Prince framed the shift as a response to a tipping point. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," he said. Cloudflare says bots have, for the first time, overtaken human visitors online, and that more than 50 per cent of AI crawler traffic simply re-fetches pages that have not changed.

Alongside the block, Cloudflare is rolling out a payment layer called Pay Per Use, an evolution of its earlier Pay Per Crawl marketplace. It lets publishers charge AI firms when their material is fetched and turned into commercial value. Ceramic.ai and You.com are named among the first partners.

Why It Matters

This is the clearest attempt yet to convert the open web's implicit bargain into an explicit invoice. For two decades the deal was simple: let search engines crawl your pages, and they send readers back to you. AI assistants broke that loop. They ingest the work, answer the question inside the chat window, and the traffic, along with the advertising revenue it carried, never arrives.

Cloudflare is trying to rebuild the fence and install a tollbooth in the same move. The company points to an uneven playing field, noting that Google offers a Google Extended toggle allowing sites to opt out of AI training while remaining in Search, an option most rivals do not extend. By forcing crawlers to declare their purpose, Cloudflare wants to end the practice of AI scrapers hiding behind search-friendly reputations.

The last comparable inflection was the arrival of ad-blocking at scale around 2015, which forced publishers to confront how fragile their revenue base had become. This is that reckoning repeated, only now the disputed asset is the text itself rather than the advertising wrapped around it.

Indian Angle

For Indian publishers, from legacy newspaper groups to the fast-growing crop of regional-language digital outlets, the timing is significant. Many depend heavily on search-driven traffic and thin programmatic advertising, a model already under strain. A default block on unpaid AI crawling hands them leverage they have never had, but only if they sit behind Cloudflare or a comparable provider and actively manage the settings. Smaller Indian newsrooms that lack technical teams risk being left with the protections switched off.

There is a home-grown dimension too. India's own foundation-model builders, including Sarvam and Krutrim, are training systems that lean on multilingual Indian web content. A pay-to-crawl regime raises their input costs in dollars at a moment when domestic AI budgets are far smaller than those of OpenAI or Google. Cheap access to Hindi, Tamil and Bengali text has been a quiet advantage for Indian startups; that advantage now carries a price tag.

Regulators are watching the same terrain. MeitY's ongoing work on AI governance and the Digital India framework has so far said little about content compensation, while debate over a possible link between AI training and copyright continues. Cloudflare's private tollbooth may end up setting a de facto standard before New Delhi writes a public one.

FAQ

When does the policy take effect?

The default blocking of mixed-use AI crawlers begins on 15 September 2026. It applies to new Cloudflare customers, newly added sites, and all existing free-tier customers, who must adjust settings if they want different behaviour.

Does this stop search engines from indexing my site?

No. The policy targets crawlers used for AI training and agents, not pure search indexing. Cloudflare's aim is to separate the two so publishers can stay discoverable in search while charging or blocking AI ingestion.

How can publishers get paid?

Through Cloudflare's Pay Per Use system, which evolved from the earlier Pay Per Crawl marketplace. It lets site owners charge AI companies when their content is fetched. Ceramic.ai and You.com are among the launch partners.

What does this mean for Indian AI startups?

Model builders such as Sarvam and Krutrim that rely on Indian-language web data could face higher, dollar-denominated data-sourcing costs, narrowing a cost advantage they have enjoyed against much larger global rivals.

Where can I read the original announcement?

The policy was reported by TechCrunch, linked in full below.

This story was reported by TechCrunch. Read the full original coverage at TechCrunch.