Cloudflare Splits AI Crawlers From Search Bots With New 'Mixed-Use' Controls

Cloudflare has published a policy proposal that would let website owners separate two jobs that have been bundled together since generative AI tools began scraping the open web: indexing pages for search results and feeding those same pages into model training. Under the plan, a crawler would have to declare which purpose applies, and a site could answer differently for each — staying visible in search while refusing the training use of its content. The company calls the category "accountable mixed-use AI crawlers," and it is aimed squarely at the operators whose bots do both.

The problem Cloudflare is addressing is structural, not technical. Search crawlers and training crawlers frequently arrive from the same companies, and most publishers depend on search referral traffic to survive, which leaves them with almost no room to block a bot outright. A blanket refusal in robots.txt takes down the indexing along with the training, killing the traffic that pays for the site. Cloudflare's answer is to split the mix into distinct, declared purposes at the network level, so permission granted for one use does not automatically carry over to the other.

Accountability is the load-bearing term in the proposal. A crawler seeking access for training would have to identify itself and accept terms tied to that specific use, rather than inheriting whatever access it was granted for search. That distinction matters because robots.txt has no vocabulary for intent — a directive that blocks a bot blocks it everywhere. Cloudflare is pitching declared purpose as a way to give site owners a finer control than the all-or-nothing switch they have now.

Cloudflare occupies an unusual position to push this. It fronts a large share of the web through its CDN and bot-management products, so a policy it enforces at the edge reaches sites that would never negotiate directly with model developers. The company has spent the past two years shipping tools that let customers observe and charge AI crawlers, and this proposal extends that posture from monetization toward permission design. Whether that leverage translates into adoption is a separate question from whether the mechanism works.

For publishers, the practical test is whether crawler operators comply voluntarily or whether blocking becomes the default response. If major AI developers adopt the declared-purpose model, the split could harden into an industry norm; if they ignore it, site owners are back to blanket blocks and the search traffic losses that come with them. Cloudflare has not said which crawlers have signed on, how it will verify a declared training purpose, or what happens to a bot that misrepresents its intent — and those details will determine whether mixed-use crawling becomes a workable middle ground or just another signal that gets ignored.

The enforcement gap is the part worth watching. A declaration is only as useful as the penalty behind it, and Cloudflare has not yet described what happens when a crawler claims search-only access and trains anyway. Until that is settled, the proposal reads as a framework for negotiation rather than a finished control.