Cloudflare's AI Bot Controls: What Publishers Need to Know

Key Points

• Cloudflare is now blocking mixed-use AI crawlers by default on ad-supported pages, shifting the burden from opt-in to opt-out.
• A new Pay Per Use model expands on the earlier Pay Per Crawl framework, aiming to compensate content creators based on how AI agents actually use their content.
• Early partnerships, including one with beehiiv, signal that this framework is being productized for publishers at multiple scales.
• Ad-supported publishers sit directly in the path of this change and need to understand what it means for their traffic, their content licensing position, and their revenue.
• The traffic you still control is worth more than ever. Monetizing it well is the real lever.

What Cloudflare Announced

According to Simply Wall St's reporting on the development, Cloudflare has rolled out new AI bot control classifications and analytics that give website owners more granular visibility into which AI crawlers are hitting their content. The headline change: Cloudflare is now blocking mixed-use AI crawlers by default on ad-supported pages unless site owners explicitly opt out.

That default flip matters. Previously, publishers had to actively configure blocking. Now they're configured to block unless they decide otherwise. Cloudflare is also expanding its Pay Per Crawl framework into a Pay Per Use model, where payment to content creators is tied to how AI agents actually use their content, not just whether they crawled it.

New commercial partnerships are forming around this framework. The beehiiv integration is the concrete example on the table right now, targeting the long tail of creators and newsletter publishers. Larger AI platforms are the implied target for the broader Pay Per Use model.

See It In Action:

• AI Crawler Protection Grader: Run a fast assessment of your site's current crawler exposure and protection status
• AI Crawler Impact on Lifestyle Publisher Traffic: Real traffic data showing how AI crawlers are affecting publisher sessions across a major vertical
• Agency Revenue Drops Signal AI Search Traffic Shift: Industry-level data on how AI search is reshaping the traffic and revenue picture for publishers

Why This Matters for Ad-Supported Publishers

This announcement lands squarely on ad-supported publishers. The default blocking behavior is specifically scoped to ad-supported pages. If your site runs display advertising, Cloudflare's infrastructure is now pushing toward blocking mixed-use AI crawlers on your content unless you change the setting.

The "mixed-use" classification is where the real complexity sits. These aren't pure scrapers and they aren't pure search crawlers. They're AI agents that do both: indexing content for training, for retrieval-augmented generation, for real-time summarization. Cloudflare is treating that dual-use behavior as distinct from a standard search bot and applying commercial logic to it. Publishers who want the full breakdown on how AI scraping differs from traditional SEO crawling should dig into that distinction before making any configuration changes.

For publishers on Cloudflare's network, the operational question is immediate. Do you leave the default blocking in place? Do you opt in to Pay Per Use and try to capture some revenue from AI access? Do you opt out entirely and let crawlers through as before?

Essential Background Reading:

• AI Crawler Resource Center for Publishers: The full technical reference on AI crawler classification, blocking options, and publisher strategy
• AI Bot Traffic Is Now 40% of the Web: Why the scale of AI crawler activity demands publisher attention right now
• AI Scraping vs. Traditional SEO Crawling: The foundational distinction between search bots and AI scrapers that every blocking decision depends on
• What Publishers Need to Know About AI Bot Traffic in 2026: A current-year guide to how AI bot traffic is evolving and what publishers should monitor

What Publishers Should Do

There's no universal right answer here, but there is a useful decision framework. These are the questions worth working through now.

Your content's value to AI systems matters: if your site produces high-quality, original editorial content, it has meaningful training and retrieval value. That changes the calculus on whether Pay Per Use is worth engaging with versus simply blocking.

Check your Cloudflare configuration: go verify what your current AI bot control settings look like. The default change to blocking on ad-supported pages may already be active depending on your plan and configuration. Don't assume you know what's running.

Understand what "mixed-use" means for your traffic: not all AI crawlers are classified the same way. Pure search indexers from established engines are treated differently from mixed-use crawlers. Blocking AI training crawlers indiscriminately may affect your AI referral traffic in ways you don't intend.

Consider the Pay Per Use model carefully: the commercial framework is new and adoption is unproven. The beehiiv partnership shows the concept has legs for smaller publishers, but the revenue mechanics, payment reliability, and adoption rates among AI companies are still developing. Engaging early gives you a read on whether it's viable for your situation.

The core tension here is the same one publishers have been navigating since AI Overviews started eating into organic search traffic. Block, license, or do neither. Each has tradeoffs that depend on your content type, traffic profile, and revenue model.

Related Content:

• How to Block AI Scrapers: The Legal Landscape Publishers Need to Know: The legal framework behind blocking decisions, from robots.txt to litigation exposure
• The Atlantic's AI Bot Blocking Strategy: How a major publisher approached crawler blocking and what smaller publishers can take from it
• Common Crawl Is an AI Training Pipeline: Why the industry's most widely used open dataset is a direct feed into AI training and what publishers should know
• Anthropic's $1.5B Copyright Settlement: The legal precedent forming around AI training on publisher content and what it signals for compensation frameworks
• Block AI: Playwire's direct resource for publishers evaluating AI crawler blocking options

The Broader Shift Cloudflare Is Making

Cloudflare is positioning itself as a policy and payment layer between content owners and AI companies, not just an infrastructure provider. That's a meaningful strategic move. Traditional CDN and security players compete on performance and uptime. Cloudflare is adding content licensing governance to that stack.

Whether this evolves into a real revenue model for publishers or stays primarily a positioning play for Cloudflare is still an open question. Adoption by AI platforms is the variable that determines the outcome. If large AI companies resist the Pay Per Use model and find routing workarounds, the framework loses leverage fast.

What is clear: the default blocking change on ad-supported pages is real and active. Publishers on Cloudflare's network are already affected whether they know it or not. Publishers organizing collectively against AI scraping are arriving at a similar conclusion from a different angle: the infrastructure layer is where the leverage is.

Next Steps:

• AI Traffic Is the New SEO: How Publishers Can Optimize for AI Referrals: If you're not blocking everything, this is how to position your content for AI-driven discovery
• AI Citation Ranking Factors: What Publishers Need to Know Now: The signals that influence whether AI systems cite your content, and how to improve them
• Google's AI Search Updates Give Publishers a Real Shot at More Clicks: How recent Google changes open new traffic opportunities for publishers who know where to look
• AI and Publishers Resource Center: The full hub for Playwire's guidance on navigating AI's impact on publisher traffic, revenue, and strategy

Maximize the Traffic You Still Have

Blocking AI crawlers is a defensible choice for some publishers. It doesn't solve the underlying revenue equation. The sessions you're still generating from real human users are where your ad revenue lives, and those sessions need to work harder.

Publishers seeing RPS pressure from declining organic traffic driven by AI search need to look at yield architecture, not just traffic acquisition. That means ad layout optimization, demand path quality, and header bidding configuration that actually reflects current market conditions. Viewability targets in the 70–90% range attract premium programmatic demand. Leaving money on the table with the audience you already have is the most fixable problem in the stack.

Our RAMP platform is built specifically for that problem. Whether you're running a managed service or want direct control through self-service, the goal is the same: more revenue from every session, with full transparency into what's driving results.

If you want to pressure-test where your current setup stands, our AI Crawler Protection Grader is a fast way to see your crawler exposure. And our AI crawler resource center has the technical detail you need to make an informed call on blocking strategy.

The Cloudflare announcement changes the infrastructure layer. Your monetization strategy is still yours to control.