Cloudflare's Default AI Bot Block Is Killing Your GEO Strategy

Key Points

• Cloudflare enables "Block AI bots" by default on all plans, meaning millions of websites are blocking AI crawlers without ever making a deliberate choice to do so.
• This setting operates at the infrastructure level, before robots.txt is ever read, so even a perfectly configured robots.txt file won't save you.
• Affected crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-CloudVertexBot, and a dozen others.
• The fix takes about 90 seconds in your Cloudflare dashboard.
• If your GEO efforts aren't gaining traction, check your infrastructure before blaming your content strategy.

What Happened

Independent testing by AEOGeoAI, reported via MSN, confirmed that Cloudflare activates its "Block AI bots" feature on every new domain before the site owner has touched a single setting. The test was simple: add a brand new domain, observe the default configuration. The block was already on.

Cloudflare powers a significant portion of the web's infrastructure. That makes this a widespread, silent problem affecting publishers and businesses who have no idea it's happening.

Why This Matters for Publishers

Most publishers focused on AI visibility have been working at the content layer: structured data, citation-worthy writing, topical authority. Those efforts are worthless if AI crawlers are being turned away at the door.

The particularly frustrating part is the robots.txt blindspot. Cloudflare's block operates upstream, at the infrastructure level. A crawler like GPTBot never even reaches the server. It never reads your robots.txt. The explicit permissions you set there are completely irrelevant when this setting is active.

According to AEOGeoAI's research, an invisible infrastructure block is one of the most commonly overlooked causes of low AI visibility scores. A brand can have solid content, strong third-party coverage, and a clean robots.txt, and still score zero across AI models.

That's not a content problem. That's a plumbing problem.

Essential Background Reading:

• AI Crawler Resource Center for Publishers: The full framework for understanding AI crawlers, what they do, and how to manage them across your infrastructure.
• AI and Publishers Resource Center: A broader look at how AI is reshaping publisher strategy, traffic, and monetization.
• AI Content Info: What publishers need to understand about how AI platforms use, index, and cite content.

Which AI Crawlers Are Affected

The block isn't selective. When Cloudflare's "Block AI bots" is enabled, it stops all of the major verified AI crawlers, plus unverified bots that exhibit similar behavior. Here's what gets blocked:

If you're investing in visibility across any of these platforms and your Cloudflare settings are untouched, none of that investment is reaching its destination.

Related Content:

• Block AI: Playwire's dedicated resource for publishers evaluating whether and how to block AI crawlers from their properties.
• Blocking Strategy: How to build a deliberate blocking approach that protects revenue without creating unintended gaps in visibility.
• Why Build a Blocking Strategy: The business reasoning behind making blocking decisions intentionally rather than by default.
• Generative AI: How generative AI platforms are changing the publisher landscape and what it means for content strategy.

How to Fix It

The fix is fast. The diagnosis is the hard part, because there's no obvious signal that anything is wrong. Your site loads fine for users. Your SEO crawlers get through. You just don't appear in AI-generated answers.

Here's how to check and resolve the issue:

• Step 1: Log into your Cloudflare dashboard.
• Step 2: Navigate to Security Settings.
• Step 3: Filter by Bot traffic.
• Step 4: Find "Block AI bots."
• Step 5: Set it to "Do not block (off)." The change takes effect immediately.

For publishers who want more granular control rather than a blanket toggle, Cloudflare's AI Crawl Control lets you manage individual crawlers separately. You can allow GPTBot while still blocking less reputable bots, or do the reverse, depending on your strategy.

Next Steps:

• AI Crawler Protection Grader: See exactly how your site is configured across major AI crawlers right now, before you make any changes.
• Manage Blocking: Hands-on guidance for managing your crawler and bot blocking configuration at the yield layer.
• Building a Blocking Strategy: Step-by-step guidance on constructing a crawler access policy that matches your business goals.
• The Quality vs. Quantity Revolution: Why ad load and traffic stability matter more than raw volume when building a durable monetization strategy.

The Broader Decision: Block, Allow, or Selectively Permit

Unblocking AI crawlers isn't the only valid choice. Some publishers have made a deliberate decision to block AI bots entirely, whether for content licensing reasons, concerns about training data use, or audience protection. That's a legitimate business call.

The problem is when the block is invisible. When Cloudflare makes that choice for you by default, you're not exercising a strategy. You're operating blind.

Publishers who want to appear in AI-generated answers need to actively opt into that access. Publishers who want to block AI crawlers should make that choice deliberately, with a clear understanding of the trade-offs of blocking AI traffic and the revenue impact. The worst outcome is the one most publishers are currently in: an active block with no idea it exists.

Our AI Crawler Protection Grader can show you exactly where your site stands across major AI crawlers. Our AI crawler resource center walks through the full decision framework, including what to allow, what to block, and how to think about the revenue implications of each.

See It In Action:

• AI Info: How Playwire approaches AI-era publisher challenges across crawlers, traffic, and monetization infrastructure.
• Publisher Ad Revenue Maturity Model Guide: Understand where your monetization setup stands today and what the next level of revenue performance actually looks like.
• Independent Web Publishers: How Playwire supports independent publishers navigating infrastructure, monetization, and the AI transition.

Make the Most of the Traffic You Have

Sorting out AI crawler access is step one. Step two is making sure the traffic you do receive, from search, direct, social, and AI referrals, is working as hard as possible for you.

Publishers investing in GEO and content discoverability are building for the next phase of traffic acquisition. The yield infrastructure underneath that traffic needs to be just as dialed in. An AI-cited article that drives a reader to a poorly monetized page is a missed opportunity.

To audit your full monetization setup alongside your crawler strategy, talk to our team. We'll tell you what we find, not what you want to hear.