Using Cloudflare to Block AI Crawlers: Setup and Configuration Guide

AI crawlers scrape your content to train large language models and power AI search tools. Most of this happens without your permission, compensation, or meaningful traffic referral back to your site. For publishers looking to take control, our complete guide to blocking AI crawlers covers strategies for blocking, allowing, or optimizing crawler access for maximum revenue.

The scale is staggering. Cloudflare reports that GPTBot now surpasses traffic volumes from major traditional tech players like Amazon and ByteDance. OpenAI's crawl-to-referral ratio sits at approximately 1,700:1. Your content trains their models, but you see essentially zero traffic in return.

For publishers relying on ad revenue, every visitor that instead gets their answer from an AI chatbot represents lost pageviews and lost revenue. Cloudflare AI crawlers blocking gives you the power to decide who accesses your content and on what terms.

Need a Primer? Read this first:

• Complete Guide to Blocking AI Crawlers: Strategies for blocking, allowing, or optimizing AI crawler access for maximum revenue
• The Complete List of AI Crawlers: Deep dive into each AI bot, its operator, and specific blocking instructions

Understanding Cloudflare's AI Bot Protection Tiers

Cloudflare offers multiple layers of AI bot protection, from simple toggle switches to sophisticated custom rules. Your approach depends on your technical comfort level and how granular you need your control to be.

Bot Fight Mode: The Foundation

Bot Fight Mode represents Cloudflare's baseline bot protection, available on all plans. This feature challenges bots from cloud providers and known malicious sources.

One important limitation exists here. Bot Fight Mode cannot be customized or bypassed using WAF custom rules. For more granular control, Super Bot Fight Mode (available on Pro plans and above) offers additional flexibility.

The AI Scrapers and Crawlers Toggle

Cloudflare launched their "AIndependence" feature in July 2025, providing a one-click solution for blocking AI bots. This managed rule blocks verified AI crawlers and includes signatures of AI bots that don't follow rules or attempt to hide their behavior.

The toggle blocks the following major AI crawlers:

For a deeper dive into each of these bots and their behaviors, check out our complete list of AI crawlers with instructions for blocking each one.

The rule also catches unverified bots that behave similarly to known AI crawlers. Cloudflare continuously updates these signatures, so protection evolves without action on your part.

Enabling the AI Bot Block

The setup process takes less than a minute.

Log into your Cloudflare dashboard and select your domain. Navigate to Security, then Bots. Find the toggle labeled "AI Scrapers and Crawlers" and switch it on.

Cloudflare will immediately begin blocking AI crawlers matching their managed rule signatures. The feature updates automatically as Cloudflare identifies new AI bot patterns.

Related Content:

• How to Block AI Bots with robots.txt: Complete publisher-focused instructions for robots.txt configuration
• How to Block Google AI Overview: Prevent your content from appearing in AI-generated search summaries
• Ads.txt Guide: Ensure only authorized buyers can purchase your ad space

How to Create WAF Custom Rules for AI Bot Blocking

The one-click toggle works for most publishers, but some situations require more surgical precision. Maybe you want to block most AI crawlers but allow a specific bot for a partnership. Perhaps you need to protect certain content while leaving other sections accessible. WAF custom rules deliver this granular control.

Publishers seeking a comprehensive technical approach should review our step-by-step guide to blocking AI from scraping your website for additional implementation methods.

Building Your First AI Bot Custom Rule

Custom rules in Cloudflare's WAF use expressions that evaluate incoming requests against specific criteria. These expressions can reference bot scores, user agents, verified bot categories, and other request attributes.

A basic rule to block AI crawlers based on Cloudflare's verified bot category looks like this:

(cf.verified_bot_category in {"AI Crawler"})

This expression catches any request from a bot Cloudflare has verified as an AI crawler. The action you assign determines what happens: block, challenge, or log.

Advanced Rule Configurations

Publishers often need rules that combine multiple conditions. The following table shows common rule patterns:

Note that the bot score threshold rule (score less than 30) requires Bot Management, which is an Enterprise add-on. Free and Pro users can still use verified bot category and user agent rules effectively.

User Agent Blocking: A Complementary Approach

Cloudflare's User Agent Blocking tool provides another method for targeting specific bots. This tool sits under Security, then WAF, in the Tools tab. You can create blocking rules for specific user agent strings like "GPTBot/1.0" or "ClaudeBot."

The limitation here is that user agent blocking doesn't support wildcards. For AI bots that follow the rules and honestly identify themselves, this works fine. For bots that spoof their user agents, you'll need bot score-based rules.

Rule Priority and Execution Order

Cloudflare processes rules in a specific sequence. Custom rules execute before the AI bot blocking managed rule, meaning a custom "allow" rule lets requests through before the managed AI block can catch them.

The priority flows in this order: Custom Rules, Block AI Bots (managed rule), then Allow Verified Bots rules. Understanding this sequence helps you build rules that work together rather than against each other.

How to Monitor Blocked AI Crawler Requests

Blocking AI crawlers without visibility into the results leaves you flying blind. Cloudflare provides several dashboards for tracking blocked requests and analyzing bot traffic patterns.

Security Analytics Dashboard

Security Analytics displays information about all incoming HTTP requests, including those not handled by Cloudflare security products. Access it through Security, then Analytics in your Cloudflare dashboard.

The dashboard reveals several key insights for AI bot monitoring:

• Bot Analysis: Shows requests categorized as Automated, Likely Automated, Likely Human, or Verified Bot
• Request Rate Analysis: Displays traffic patterns and helps identify unusual bot activity spikes
• Top Statistics: Surfaces the most common IP addresses, user agents, and request characteristics

The Insights panel highlights traffic that might need attention. If you see a spike in automated traffic that isn't being mitigated, you can click through to investigate and potentially create new rules.

Security Events Dashboard

Security Events shows every action taken by Cloudflare's security products. Navigate to Security, then Events to access this view.

Requests challenged or blocked by Bot Fight Mode appear with "Bot Fight Mode" in the Service field. Filter options let you narrow down to specific time periods, actions, or rule sources.

Bot Analytics for Enterprise Users

Enterprise Bot Management customers gain access to Bot Analytics, a dedicated dashboard for bot traffic analysis. This view provides powerful capabilities including requests by bot score, score distribution charts, detection source identification, and detailed request attribute breakdowns.

Bot Analytics displays up to 72 hours of data at a time and retains data for 30 days. The real-time data helps you understand exactly which bots are hitting your site and how your rules perform.

Creating Custom Alerts

Proactive monitoring beats reactive firefighting. While Cloudflare doesn't offer native alerting for bot traffic thresholds, you can export logs to third-party SIEM tools using Logpush.

Enterprise customers can also use Log Explorer to query HTTP requests and security events directly within the dashboard. This gives you the visibility needed to catch issues before they impact your revenue.

Best Practices for Publisher AI Bot Management

These best practices come from real-world publisher experiences with AI bot blocking.

Start with Logging, Then Block

Before enabling blocking actions, run your rules in Log mode for 24 to 48 hours. Review the logged requests to confirm you're targeting the right traffic before switching to Block.

This approach prevents false positives from impacting legitimate traffic. You can always tighten restrictions after validating your configuration.

Protect Your SEO Crawlers

AI blocking rules can accidentally catch legitimate search engine crawlers if configured too broadly. Cloudflare's managed AI blocking rule excludes verified bots in "Search Engine" categories.

When building custom rules, include logic to exclude Googlebot and Bingbot. Your ad revenue depends on organic search traffic, so protecting these crawlers is essential. Speaking of protecting your inventory, understanding how ads.txt works helps ensure only authorized buyers can purchase your ad space.

Layer Your Defense Strategy

Cloudflare's blocking operates at the network edge, stopping bots before reaching your server. Your robots.txt still matters for compliant bots, but many AI bots either don't check robots.txt or ignore its directives. For publishers who want to configure robots.txt alongside Cloudflare, our guide to blocking AI bots with robots.txt provides complete publisher-focused instructions.

Layer both approaches for maximum protection. Use robots.txt for well-behaved bots and Cloudflare blocking for those that ignore your preferences.

Monitor and Review Regularly

The AI crawler landscape evolves constantly. New bots emerge, existing bots change behavior, and Cloudflare updates detection signatures.

Schedule quarterly reviews of your blocking configuration. Check your analytics to ensure you're blocking what you intend and not missing new threats. Google's AI Overview feature deserves special attention—learn how to block Google AI Overview from using your content if you want to prevent your content from appearing in AI-generated search summaries.

Frequently Asked Questions About Cloudflare AI Crawler Blocking

What AI bots does Cloudflare block by default?

Cloudflare's AI Scrapers and Crawlers toggle blocks verified bots including GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Meta-ExternalAgent (Meta), Bytespider (ByteDance), Amazonbot (Amazon), and others. The rule also blocks unverified bots exhibiting similar behavior patterns.

Does blocking AI crawlers affect my Google search rankings?

No. Cloudflare's AI blocking rules specifically exclude verified search engine crawlers like Googlebot and Bingbot. Your SEO remains unaffected while AI training bots are blocked.

Can I allow specific AI crawlers while blocking others?

Yes. WAF custom rules let you create precise blocking logic. You can block most AI crawlers while allowing specific bots for partnerships or services that provide value through citations and referrals.

Is Cloudflare AI bot blocking available on free plans?

Yes. The one-click AI Scrapers and Crawlers toggle is available on all Cloudflare plans, including the free tier. Custom WAF rules require at least a Pro plan for advanced configurations.

How do I know if my AI blocking rules are working?

Use Security Analytics and Security Events dashboards to monitor blocked requests. Filter by service type or action to see specifically which AI bots have been blocked and how often.

Protecting the Traffic That Drives Revenue

For publishers focused on ad monetization, blocking AI crawlers connects directly to protecting revenue. Every pageview represents potential ad impressions. Cloudflare's AI blocking tools give you the ability to make AI companies negotiate for access to your content rather than taking it freely.

The imbalance between what AI companies take and what they return is stark. With crawl-to-referral ratios exceeding 1,000:1 for major AI operators, publishers subsidize AI development without compensation. Taking control of crawler access is one concrete step toward rebalancing this equation.

Once you've protected your traffic, the next step is maximizing what each visitor is worth. 

Next Steps:

• Programmatic Ad Monetization Platform Setup: Automate demand optimization after protecting your traffic
• Header Bidding Fundamentals: Capture maximum value from every impression
• Rewarded Video Ads: Deliver significantly higher CPMs than traditional display units

Maximizing Revenue from Protected Traffic with Playwire

Blocking AI crawlers preserves your traffic, but maximizing revenue from that traffic requires the right monetization partner. Playwire helps publishers extract maximum value from every pageview through advanced yield optimization, premium demand relationships, and proprietary AI and machine learning technology.

Our platform goes beyond basic ad serving. We analyze millions of data points to optimize every impression, connecting your protected content with the advertisers willing to pay premium rates.

Ready to turn your protected traffic into real revenue? Contact Playwire to learn how we can amplify your ad revenue while you focus on protecting and growing your audience.