AI Crawler Strategy: Blocking Isn't Always the Answer

Key Points

• Man of Many welcomes 1.3 million AI crawler visits per week and has seen organic search traffic climb 49% since February, suggesting that open access to crawlers can coexist with search growth.
• AI referral traffic is still negligible: just 0.3% of total sessions in July, with Google organic and direct traffic still accounting for roughly 70% of the publisher's audience.
• The "AI traffic is higher intent" narrative doesn't hold up in Man of Many's data. Google organic visitors had a 22% higher engagement rate and spent 43% longer on site.
• The publisher's organic search gains are tied directly to a hard ban on AI-generated content. Human-reported, first-hand editorial is what's outperforming.
• Publishers don't have a single correct answer here. The right call depends on your content model, your audience, and what you're actually trying to protect.

What Happened

B&T's report on Man of Many's AI crawler strategy is making rounds this week, and it deserves a close read. Ahead of and following his appearance at the IAB Australia Discovery: AI & Search Summit, Man of Many co-founder Scott Purcell shared traffic data that complicates the blocking conversation considerably.

The publisher has opened its doors to AI crawlers from Amazon, ChatGPT, Gemini, Claude, Perplexity, and Apple. Crawler traffic now sits at 1.3 million visits per week, tracked through Cloudflare and Tollbit. Referrals from Claude and Gemini have surged 594% and 539% respectively. Organic search is up 49% since February.

Purcell's argument is straightforward: blocking removes your content from AI surfaces without any compensating benefit. "You're choosing for your content not to be visible on those surfaces, and there's no benefit that comes from doing so," he told B&T.

See It In Action:

• Entertainment Content Website Case Study: How an entertainment publisher maximized ad revenue and audience yield through full-stack monetization optimization.
• Entertainment Website Ad Revenue Guide: A practical guide to driving stronger ad revenue from entertainment audiences, including format strategy and yield optimization.
• Lifestyle, Health & Travel Ad Revenue Resource Center: Vertical-specific monetization resources for publishers in lifestyle categories, directly relevant to Man of Many's audience type.

Why This Data Is Worth Taking Seriously

The crawler-welcome strategy is a legitimate business decision, and Purcell's numbers give it credibility. But the data cuts both ways, and publishers should read the full picture before drawing conclusions.

AI referrals are still tiny. Despite 1.3 million crawler visits weekly, LLM referrals accounted for 0.3% of total sessions in July. Google organic and direct traffic still drive roughly 70% of Man of Many's audience. The crawler activity is real. The downstream traffic impact is not, yet.

The engagement data is the most interesting finding here. Man of Many's data shows Google organic visitors had a 22% higher engagement rate and spent 43% longer on site than visitors from AI platforms. Purcell said it plainly: "The industry line that AI traffic is higher intent does not hold in our data."

ChatGPT was the one exception worth noting. Users who didn't bounce immediately viewed 15% more pages per session than Google organic visitors. Kagi, Claude, and Gemini also sent better-qualified visitors than ChatGPT overall, though each represented less than 0.05% of total traffic. These are signal, not scale.

Essential Background Reading:

• AI Crawler Resource Center for Publishers: Everything publishers need to understand about AI crawlers, blocking options, and strategic decisions in one place.
• AI and Publishers Resource Center: Broader context on how AI is reshaping the publisher landscape, with practical guidance on what to do about it.
• Generative AI for Publishers: An overview of how generative AI intersects with publisher content, monetization, and audience strategy.
• AI and Ad Tech: What Publishers Should Know: Foundational context on how AI is changing the ad tech ecosystem and what that means for publisher revenue.

The Organic Search Piece Is the Real Story

The traffic and engagement numbers are interesting context. The more important signal is the organic search growth.

Man of Many has grown organic search traffic while much of the publishing industry has moved the other direction. Purcell attributes this directly to one policy: a hard ban on publishing AI-generated content. The team uses AI in research and preparation, but every article that goes live is written by human journalists.

"One of the key contributors to that is we've banned the publishing of AI content to our sites," Purcell said at the IAB Summit.

The implication is clear. Publishers flooding their CMS with AI-written content to chase volume are likely contributing to their own search declines. Man of Many's first-hand product testing, original photography, and expert editorial are exactly what Google is rewarding right now, and what AI platforms cite as trusted sources.

Related Content:

• Block AI Crawlers: Publisher Options Explained: A practical breakdown of what blocking AI crawlers actually does, what it doesn't do, and how to implement it.
• Blocking Strategy for Publishers: How to think about blocking decisions holistically across your ad stack and content access strategy.
• Answer Engine Optimization Is the New SEO (ish): How the shift from search queries to AI-generated answers is changing the optimization playbook for publishers.
• The Quality vs. Quantity Revolution: Why ad load and traffic stability matter more than volume, and how quality content strategy drives better yield outcomes.

How to Think About Your Own Blocking Decision

This story doesn't settle the debate. It adds a well-documented data point to one side of it.

The blocking question is genuinely situational. A few factors worth mapping against your own situation:

• Your content model matters: Purcell's editorial approach produces the kind of trusted, first-hand content AI platforms want to cite. If your content is more commodity or easily replicated, your calculus may differ.
• Your licensing position matters: Large publishers like Nine and News Corp are negotiating commercial deals with AI platforms. If you have the leverage for that conversation, blocking is a negotiating tool. If you don't, it may just be invisibility.
• Your traffic sources matter: If AI referrals are already 5-10% of your sessions, you have a real dependency question. If they're 0.3% like Man of Many, the blocking decision is less consequential either way.
• Your content protection matters: Blocking crawlers doesn't stop scraping. It signals intent and may influence training data inclusion, but it's not a technical lock.

The Guardian's Zoe Featherstone, also on the IAB panel, offered a complementary perspective: trusted journalism becomes more valuable during major news moments, not less. Direct relationships through apps, newsletters, and podcasts are where publishers can hold their audience regardless of how AI search evolves.

Both positions are coherent. Neither is obviously wrong.

Next Steps:

• AI Crawler Protection Grader: Run your current robots.txt and crawler configuration through this tool to see exactly where your protection gaps are.
• Building a Blocking Strategy: Step-by-step guidance on constructing a crawler and ad blocking strategy that fits your content model and revenue goals.
• AI Content and Publisher Rights: What publishers need to understand about how AI platforms use your content and what options you have to manage it.
• Manage Blocking in Your Ad Stack: Practical controls for managing blocking decisions within your ad yield management setup.

Maximize the Traffic You Have

Wherever you land on the blocking question, one thing isn't situational: the traffic you do have needs to work as hard as possible.

Man of Many's data shows that most AI referral traffic underperforms Google organic. That's an argument for optimizing your core traffic sources, not chasing AI referral volume that hasn't materialized at scale.

For publishers serious about yield, the session-level revenue question matters more than the crawler access question right now. RPS from your Google organic audience, your direct traffic, your newsletter readers: that's where the money is today.

We built our RAMP platform specifically to maximize revenue from the audiences publishers already have. If your ad stack isn't keeping pace with the quality of your traffic, that's a gap worth closing before the AI referral question resolves itself.

Check our AI Crawler Resource Center if you're still working through the blocking decision. Or run your current configuration through our AI Crawler Protection Grader to see where you stand.

The crawler debate will keep evolving. Your revenue optimization shouldn't wait for it.