The Atlantic's AI Bot Blocking Strategy: What Publishers Need to Know

The Atlantic just revealed something every publisher should be paying attention to: a systematic approach to deciding which AI crawlers get access to their content and which get shown the door.

And their findings? Pretty telling. One AI crawler tried to hit their site 564,000 times in just seven days. That's not a typo.

The Core Problem

Here's the fundamental tension The Atlantic's CEO Nick Thompson nailed: most AI platforms drive almost zero traffic back to publishers. By design.

Think about that for a second. These crawlers are scraping your content to train their models, improve their outputs, and build products that compete with you, all while sending you essentially nothing in return. Not traffic. Not subscribers. Not revenue.

As Thompson put it, the amount of traffic publishers get from AI bots they've blocked is "de minimis." Translation: blocking them costs you almost nothing because they weren't giving you anything anyway.

The Atlantic's Scorecard Approach

Rather than taking a blanket "block everything" stance, The Atlantic built a rating system that tracks each AI crawler's actual value. Their weekly review process examines which bots send referral traffic and which lead to subscription conversions.

The math is straightforward: if an AI bot drove 1,000 subscribers at $80 each, that's $80,000 in revenue. Worth keeping around. If it drove one subscriber? That's a different conversation entirely.

This data-driven approach makes sense for premium publishers with the resources to track and analyze bot behavior at this level. But for most publishers, the technical overhead of monitoring dozens of AI crawlers and correlating that data with traffic and conversion metrics is substantial.

What This Means for Your Ad Revenue

Here's where this gets relevant for publishers focused on monetization: traffic you don't get is revenue you don't earn.

Every time an AI engine answers a user's question using your content without sending them to your site, that's a pageview that never happened. A session that never started. Ad impressions that never served. Revenue that never materialized.

The Atlantic's approach highlights an uncomfortable truth, you might be feeding the very systems that are reducing your traffic potential. And if your monetization strategy depends on session volume and engaged users, that's a problem worth solving.

The Google Complication

Thompson acknowledges the elephant in the room: Google. Publishers can block Google's dedicated AI crawler (Google-Extended), but Google's AI Overviews still pull from content indexed by the regular search crawler. Block one, and you risk your search visibility. Accept the other, and your content feeds AI summaries that may reduce click-throughs.

It's a Catch-22 that doesn't have a clean solution yet. The Atlantic is planning to use Cloudflare's new Content Signals Policy to communicate their preferences, but there's no guarantee Google will comply.

The Bottom Line

The Atlantic's strategy represents a shift from reactive bot-blocking to proactive value assessment. Instead of asking "should we block AI bots?" they're asking "what is each bot actually worth to us?"

For publishers, this reframes the conversation. Your content has value. AI companies know it, that's why they're scraping it. The question is whether you're getting fair value in return, or whether you're subsidizing competitors who give you nothing back.

The publishers who figure out how to protect their content while maximizing legitimate traffic sources will be the ones best positioned to maintain sustainable ad revenue. Because at the end of the day, you can't monetize traffic that never arrives.

Learn more with our extensive resource center around AI crawlers and blocking strategies.

Looking to maximize the traffic you do get? Learn how Playwire helps publishers amplify their ad revenue with AI-driven yield optimization and premium demand access.