IAB's AI Crawler Guidance: What Publishers Need to Do Now

Key Points

• IAB Tech Lab has released new guidance on bot and crawler management strategies, open for public comment until June 25, 2026.
• The guidance is aimed at non-technical leadership teams evaluating how to handle AI systems and non-human traffic.
• Blanket bot-blocking is no longer a viable strategy. Publishers need graduated, standards-based controls that distinguish beneficial crawlers from extractive ones.
• The guidance connects directly to the CoMP API V1 framework, which revealed most publishers have no formal bot management strategy at all.
• Publishers who get this right protect their content, improve traffic quality, and put themselves in a stronger position to negotiate value from AI systems.

What Happened

IAB Tech Lab has released new guidance on bot and crawler management, and it's now open for public comment through June 25, 2026. According to bestmediainfo.com, the guidance was developed by the CoMP Working Group after it became clear that many publishers lacked formal strategies for managing non-human traffic, a gap the CoMP API V1 itself couldn't fill.

The document outlines a range of management approaches, each with trade-offs, and is explicitly designed for non-technical leadership teams. IAB Tech Lab CEO Anthony Katsur put it plainly: "This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace."

See It In Action:

• The Atlantic's AI Bot Blocking Strategy: How a major publisher approached AI bot blocking and what smaller publishers can take from their playbook.
• AI Crawler Impact on Lifestyle Publisher Traffic: Real data on how AI crawlers are affecting traffic patterns for lifestyle publishers.
• Agency Revenue Drops Signal AI Search Traffic Shift: What declining agency revenue signals tell us about the broader AI-driven shift in publisher traffic.

Why This Matters to Publishers

The CoMP API V1 release exposed something uncomfortable. Most publishers haven't thought seriously about bot and crawler management as a business strategy. They've treated it as a background technical task, and the IAB Tech Lab guidance is a signal that posture is no longer acceptable.

Scott Messer, Principal and Founder of Messer Media, described it accurately: "The AI revolution has caught many publishers with their front doors wide open, essentially leaving the lights on for bots and crawlers to pilfer intellectual property." Non-human traffic now represents a meaningful and growing share of web activity, and AI crawlers are accelerating the curve.

Dave Bellous, VP of Strategy at Metal Toad, made a point worth sitting with: "Blanket bot-blocking is no longer a useful strategy." The problem isn't that crawlers exist. Most publishers simply can't tell the difference between a crawler that drives value and one that drains resources without giving anything back.

The IAB guidance introduces a framework for making that distinction. It's not a technical spec for developers. It's a decision framework for the people who own the business strategy, and that's exactly where this conversation needs to happen.

Essential Background Reading:

• AI Crawler Resource Center for Publishers: Everything publishers need to understand about AI crawlers, blocking strategies, and content protection in one place.
• AI and Publishers Resource Center: Comprehensive guidance on how AI is reshaping the publishing landscape and what publishers can do about it.
• AI Info for Publishers: Core concepts and context publishers need before evaluating any AI crawler management strategy.

What Publishers Should Do

Publishers who haven't established formal bot and crawler policies should treat this guidance as a starting point, not a checklist. The goal is a strategy that fits your specific business, not a one-size answer applied to every crawler that hits your server.

Here's a practical way to approach it:

• Audit your current exposure first: Understand which crawlers are accessing your content, at what volume, and what they're doing with it. You can't make good decisions without that baseline.
• Distinguish allies from extractors: Not all non-human traffic is the same. Search engine crawlers, legitimate AI licensing partners, and scraper bots all behave differently. Your policy should reflect those differences.
• Define your threshold by content type: High-value original content warrants stricter controls. Reference content or evergreen material may benefit from being accessible to AI systems that drive citations or syndication value.
• Engage the public comment period: IAB Tech Lab is actively collecting industry feedback through June 25, 2026. If you have a point of view, submit it. These standards affect your business and you have standing to influence them.
• Align your technical and leadership teams: This guidance was written for non-technical leaders. Make sure your ad ops, dev, and business strategy teams are working from the same framework.

The comment period is open now. Publishers who participate help shape what this standard actually looks like in practice.

Related Content:

• Block AI Crawlers: A practical overview of AI blocking options, trade-offs, and implementation approaches for publishers.
• AI Content Info: What publishers need to know about how AI systems consume and use their content.
• Generative AI and Publishers: How generative AI systems interact with publisher content and what it means for revenue and rights.
• Ad Load and Traffic Stability: Why traffic quality and stability matter more than raw volume for publisher monetization performance.

The Revenue Connection

Bot management doesn't exist in a vacuum. Publishers managing large volumes of non-human traffic face a direct impact on their monetization signals. Inflated crawl traffic distorts session-level data, muddies audience signals, and can affect the quality metrics that demand partners use to evaluate inventory.

Cleaning up non-human traffic isn't just content protection. It's yield hygiene. Better traffic quality translates to cleaner data, which translates to better RPS performance across your programmatic stack.

Next Steps:

• AI Crawler Protection Grader: Get a concrete assessment of your site's current crawler exposure and where your defenses stand.
• AI Training vs. AI Search Crawlers: Understand the critical distinction between crawlers you should block and those that could drive referral traffic to your site.
• The AI Search Reckoning: Why publishers need a deliberate strategy for the AI search shift rather than reactive blocking decisions.
• AI Traffic Is the New SEO: How to optimize for AI referral traffic and turn crawler activity into a traffic and revenue opportunity.

How We Think About This

We've been watching the AI crawler conversation closely, and we've built tools to help publishers assess where they stand. Our AI Crawler Protection Grader gives publishers a concrete read on their current exposure. Our AI Crawler Resource Center pulls together the practical guidance publishers need to make informed decisions, without the panic spiral.

The IAB Tech Lab guidance reinforces what we've been saying: this is a business strategy decision, not a firewall configuration. Publishers who treat it that way will be better positioned to protect their content, maintain traffic quality, and keep their monetization signals clean.

Quality, Performance, Transparency. All three get harder when your traffic data is polluted by crawlers you haven't thought carefully about.