IAB's CoMP Crawler Guidance: What Publishers Need to Know

Key Points

• IAB Tech Lab released new operational guidance on bot and crawler management to complement its Content Monetization Protocol (CoMP) API V1, helping publishers build formal strategies for non-human traffic.
• Blanket bot-blocking is no longer a viable default. The guidance explicitly frames graduated, standards-based controls as the baseline requirement.
• The CoMP framework enables contractual agreements between AI systems and content owners before content is crawled or used, a meaningful structural shift in how access gets negotiated.
• Publishers without a formal bot management strategy are already behind. This guidance exists precisely because most hadn't developed one after the CoMP V1 release.
• Public comment on CoMP API V1 is open through June 26, 2026. That window matters if you want a seat at the table.

What IAB Tech Lab Actually Released

According to MediaPost, IAB Tech Lab's CoMP Working Group published guidance designed to help publishers and content owners build strategies for managing non-human agentic traffic. This is operational guidance, not a technical standard. It addresses the business and risk evaluation layer that the CoMP API V1 didn't cover.

Hillary Slattery, senior director of product management and programmatic at IAB Tech Lab, put it plainly: "The API solved a technical standardization problem, but the industry also needed business and operational guidance around bot and crawler management." That gap became obvious once publishers started trying to actually implement CoMP V1.

The CoMP framework, the Content Monetization Protocol, is a structure for negotiating access and agreements between AI systems and content owners before crawling happens. Think of it as a handshake protocol for the pre-crawl conversation that currently doesn't exist for most publishers.

Why Blanket Bot-Blocking Is Now a Losing Strategy

The old instinct was simple: block everything you don't recognize. That made sense when most non-human traffic was spam bots and scrapers. The landscape has shifted considerably since then.

Dave Bellous, VP of strategy at Metal Toad, said it directly in the announcement: "Non-human traffic now dominates most of the open web, and AI agents and crawlers are steepening the curve. Blanket bot-blocking is no longer a useful strategy."

The guidance frames this around graduated, standards-based controls. Not all non-human traffic is equal. Some bots index your content for search. Some are AI training crawlers. Some are agentic systems that may actually send referral traffic or pay for access under a CoMP agreement. Treating all of them identically leaves value on the table while also failing to stop the bad actors.

The real question isn't "block or allow." It's "what terms do I set, and with whom?"

Essential Background Reading:

• AI Crawler Resource Center for Publishers: The full hub for publisher-focused AI crawler strategy, tools, and guidance in one place.
• What Publishers Need to Know About AI Bot Traffic in 2026: A foundational overview of how AI bot traffic works and what it means for publisher ad revenue.
• AI Bot Traffic Is Now 40% of the Web: The data behind non-human traffic growth and why publishers can no longer treat it as a fringe issue.
• AI Scraping vs. Traditional SEO Crawling: A clear breakdown of the difference between beneficial crawlers and AI scrapers. Essential context before building any blocking strategy.

What the Guidance Helps Publishers Evaluate

The CoMP Working Group built this guidance to walk through the full range of available approaches, tradeoffs included. It's specifically designed to help non-technical decision-makers, which matters, because most bot strategy conversations are happening in boardrooms, not engineering backlogs.

The guidance prompts publishers to evaluate four dimensions:

• Financial costs: What does it cost operationally to implement and maintain different levels of access control?
• Risk exposure: What are the downside scenarios for each approach, whether over-blocking beneficial traffic, under-blocking bad actors, or failing to monetize AI access entirely?
• Data value: What is your content actually worth to AI systems, and are you capturing any of that value today?
• Business alignment: Which approach fits your revenue model, your audience, and your relationship with AI platforms?

If you're running a content-heavy publisher site and your traffic analytics show a rising share of sessions with no monetizable user signal, the "do nothing" strategy already has a cost.

Related Content:

• Block AI: Playwire's tools and guidance for publishers who want to take direct action on AI crawler access.
• The Atlantic's AI Bot Blocking Strategy: A real-world example of how a major publisher approached formal AI crawler policy, and what others can take from it.
• How to Block AI Scrapers: The Legal Landscape: Covers the legal dimensions of blocking AI crawlers, a layer the CoMP guidance directly intersects with.
• Common Crawl Is an AI Training Pipeline: Why publishers need to understand Common Crawl's role in AI training before finalizing any access policy.
• Anthropic's $1.5B Copyright Settlement: The legal and financial stakes of AI content use, directly relevant to evaluating your content's economic value to AI systems.

Where CoMP Fits in Your Stack Right Now

CoMP V1 is still in public comment through June 26, 2026. The spec is live and available, but the ecosystem is still forming around it. Publishers who engage now, whether by commenting, building evaluation frameworks, or at minimum understanding the protocol, will be better positioned when adoption accelerates.

The future IAB is describing is one where bots negotiate access through content marketplaces rather than directly with individual publishers. That's a material change in how distribution and monetization relationships work. The infrastructure you build now, or don't build, affects your position in that future market structure.

For publishers already using tools like our AI Crawler Protection Grader or reviewing strategy through the AI crawler resource center, the CoMP guidance adds a useful business evaluation layer on top of the technical controls you may already have in place.

Next Steps:

• AI Crawler Protection Grader: Assess your current crawler exposure in minutes and see exactly where your strategy has gaps.
• AI Traffic Is the New SEO: Once you've secured your crawler policy, this guide covers how to optimize for AI referral traffic as a revenue opportunity.
• AI Search Is Eating Publisher Traffic: Practical steps for protecting and rebuilding traffic in an AI-dominated search environment.
• AI Citation Ranking Factors: How to position your content so AI systems cite it. Turning crawler access into a traffic and authority driver.

What Publishers Should Do Before June 26

The public comment window gives publishers a concrete near-term action. Here's how to prioritize:

• Review CoMP API V1: Understand what the protocol enables before the comment window closes. If the spec shapes your future revenue relationships, you should have a perspective on it.
• Audit your current bot strategy: Do you have a formal written approach? If not, the IAB guidance gives you the framework to build one. "We block what we don't recognize" is not a strategy anymore.
• Segment your non-human traffic: Not all bots carry the same risk or the same potential value. Start distinguishing between search indexers, AI training crawlers, and agentic traffic now. Understanding how AI scraping differs from traditional SEO crawling is a useful starting point for that segmentation.
• Evaluate your content's economic value to AI systems: This requires honest assessment. If AI companies are training on your content, what publishers can learn from Reuters' AI licensing strategy is worth reviewing. Is that relationship currently generating any revenue for you?
• Submit public comment if you have a stake: The spec is still being shaped. Publisher voices in that process matter more than most publishers realize.

See It In Action:

• AI Crawler Impact on Lifestyle Publisher Traffic: Real traffic data showing how AI crawlers are affecting publisher sessions and revenue. Not theory, not projections.
• Agency Revenue Drops Signal AI Search Traffic Shift: Industry-level evidence of how the AI traffic shift is already showing up in publisher revenue figures.
• Reuters' AI Licensing Strategy: A concrete example of a publisher monetizing AI access rather than simply blocking it. The commercial model CoMP is designed to enable.

How We Think About This

The CoMP guidance reflects a broader reality: the web's traffic mix has changed structurally, and the monetization frameworks publishers rely on were built for a human-traffic world.

We work with publishers across gaming, education, entertainment, and news who are already seeing the downstream effects of AI on their traffic and revenue patterns. The tools and strategies that maximize revenue from your real human audience matter more as that audience becomes a smaller share of total requests.

The infrastructure side is where we spend our time: making sure every legitimate impression from every human user is monetized as effectively as possible, while giving publishers the visibility and controls to make informed decisions about their non-human traffic policies.

If you want to understand where your crawler exposure stands today, our AI Crawler Protection Grader is a fast way to see it.