The
IAB Tech Lab has released new guidance on AI system bot-and-crawler management strategies to help publishers and content owners develop approaches that inform non-human agentic agents.
The CoMP Working Group developed this guidance to highlight all available approaches -- the advantages and tradeoffs -- of each strategy.
The goal is to give
companies across the advertising ecosystem a clearer understanding of how to manage and give access to non-human traffic on their websites and to provide a way to help companies determine the type of
access they will permit that aligns with their business goals.
advertisement
advertisement
“The API solved a technical standardization problem, but the industry also needed business and operational guidance around
bot and crawler management," Hillary Slattery, senior director of product management and programmatic at IAB Tech Lab, told MediaPost. "That became more apparent as adoption discussions began after
the V1 release.”
The guidance will help executives evaluate the financial costs, operational risks and value of data by evaluating whether to allow or block non-human traffic.
These guidelines also should also guide companies, helping them to make more informed decisions when considering the adoption of the CoMP API V1 -- the IAB Tech Lab Content Monetization Protocol, a
framework that enables communication and contractual agreements between AI systems and content owners before content is crawled or used.
Following the CoMP API V1 release, it became clear that
many content owners and publishers had not developed formal strategies for their approach to bots and crawlers.
In the future, bots are expected to have agreements with content marketplaces
rather than content owners.
The CoMP Working Group addressed this gap by developing a range of approaches to provide companies with a clear understanding of how to manage access to their company's
content from all non-human traffic.
These guidelines help non-technical decision-makers navigate complex, resource-intensive AI operations.
By providing key questions, the text helps
leaders assess the costs and value of each strategy, enabling organizations to make better decisions, reduce risks, and lower operating costs.
Ultimately, these strategies foster vital
partnerships between content owners and AI systems that can build a more sustainable content marketplace.
“Non-human traffic now dominates most of the open web, and AI agents and
crawlers are steepening the curve," said Dave Bellous, vice president of strategy, Metal Toad. "Blanket bot-blocking is no longer a useful strategy. For publishers, advertisers, and ad-tech alike,
distinguishing beneficial automation from invalid traffic through graduated, standards-based controls has become a baseline requirement.”
The CoMP API V1 document is open for public
comment through June 26, 2026.