The flood of AI crawlers and automated agents is forcing publishers and content owners to face a challenger: Deciding who gets access to their content, and under what terms.
To help address that problem, IAB Tech Lab released new guidance today on bot and crawler management strategies, opening the document for public comment through June 26, 2026.
The guidance is designed to help publishers, content owners, and ad tech companies think through how they manage non-human traffic, including AI systems scraping or accessing content. It also complements the organization’s recently released CoMP API V1, a framework intended to support communication and permissions between AI systems and publishers.
“Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way,” Anthony Katsur, CEO of IAB Tech Lab, said in a statement. “This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.”
Should you give AI access to your content?
The IAB paper underlines how quickly AI crawling became a business issue across publishing and advertising. Many organizations don’t have formal policies on bots and crawlers, despite growing concerns about AI systems consuming content, straining infrastructure, and complicating monetization.
According to IAB Tech Lab, the guidance was created because many publishers and content owners were struggling to navigate the trade-offs involved in managing AI access. Blanket bot blocking, the group argues, is no longer practical as AI systems become more embedded across the web ecosystem.
Instead, the document outlines a range of approaches companies can take, along with the operational, financial, and strategic implications of each.
The SEO toolkit you know, plus the AI visibility data you need.
The guidance is aimed less at engineers and more at business leaders trying to understand the costs and consequences of AI crawler management.
“Content owners are being asked to make important decisions quickly, often without clear frameworks,” said Shailley Singh, EVP and COO at IAB Tech Lab. “This guidance helps break down the options so they can choose an approach that aligns with their goals.”
IAB Tech Lab said it will continue working with members and industry participants to support the implementation of both the bot management guidance and the CoMP API. The organization is accepting public comments on the proposal through June 26 before finalizing the framework.
Go here to read and comment on the guidance.
The post IAB Tech Lab tackles the growing AI bot problem appeared first on MarTech.