Cloudflare has released to general availability of AI Crawl Control, a novel feature that provides website owners with enhanced authority over the interactions of AI crawlers. The initiative follows the company's prior decision in July to default to blocking AI bots unless explicitly permitted. This all in response to reports of extensive scraping by AI companies without proper attribution or compensation.
With the new AI Crawl Control, site owners now have the ability to issue a customized HTTP 402 "Payment Required" response when AI crawlers scrape content. Rather than being simply blocked, crawlers are redirected to a publisher's preferred licensing and/or API page. This effectively transforms what was previously unauthorized scraping into a potential business dialogue. Cloudflare reports that it is currently processing over one billion 402 responses daily across its network! Whoa bubba!
This update builds upon Cloudflare's AI Audit tool, which enabled publishers to identify which AI bots were accessing their domains. Now paid Cloudflare customers can establish detailed policies regarding which bots are permitted, which are blocked, and which receive the 402 "pony up" message. The company also intends to enhance the metadata provided with 402 responses, including content freshness and licensing conditions.
For SEOs and site owners, this tool signifies a transition from an "all or nothing" approach to AI bots. Rather than outright blocking, AI Crawl Control empowers publishers to define terms for engagement—potentially converting crawlers into collaborators, rather than mere opportunists.
As the CEO and founder of Pubcon and WebmasterWorld., Brett Tabke has been instrumental in shaping the landscape of online marketing and search engine optimization. His journey in the computer industry has spanned over three decades and has made him a pioneering force behind digital evolution. Full Bio
Visit Pubcon.com