Cloudflare is rolling out granular controls for AI bot traffic, replacing its previous all-or-nothing blocking approach. Site owners can now manage three distinct bot categories separately: Search bots, Training bots, and Agent bots.
The shift addresses a real problem. Website owners want different policies for different crawler types. Search bots improve discoverability. Training bots feed AI models. Agent bots perform autonomous tasks. Blocking all three indiscriminately wastes legitimate traffic while preventing sites from benefiting from search indexing.
Starting September 15, 2026, Training and Agent bots will face blocks by default on ad-supported pages. This protects publisher interests. Ad-supported sites lose revenue when their content trains commercial AI models without compensation. Search bots remain unblocked by default since they drive traffic.
The granular approach gives site owners real choice. A news publisher might allow search crawlers, block training bots, and allow agent bots for customer service tasks. A SaaS company might do the opposite. A data provider could monetize training access while blocking agents.
This matters because AI companies increasingly rely on web crawling for model training. Without clear controls, websites face a choice between accepting all crawlers or rejecting them entirely. Cloudflare's implementation lets publishers preserve their economic interests while staying accessible to legitimate search and discovery.
The default policy protecting ad-supported sites signals Cloudflare's understanding of publisher concerns. Training bots represent direct value extraction without permission or compensation. Agents pose security and resource drain risks. Search bots, by contrast, generate referral traffic.
Implementation remains simple. Cloudflare customers access a settings panel to enable or disable each bot category per domain. The company handles detection and routing behind the scenes. No manual configuration required.
This represents infrastructure-level policy enforcement. Rather than relying on robots.txt files that AI