AI agent crawlers, which are bots that fetch pages in real time on behalf of people waiting for an answer, will be blocked by default on parts of the web starting September 15th. Cloudflare announced the change on July 1st, and most of the coverage since then has focused on Google. The more useful part is what you demand from all your building agents and what you offer them in return.
Cloudflare replaced the single block AI bot switch with three categories: Search includes bots that index pages to later answer questions about them. Agents cover automated systems that work in real-time on behalf of users, such as ChatGPT’s fetch bots and browser-driven agents. Training covers the crawler, which ingests the content into the weights of the model. This control went into effect on July 1 for all customers, including free tier customers.
Starting September 15th, the default will change. Training and agents will be blocked on pages that display ads, but search will still be allowed. The new defaults apply to newly onboarding domains to Cloudflare, new sites set up by existing customers, and all existing free tier customers.
Those who don’t want them can opt out through their security settings before the date.
Cloudflare’s logic is that ads are proof that a page was built for human access. Search crawlers that send readers back are referrals. Bots that read pages and pass answers to others are not.
What AI agent crawlers are currently encountering
Agent deployment is built on the premise that the open web remains open. A research agent retrieves a competitor’s pricing page. Monitoring tools check supplier announcements. The customer service agent obtains the manufacturer’s specifications. None of these require a license, and none of them have required a license so far.
Cloudflare accounts for the majority of the world’s web traffic, and its blocking operates at the network level, rather than as a robots.txt suggestion that crawlers can ignore. Ad-supported pages are exactly what agents want because they feature news, reviews, pricing, and product content. Enterprise agent failure mode is not a lawsuit. It’s an answer built from silence, or whatever it can reach.
There is a complication in the form of Google. Googlebot crawls both search and training in one bot, so under the most restrictive rules, sites that block training will also block Googlebot. Cloudflare CEO Matthew Prince said the company hopes the change will “encourage mixed-use crawlers to separate search from agent usage and training,” which is a polite way of saying pressure is important.
Obtaining permission for the AI agent crawler
Anyone running an agent should start by considering which Cloudflare account will be read as the agent class. This classification is based on behavior, not user choice, so any investigative agent browsing in real time will be captured regardless of whether operators consider it a crawler.
Since the block will hit ad-supported pages and leave the rest of the pages accessible, expect a reduction in coverage rather than a complete failure. Rather than rewriting the user agent string, negotiated access is the way to go.
Publishers have different homework lists. Existing free tier customers will be automatically transitioned to the new default on September 15th, so check your tier first. Most details are omitted. Next, decide if blocking training is worth the cost, as it requires Googlebot and comes with search visibility.
The notable mechanism is money. Pay Per Crawl is becoming Pay Per Use, with Ceramic.ai paying publishers when their content appears in AI search results and You.com paying agents when they reach their premium content. Cloudflare says more than half of its AI crawler’s traffic is spent refetching pages that haven’t changed, and there’s waste on both sides that’s worth putting a price on.
This is the first round of the content battle, and the answer presented is rates, not walls.
One weakness lies in the classification itself. Search, agent, and train are behaviors that AI companies declare for their bots, and there is a clear incentive for companies to not want their training to be classified as training. The announcement did not explain what would stop it.
Access to the open web has been free and unrestricted for 30 years, and the bill is now itemized. Agent builders who organize access before September will have a viable issue. Anyone who discovers it from a 403 will rebuild it on the fly.
See also: Visa ChatGPT integration enables AI agents to make retail purchases
Want to learn more about AI and big data from industry leaders? Check out the AI & Big Data Expos in Amsterdam, California, and London. This comprehensive event is part of TechEx and co-located with other major technology events such as Cyber Security & Cloud Expo. Click here for more information.
AI News is brought to you by TechForge Media. Learn about other upcoming enterprise technology events and webinars.
