Context: Cloudflare’s AI Crawler Rules and What Happens Next

According to Search Engine Journal, Cloudflare is updating how it identifies and blocks AI crawlers. Starting September 15, websites that block AI training may also unintentionally block Googlebot. This is because Cloudflare now groups crawlers by behavior, and a single rule can affect both search indexing and AI training.

For Australian small and mid-sized businesses, this is a big deal. Many use Cloudflare for speed and security without realising the default settings could harm their Google search rankings. If Googlebot can’t access your site, your pages won’t appear in search results. That means less traffic and fewer customers.

Why This Shakes Up SEO and Search Rankings for Businesses

Google’s algorithm depends on crawling to discover and rank your content. If you block Googlebot — even by accident — your pages stop being indexed. Over time, your search rankings drop, and competitors who allow crawling will take your place.

AI SEO tools are becoming more common, but this update shows a hidden risk. Many businesses block AI crawlers to protect their content from being used to train models. However, the same crawler that trains AI often also handles search indexing. You can’t block one without risking the other. The line between “search” and “training” is blurry, and Cloudflare’s new rules make that trade-off very real.

This is not just a technical setting change. It’s a strategic decision about how visible you want your website to be. If you value search rankings, you may need to allow some AI crawlers to keep Googlebot happy.

What This Means for Australian SMBs

Many Australian small businesses run on tight budgets and use Cloudflare’s free tier. The default changes on September 15 will automatically block Training and Agent crawlers on new sites. If you previously turned on the “Block AI bots” setting, your site could lose Googlebot access without you knowing.

For a local café, tradie, or family-owned retail store, losing organic search traffic means fewer walk-ins and online orders. The Australian market is competitive, and every click counts. Ignoring this update could quietly damage your digital presence.

What You Can Do Now

  • Log into your Cloudflare dashboard before September 15 and review your AI crawler settings. Make sure “Search” crawlers are allowed, even if you block “Training” or “Agent” bots.
  • Check if you have the older “Block AI bots” toggle enabled. If yes, disable it and instead set separate rules for each crawler behaviour.
  • Review your robots.txt file to ensure it doesn’t accidentally block Googlebot. Cloudflare’s new signals work alongside robots.txt, but don’t rely on one alone.
  • Test your site using Google’s URL Inspection tool to confirm that Googlebot can still crawl your key pages.
  • Talk to your IT support or web developer about your specific needs. A one-size-fits-all block on AI crawlers may cost you more than it saves.

Staying visible on Google while protecting your content is a balancing act. MS&VG’s AI-powered SEO service can help Australian SMBs navigate these changes, ensuring your site stays indexed without giving away your data for free.