Why This News About Robots.txt Matters

According to a recent report covered by Search Engine Journal, OpenAI has stated that its ChatGPT-User bot may not be bound by the robots.txt rules that websites set. This is a big deal because robots.txt has long been the main way site owners tell bots which pages they can and cannot visit.

If the rule suddenly becomes optional for some AI bots, businesses lose a key tool for controlling how their content gets used. It changes the basic agreement between website owners and automated visitors, and that affects everything from traffic to brand safety.

What This Means for Your Google Rankings and AI SEO

For years, SEO was about making sure Google’s crawler could read your site while blocking spam bots. Now, AI bots like ChatGPT-User can ignore those blocks if a user asks for a specific page. That means your content could end up inside an AI answer without your permission, potentially reducing the number of people who actually click through to your site.

Google’s own algorithm is also paying closer attention to how AI uses web content. If your pages are being fetched by AI bots and then summarised, Google may treat that as a signal that your content is less original or less valuable. That can hurt your search rankings over time.

The traditional SEO playbook of “just block unwanted bots” no longer works as cleanly. You need a strategy that acknowledges AI bots are different from standard search crawlers, and that outright blocking might cut you out of emerging AI-driven discovery channels.

What This Means for Australian SMBs

Small and mid-sized businesses in Australia often rely heavily on organic search traffic to attract local customers. If an AI tool pulls information from your site and presents it in a chat response, a potential customer might never visit your website at all. That’s lost visibility and lost opportunities.

At the same time, Australian businesses operate under strict privacy and data use laws. Having an AI bot access pages you intended to block could create legal grey areas. It’s not just about SEO – it’s about knowing exactly what data is being taken and how it’s being used.

What You Can Do Now

  • Review your server logs or CDN reports to see which bots are actually hitting your site, especially ChatGPT-User, Bytespider, and Youbot. Don’t rely only on what your robots.txt says.
  • Update your robots.txt file to explicitly disallow both ChatGPT-User and OAI-SearchBot if you want to block AI use. Remember that blocking OAI-SearchBot also removes your site from ChatGPT search results – a trade-off you should consider carefully.
  • Use a web application firewall or a service like Cloudflare to enforce crawler rules at the network level, not just the text file level. This gives you more control even if a bot claims to ignore robots.txt.
  • Create high-value, original content that AI models cannot easily summarise – local insights, customer stories, and interactive tools that require a real visit to be useful.
  • Monitor changes from major platforms like Cloudflare and Google. Starting mid-2026, new Cloudflare domains will block training bots by default on ad-supported pages, which could shift how Australian businesses manage their AI exposure.

Navigating this new AI-driven search landscape requires a clear understanding of how bots behave and what your options are. MS&VG’s AI-powered SEO service helps Australian small and mid-sized businesses adapt to changes in search rankings and bot management, so you stay visible and in control.