Why a Simple Text File Still Shapes SEO in the AI Era
Search Engine Journal recently published a guide on robots.txt best practices for search and AI crawlers, and it is a useful reminder that the humblest file on a website can decide who ever sees your content. The file sits at the root of a site and tells visiting bots which parts of the site they are allowed to crawl.
What makes the topic timely is that AI tools and agents now visit websites alongside traditional search crawlers. Different bots can follow different rules, so one careless line can quietly lock your best pages away from Google, from AI systems, or from both.
The Real Fight for Search Rankings Is Now About Access
Most conversations about Google algorithm changes focus on content, links and page speed. Robots.txt rarely gets a mention, yet it decides whether your content is even eligible to compete for search rankings in the first place.
The common mistake is treating blocking as a safety switch. Blocking more does not make a site safer or faster by default. It can simply remove pages from view, sometimes for years, without anyone noticing.
The rules themselves are deceptively simple. Only two wildcard characters exist, URL matching is case-sensitive, and a more specific rule wins over a broader one. A small typo can therefore block far more than you intended.
AI has raised the stakes further. A blocked AI crawler may mean your brand never appears in an AI-generated answer, and those answers are quickly becoming a real entry point for customers doing research.
There is a crawl budget angle too. Large sites with filter pages, internal search results and add-to-cart links can send bots through endless near-duplicate URLs. Careful rules keep that attention on the pages that actually sell.
What This Means for Australian SMBs
Many Australian small and mid-sized businesses run on WordPress, Shopify or a managed platform where robots.txt has not been touched since launch. That default file was written for search engines of an earlier era, not for AI crawlers or modern filtering pages.
For a local retailer or service business, the risk is rarely dramatic. It is quieter — a handful of valuable pages that never rank, or a brand that never gets mentioned when someone asks an AI assistant for a recommendation in their city. Fixing it is cheap; ignoring it is not.
What You Can Do Now
- Open yoursite.com/robots.txt and read it. If you cannot explain what each line blocks, that is your first task.
- Treat search crawlers and AI crawlers as separate decisions. You may want Google to index a page while keeping it out of AI training data, or the reverse.
- Block low-value URLs, such as internal search results, filter and sort combinations, and cart or wishlist actions.
- Never rely on robots.txt to protect private information. Use passwords and proper access controls instead.
- Check your crawl stats before and after any change, so you can see what actually shifted rather than guessing.
If this feels like one more thing on a long list, MS&VG provides AI-powered SEO services for Australian businesses, including practical technical reviews of settings exactly like these.