Should a Small Business Let AI Crawlers Like GPTBot Read Its Website?

In every Cloudflare account, and in a growing number of hosting panels, there is now a switch that blocks AI crawlers in one click. A tradie hears two opposite things about it. The web developer says flick it, because AI companies are copying your content for nothing. The marketing email says never flick it, because ChatGPT is where people search now and you want to be the plumber it names.

Both pieces of advice treat "AI crawlers" as one thing. They are not. There are three kinds, run by the same handful of companies, and they have almost nothing to do with each other beyond a shared word in the name. The kind a tradie might want to keep out costs nothing visible to keep out. The kind that costs a ChatGPT answer when blocked is not the kind the switch was built for.

This post is for the owner of a plumbing, electrical, roofing or landscaping business in Sydney and Greater NSW whose website is a dozen pages of services, suburbs and photos, the same reader as the Tradies and home services page. It sets out which crawlers exist, what each one does to that site, what robots.txt can and cannot do, and where Durbacti has landed on its own site. It is the follow-up promised in SEO, AEO and GEO: What Is the Difference, and Which Does Your Business Need?, which left the training crawler question for another day.

Which AI crawlers are there, and what does each one do?

AI crawlers come in three kinds, and the companies that run them publish the distinction themselves.

The first kind collects pages to train a model. OpenAI's is GPTBot, which its crawler documentation, read on 26 October 2026, says "is used to crawl content that may be used in training our generative AI foundation models". Anthropic's is ClaudeBot, which its crawler support article, read the same day, describes as collecting web content that could be used for training. Google's is not a crawler at all but a robots.txt token called Google-Extended. Google's crawler documentation, read the same day, says the token controls whether crawled content may be used to train future Gemini models and for grounding, and that it "doesn't have a separate HTTP request user agent string": the ordinary Googlebot does the fetching and the token only sets permission. Common Crawl's CCBot, Apple's Applebot-Extended and Meta's meta-externalagent sit in the same group.

The second kind builds a search index so an assistant can look things up live. OpenAI's is OAI-SearchBot, which the documentation says "is used to surface websites in search results in ChatGPT's search features". Anthropic's is Claude-SearchBot, which the support article says indexes content to make search responses more relevant and accurate.

The third kind is not a crawler either. ChatGPT-User and Claude-User fetch a page at the moment a person asks a question that needs it. OpenAI's page says ChatGPT-User "is not used for crawling the web in an automatic fashion", and that because the fetch is initiated by a user, robots.txt rules may not apply to it. Anthropic says blocking Claude-User "prevents our system from retrieving your content in response to a user query".

The sentence that settles most of the argument is in OpenAI's documentation: "Each setting is independent of the others." A site can disallow GPTBot and allow OAI-SearchBot, and OpenAI's own example does exactly that. Blocking the training crawler does not block the search crawler, and allowing the training crawler does not let the search crawler in. They are separate lines in the same file.

What happens to a tradie's website if the wrong one is blocked?

Blocking the search and user crawlers removes a tradie's website from the answers. OpenAI's documentation says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links", and that a robots.txt change takes about 24 hours to reach its systems. Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". The scenario looks like this. A homeowner in Cronulla with a leaking hot water system asks ChatGPT for a plumber nearby who can come today. ChatGPT searches. A plumber whose developer blocked OAI-SearchBot in 2024 is not in the set of pages it can draw on, however well the site ranks on Google, and the plumber two suburbs over is.

Blocking the training crawlers removes nothing a tradie can see. OpenAI describes a GPTBot disallow as a signal that "a site's content should not be used in training generative AI foundation models", and Anthropic says a ClaudeBot block "signals that the site's future materials should be excluded from our AI model training datasets". Neither company says a training block changes what its assistant shows in search, and both publish separate bots for search so that it does not.

Google is the case that catches people. Google-Extended, in Google's own words, "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Google's page on AI features and your website, read on 26 October 2026, says there are no additional requirements to appear in AI Overviews or AI Mode, and that the robots.txt rules for Googlebot are the control over how a site is crawled for Search. A tradie who blocks Google-Extended has told Google not to train Gemini on the site and not to use it for grounding in Gemini apps; the Google listing and the AI Overview do not change. A tradie who blocks Googlebot to "stop the AI" has removed the site from Google.

Does allowing GPTBot get a business recommended by ChatGPT?

No, and the data from the company that publishes the most of it says why. Cloudflare, which reports crawler traffic across the sites on its network, published its figures on 29 August 2025: over the previous 12 months, 80% of AI crawling was for training, 18% for search and 2% for user actions. The same post gives each company's crawl-to-refer ratio, the number of pages its bots fetched for every visit they sent back. In July 2025 Anthropic fetched about 38,000 pages per referred visit, OpenAI 1,091, Perplexity 195 and Google 5.4. Cloudflare's 2025 Year in Review, reported by Search Engine Journal on 15 December 2025, put Anthropic's ratio anywhere between 25,000:1 and 100,000:1 in the second half of the year and OpenAI's as high as 3,700:1 in March, so the numbers move a great deal from month to month. The shape does not. Training crawls are most of the traffic and send back almost none of the visitors.

What a training crawl does for a tradie is therefore hard to state and impossible to measure. A model that read a plumber's website during training may, in some way nobody can inspect, be more likely to know the business exists. But a model answers a "plumber near me" question by searching, not by remembering, and the search goes through the search crawler's index. The thing that gets a tradie named in a ChatGPT answer is being in the OAI-SearchBot index with a page that says the suburb and the service in plain words. How to Get Your Website Cited by AI: What We Changed on Ours covers the page. The robots file only decides whether the page can be read.

Durbacti's own robots.txt allows all of them, the training crawlers included, and records the training allowance as a policy decision made in August 2026. The reasoning was that the site's content is written to be quoted, the cost of serving a crawler is trivial, and a stated position in a training set is still a stated position. Whether that decision has done anything is not measurable, and this post would rather say so than claim a result.

What can robots.txt not do?

robots.txt is a request. It is a text file at the root of a site that asks named crawlers to stay out of named paths, and a well-behaved crawler obeys it. Nothing enforces it. OpenAI and Anthropic both say their bots honour it. The trouble is the bots that do not. Cloudflare alleged in August 2025 that Perplexity was using undeclared crawlers to fetch pages from sites that had blocked its named bot, across tens of thousands of domains; Perplexity's spokesperson called the post a "sales pitch" and said the bot named in it was not operated by the company, as TechRepublic reported on 5 August 2025. Whoever is right about that case, a tradie should take one thing from it: a Disallow line is honoured by the companies that choose to honour it.

A robots.txt change is not retroactive. Anthropic's wording is "future materials". Pages a training crawler has already fetched are already fetched.

It is not a privacy control. The Office of the Australian Information Commissioner's guidance on developing and training generative AI models, published 21 October 2024, says "just because data is publicly available or otherwise accessible does not mean it can legally be used", and that "a failure of a website to implement measures to prevent data scraping should not be taken as implied consent". The guidance is written for AI developers, not for tradies. What it tells a tradie is that the decision about what sits on the page, a customer's full name and street in a testimonial, a photo of the inside of someone's home, is a separate decision from the robots file and comes first. Whether the Privacy Act applies to a particular small business, and what it asks of one that it does apply to, is a question for a lawyer.

And it is not a security control. A Disallow line is also a map of what the owner would rather nobody read. Durbacti's site has no Disallow rules at all. The two pages it wants kept out of search carry a noindex tag instead, because a crawler blocked by robots.txt cannot read that tag, and a linked-to address can still surface as a bare result.

Is it legal for AI companies to train on a tradie's website anyway?

Nobody should take that answer from a web developer, and this post does not give one. The shape of the question is this. Australia's Productivity Commission put a text and data mining exception to copyright on the table in its 2025 interim report on data and digital technology, which would have made training on published material lawful without permission in some circumstances. The Federal Government ruled it out in late October 2025. Attorney-General Michelle Rowland said there were "no plans to weaken copyright protections when it comes to AI", as King & Wood Mallesons summarised on 30 October 2025, and the Copyright and AI Reference Group is now looking at paid collective licensing, the status of AI-generated material and a small claims forum for low-value disputes. What any of that means for the text and photos on a plumber's website, and what a tradie could do about a crawler that ignored a block, is for a lawyer. The robots file is a signal of what the owner permits. It is not the law and it does not replace one.

What should a tradie's robots.txt say?

For a service business whose website exists to be found, the file is short and the decision has two parts.

What we will not do

What to ask any web developer about AI crawlers

A tradie's website is a dozen pages that exist to be found and to make the phone ring. Let the crawlers that answer questions read it, decide about the ones that train models with eyes open, and spend the worry on what the pages say. The Website development page describes how Durbacti builds a site every crawler can read, and what your current robots.txt is doing is a five-minute check at the start of a free strategy call.