AI crawlers and robots.txt:
a practical guide to who does what

Which AI crawlers should you block in robots.txt?

It depends on the job each crawler does. Training crawlers such as GPTBot and ClaudeBot collect content that may be used to train models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the index that AI products search when they answer with links. If you want to be cited but do not want your content used for training, block the training crawlers and Google-Extended and allow the rest. Blocking Google-Extended also opts the site out of grounding in Gemini Apps.

It is tempting to treat every AI bot as the same thing and write one rule for all of them. The operators, though, document at least three different jobs. A rule that blocks one job can leave another untouched. This guide goes through the main tokens using each operator's own documentation. Then it looks at what robots.txt can and cannot do.

Three different jobs

Across the documentation, the same division keeps appearing.

• Training crawlers collect content that may be used to train models.
• Search crawlers build the index an AI product searches when it answers with links.
• User-triggered fetchers visit a page because a person asked a question that needed it.

Deciding what to allow becomes simpler once each token is sorted into one of those three.

How robots.txt matching works

The Robots Exclusion Protocol was published as a standards track document, RFC 9309, in September 2022. Four of its rules matter for this topic.

• The file must live at /robots.txt at the top level of the host.
• A crawler looks for the group whose user-agent line matches its product token, ignoring case. Only if no group matches does it fall back to the * group. A bot you name therefore follows its own group and ignores your general rules.
• Within a group, the most specific matching path wins, meaning the longest one.
• Crawlers should not rely on a cached copy for more than 24 hours.

The RFC is also frank about its limits. It says the rules are not a form of access authorisation. A robots.txt file is a published request. It works because crawlers choose to honour it.

The tokens, operator by operator

OpenAI

OpenAI's crawler overview says each setting is independent. So a site can allow OAI-SearchBot while disallowing GPTBot. Disallowing GPTBot signals that the site's content should not be used in training. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. They can still appear as navigational links. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change.

ChatGPT-User is a different kind of agent. OpenAI says robots.txt rules may not apply to it, because a user starts these visits. It also says the token is not used to decide whether content appears in search. A fourth agent, OAI-AdsBot, only visits pages submitted as ads on ChatGPT.

Anthropic

Anthropic's help article says its bots honour robots.txt directives. It lists all three bots, including the user-triggered one, as things a site owner can disable there. It supports the non-standard Crawl-delay extension. It also advises against relying on IP blocking as an opt-out. A blocked crawler cannot read the robots.txt file at all. It asks owners to add rules on every subdomain they want covered.

Perplexity

Perplexity's crawler page recommends allowing PerplexityBot if you want to appear in its results. It says changes may take up to 24 hours to be reflected. Of Perplexity-User it says the agent generally ignores robots.txt rules, since a user requested the fetch.

Google

Google-Extended is easy to misread. Google's crawler documentation says it has no separate user-agent string. Crawling is done by Google's existing crawlers. The token acts only as a control. It manages whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says that it does not affect a site's inclusion in Google Search and is not a ranking signal.

That means Google-Extended does not control AI Overviews or AI Mode. Google's AI features guidance treats those as part of Search. They are governed by Googlebot rules and by preview controls such as nosnippet. Since 2026 there is also a separate Search generative AI control in Search Console. Google says it generally takes a few days to take effect and does not affect AI training.

A worked example

Suppose a publisher wants to be found and cited in AI search products but does not want its content used for model training. Going by the documentation above, the file would look like this.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

The search crawlers and user-triggered fetchers are not named. So they fall through to the general group and are allowed. Note one consequence of the Google-Extended line. It also opts the site out of grounding in Gemini Apps. That is a visibility cost as well as a training choice.

Where robots.txt stops

• It is a request. The RFC says so directly. Content that must not be fetched needs real access control.
• User-triggered agents are treated differently. OpenAI and Perplexity both say robots.txt may not apply to fetches a user asked for. Anthropic says its user agent can be disabled there.
• Your own infrastructure can block bots you want. Google's guidance mentions crawling being allowed by any CDN or hosting infrastructure as well as by robots.txt. OpenAI and Perplexity publish IP ranges so that firewalls can let their bots through.
• Nothing is instant. Expect roughly a day for OpenAI and Perplexity. Expect a few days for Google's Search Console control.

Standards work in progress

Product tokens were not designed to say how content may be used. The IETF's AI Preferences working group is chartered to standardise a vocabulary for those preferences. It also covers ways to attach them to content, including through robots.txt and HTTP response headers. Its vocabulary draft was still an Internet-Draft when I checked on 2 October 2026. It was last revised in September 2026. It states that its contents do not yet reflect working group consensus. Until that work is finished, the per-operator tokens above are the practical tool.

My view

This part is opinion. I think the decision is best made by job. The brand matters less. Blocking a search crawler removes you from the answers people are reading. That is rarely what a business wants. Blocking a training crawler is a separate and legitimate choice about how your work is reused. This site allows all of them. I want these articles to be found and cited wherever people look.

For what happens after the crawl, see how AI answer engines pick and cite sources. For checking whether any of it is working, see measuring AI visibility.

Sources

1. IETF, RFC 9309: Robots Exclusion Protocol: rfc-editor.org/rfc/rfc9309.html
2. OpenAI, Overview of OpenAI crawlers: developers.openai.com/api/docs/bots
3. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?: support.claude.com/en/articles/8896518
4. Perplexity, Perplexity crawlers: docs.perplexity.ai/docs/resources/perplexity-crawlers
5. Google, Google's common crawlers (Google-Extended): developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
6. Google Search Central, AI features and your website: developers.google.com/search/docs/appearance/ai-features
7. Google Search Console Help, Search generative AI control: support.google.com/webmasters/answer/16908024
8. IETF Datatracker, AI Preferences (aipref) working group: datatracker.ietf.org/wg/aipref/about
9. IETF Datatracker, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab): datatracker.ietf.org/doc/draft-ietf-aipref-vocab