# Configure robots.txt for AI crawlers

> Treat search crawlers and training crawlers as separate decisions. Blocking a search crawler such as OAI-SearchBot removes you from that product's answers; blocking a training token such as GPTBot or Google-Extended does not affect search inclusion.

- URL: https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/
- Guide · Officially documented
- Reviewed: 2026-10-26

## Before you start

- You can edit and deploy `/robots.txt` for every host you care about.
- You know whether your CDN or WAF also blocks bots. Google notes that crawling must be allowed "by any CDN or hosting infrastructure", not just robots.txt.

## 1. Decide per purpose

| Purpose | Token | If disallowed |
|---|---|---|
| Google Search, including AI Overviews & AI Mode | `Googlebot` | Page is not crawled for Search at all. |
| Gemini training and grounding; training of the models behind Search's AI features | `Google-Extended` | No effect on Search inclusion or AI Overviews display. |
| ChatGPT search results | `OAI-SearchBot` | Not shown in ChatGPT search answers (may still appear as navigational links). |
| OpenAI model training | `GPTBot` | Content indicated as not for training. |
| User-initiated fetches in ChatGPT | `ChatGPT-User` | robots.txt "may not apply"; not used for search inclusion. |
| Claude search results | `Claude-SearchBot` | Content not indexed for Claude's search; "may reduce your site's visibility". |
| Claude user requests | `Claude-User` | Claude cannot fetch your pages when users ask about them. |
| Anthropic model training | `ClaudeBot` | Future content excluded from training datasets. |
| Perplexity search results | `PerplexityBot` | Not surfaced in Perplexity search results. |
| User-initiated fetches in Perplexity | `Perplexity-User` | "Generally ignores robots.txt rules", since a user asked. |

> **Officially documented:**
> OpenAI states that the OAI-SearchBot and GPTBot settings are independent. A site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Changes take about 24 hours to be reflected in search.
>
> Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

> **Officially documented:**
> Anthropic separates ClaudeBot (training), Claude-User (user-requested fetches) and Claude-SearchBot (search indexing). Blocking ClaudeBot signals that future content "should be excluded from our AI model training datasets". Blocking the other two may reduce visibility in Claude. Anthropic also honours the non-standard `Crawl-delay` directive, which Google does not support.
>
> Sources: [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

> **Officially documented:**
> PerplexityBot surfaces and links websites in Perplexity search results and "is not used to crawl content for AI foundation models". Perplexity-User fetches pages when users ask questions. Because a user requested the fetch, it generally ignores robots.txt rules. Changes may take up to 24 hours.
>
> Sources: [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

> **Officially documented:**
> For Google, AI features are part of Search, so robots.txt rules for Googlebot are the crawl control. Google-Extended applies to some of Google's other AI systems, not to Search.
>
> Sources: [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

> **Observed in practice:**
> An August 2026 audit of robots.txt on 130 company sites looked at the companies that had made an explicit choice. They almost never blocked search crawlers and blocked training crawlers only. The study's authors call blocking both together the most common and most damaging mistake.
>
> Sources: DATA17 AI Research Lab · 解数智能: 你的公司还在 AI 的答案里吗：企业 GEO 行动指南 · 2026 (unpublished report, not publicly available; readers can't check it)

## 2. Write the rules

Example: appear in Google, ChatGPT, Claude and Perplexity search; opt out of model training.

```txt
user-agent: Googlebot
allow: /

user-agent: OAI-SearchBot
allow: /

user-agent: GPTBot
disallow: /

user-agent: Google-Extended
disallow: /

user-agent: Claude-SearchBot
allow: /

user-agent: ClaudeBot
disallow: /

user-agent: PerplexityBot
allow: /

user-agent: *
allow: /
```

A crawler follows only the most specific group matching its name, so rules under `*` do not add to a named group.

## 3. Verify

1. Fetch the live file: `curl -sA "OAI-SearchBot" https://example.com/robots.txt`.
2. In Google Search Console, open the robots.txt report to confirm Google fetched the new version.
3. Check server or CDN logs for requests from each bot, and confirm the source IPs against the vendor's published ranges (OpenAI publishes them as JSON for each bot).

## Rollback

Restore the previous file. Google usually refreshes its cached copy within 24 hours.

## Common mistakes

- Blocking `OAI-SearchBot` while meaning to block training.
- Assuming [Google-Extended](https://geo.casa/crawling-indexing/google-extended/) removes content from AI Overviews. It does not; use the [Search generative AI control](https://geo.casa/ai-search/search-generative-ai-control/) or snippet controls.
- Allowing bots in robots.txt while a WAF rule still returns `403`.

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google: Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- DATA17 AI Research Lab · 解数智能: 你的公司还在 AI 的答案里吗：企业 GEO 行动指南 · 2026 (unpublished report, not publicly available; readers can't check it)