# robots.txt

> robots.txt tells crawlers which URL paths they may fetch, and is standardised as RFC 9309. It controls crawling, not indexing. A disallowed URL can still be indexed without its content if other pages link to it.

- URL: https://geo.casa/crawling-indexing/robots-txt/
- Reference · Officially documented
- Reviewed: 2026-10-24

## Location and format

- Served at `/robots.txt` at the root of each host and protocol (`https://example.com/robots.txt` does not cover `https://shop.example.com/`).
- Plain text, UTF-8. Groups start with one or more `user-agent` lines followed by `allow` / `disallow` rules.

## Supported fields (Google)

| Field | Purpose |
|---|---|
| `user-agent` | Which crawler the group applies to; `*` matches any crawler without a more specific group. |
| `disallow` | Path prefix the crawler should not fetch. |
| `allow` | Path prefix that may be fetched, overriding a broader `disallow`. |
| `sitemap` | Absolute URL of a sitemap. Not tied to any group. |

> **Officially documented:**
> Google does not support other fields such as `crawl-delay`. In case of conflicting rules, including wildcards, Google uses the **least restrictive** rule.
>
> Sources: [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)

`*` (any sequence) and `$` (end of URL) are supported in paths.

## Limits and caching

> **Officially documented:**
> Google enforces a 500 KiB size limit and ignores content after it. It generally caches robots.txt for up to 24 hours, and may cache longer when refreshing fails (for example on timeouts or 5xx errors).
>
> Sources: [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)

## Crawling is not indexing

> **Officially documented:**
> A page blocked by robots.txt can still appear in results, without a description, if other pages link to it. To keep a page out of results, allow crawling and use `noindex` instead.
>
> Sources: [Google Search Central: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)

`noindex` inside robots.txt is not supported. Use the robots meta tag or `X-Robots-Tag` header instead; [Keep a page out of Google](https://geo.casa/crawling-indexing/keep-out-of-google/) has the steps.

## Baidu

> **Officially documented:**
> Baiduspider follows robots.txt, and rules can target its product-specific user agents such as `Baiduspider-image`. `Baiduspider-video` does not support the rules. Blocking `Baiduspider` removes the site from Baidu search results "and in any other the search results which is provided by Baidu". `Baiduspider-cpro` and `Baiduspider-ads` work under customer agreements and do not follow robots.txt.
>
> Sources: [Baidu: FAQs of Baiduspider](https://help.baidu.com/question?prod_id=99&class=0&id=3001)

The FAQ names no separate token for Baidu's AI products, so it documents no way to treat them differently from Baidu search. Google, by contrast, has a separate token for Gemini ([Google-Extended](https://geo.casa/crawling-indexing/google-extended/)).

## Other Chinese search engines

> **Officially documented:**
> 360 Search supports `user-agent`, `allow` and `disallow`, with tokens `360Spider`, `360Spider-Image` and `360Spider-Video`. It adds its own `Indexpage` field to mark frequently updated pages.
>
> Sources: [360 Search: Robots协议支持](https://www.so.com/help/help_3_2.html)

> **Officially documented:**
> Sogou's crawler supports robots.txt, but a changed file may take several weeks to take effect, and pages you block cannot be found in Sogou search.
>
> Sources: [Sogou: 站长指南](https://www.sogou.com/docs/help/webmasters.htm)

> **Officially documented:**
> Toutiao Search's crawler introduction (2019) gives the `Bytespider` user agent and IP ranges but does not mention robots.txt.
>
> Sources: [Toutiao Search: 头条搜索spider介绍](https://www.toutiao.com/i6759451726496940557/)

## Example

```txt
user-agent: *
disallow: /search
allow: /search/about

sitemap: https://example.com/sitemap-index.xml
```

## See also

- [Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/) — per-crawler rules for AI search and training bots.

## Sources

- [IETF: RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- [Google Search Central: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Google Search Central: Block Search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing)
- [Baidu: FAQs of Baiduspider](https://help.baidu.com/question?prod_id=99&class=0&id=3001)
- [Sogou: 站长指南](https://www.sogou.com/docs/help/webmasters.htm)
- [360 Search: Robots协议支持](https://www.so.com/help/help_3_2.html)
- [Toutiao Search: 头条搜索spider介绍](https://www.toutiao.com/i6759451726496940557/)