# Crawler documentation changes

> A dated record of changes to official crawler documentation from Google, OpenAI and the IETF, found by this wiki's weekly source checks. Each change says whether the substance we cite changed or only the location.

- URL: https://geo.casa/crawling-indexing/documentation-changes/
- Change log · Observed in practice
- Reviewed: 2026-10-25

This log records changes to the crawler documentation that entries in this wiki cite. Changes are found by `npm run check:sources`, which runs weekly. Every change is re-read before it is logged, to tell a page that moved from a policy that changed.

## Changes

### 2026-10-25: Google's crawl budget and HTTP status code guides moved to the crawling documentation site

Observed in practice. The crawl budget guide (previously search/docs/crawling-indexing/large-site-managing-crawl-budget) and the HTTP status codes guide (previously search/docs/crawling-indexing/http-network-errors) now live under developers.google.com/crawling/docs/; old URLs redirect. Re-checked the facts this wiki cites, such as the 1 million+ and 10,000+ page thresholds and 4xx/5xx handling.

Sources: [Google: Crawl budget management](https://developers.google.com/crawling/docs/crawl-budget); [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)

### 2026-10-25: Google's crawler verification guide moved to the crawling documentation site

Observed in practice. "Verifying Googlebot and other Google crawlers" moved from developers.google.com/search/docs/crawling-indexing/verifying-googlebot to developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests; the old URL redirects. Re-checked: verification is still a reverse DNS lookup (googlebot.com, google.com or googleusercontent.com) followed by a forward lookup, or matching against Google's published IP lists, now split into per-category JSON files.

Sources: [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)

### 2026-10-25: Google's robots.txt specification moved to the crawling documentation site

Observed in practice. "How Google interprets the robots.txt specification" moved from developers.google.com/search/docs/… to developers.google.com/crawling/docs/robots-txt/robots-txt-spec; the old URL now returns 404 without a redirect. Re-checked: the 500 KiB limit, the least-restrictive rule for conflicts, unsupported crawl-delay and up-to-24-hour caching are unchanged.

Sources: [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)

### 2026-10-25: OpenAI's crawler overview moved to developers.openai.com

Observed in practice. The overview of OpenAI crawlers moved from platform.openai.com/docs/bots to developers.openai.com/api/docs/bots; the old URL redirects. Re-checked that OAI-SearchBot and GPTBot are still independent settings and that robots.txt changes take about 24 hours to reflect in search.

Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

### 2026-10-25: RFC 9309 short link now opens the RFC's metadata page

Observed in practice. rfc-editor.org/rfc/rfc9309 now redirects to the info page rather than the text. The specification itself is unchanged; this wiki now links the HTML text directly.

Sources: [IETF: RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)

### 2026-10-24: Google's crawler list lives on the crawling documentation site

Observed in practice. The list of Google's common crawlers, including the Google-Extended token, is served from developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers; the former Search Central path redirects there.

Sources: [Google: Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)


## Sources

- [Google: How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Google: Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [Google: Crawl budget management](https://developers.google.com/crawling/docs/crawl-budget)
- [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)
- [IETF: RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)