# HTTP status codes and crawlers

> 4xx URLs (except 429) are not indexed, and 401 or 403 should not be used to limit crawl rate. 5xx and 429 slow crawling, and indexed URLs are kept for a while before being dropped. A 2xx makes content eligible but doesn't guarantee indexing; 301 is a strong redirect signal, 302 a weak one.

- URL: https://geo.casa/implementation/http-status-crawlers/
- Reference · Officially documented
- Reviewed: 2026-10-25

## Summary

| Code | Google's crawlers |
|---|---|
| 2xx | Content considered for indexing — not guaranteed |
| 301 | Followed; strong signal the target should be processed |
| 302 | Followed; weak signal |
| 4xx (not 429) | Not indexed; previously indexed URLs removed; no effect on crawl rate |
| 429, 5xx | Crawling slows; indexed URLs kept for a while, eventually dropped |

## 2xx

> **Officially documented:**
> "For Google Search, an HTTP 2xx (success) status code doesn't guarantee indexing." Some 2xx pages look like errors, such as an empty page or an error message. Search Console reports those as soft 404s.
>
> Sources: [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)

## 3xx

> **Officially documented:**
> For a 301, Google follows the redirect and uses it "as a strong signal that the redirect target should be processed". For a 302, Google follows it and uses it as a weak signal.
>
> Sources: [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)

## 4xx

> **Officially documented:**
> Google doesn't index URLs that return a 4xx status code. Already indexed URLs that start returning one are removed. All 4xx errors except 429 are treated the same and have no effect on crawl rate. So Google advises: "don't use 401 and 403 status codes for limiting the crawl rate."
>
> Sources: [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)

## 429 and 5xx

> **Officially documented:**
> "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling." Already indexed URLs stay in the index for a while but are eventually dropped. Once the server returns 2xx again, Google gradually increases the crawl rate.
>
> Sources: [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)

## For AI search

Google's AI features draw on the same index ([How Google Search works](https://geo.casa/foundations/how-google-search-works/)). A page that fails with 4xx or prolonged 5xx drops out of AI Overviews along with classic results. Bot-protection layers that answer crawlers with 403 or 429 have the same effects as these errors. See [AI search access audit](https://geo.casa/playbooks/ai-search-access-audit/).

## Sources

- [Google: How HTTP status codes affect Google's crawlers](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)