# Verify AI crawler traffic in server logs

> User-agent strings can be faked, so count only verified crawler requests. Check Google with reverse-then-forward DNS or its published IP lists, and Baidu with reverse DNS. OpenAI, Perplexity and Anthropic publish IP lists for their crawlers.

- URL: https://geo.casa/implementation/verify-ai-crawlers/
- Guide · Officially documented
- Reviewed: 2026-10-25

## Before you start

- Access to raw server or CDN logs with the client IP address and user-agent of each request.
- A shell with `host` (or `dig`) for DNS lookups, or a script that can fetch JSON and match IPs against CIDR ranges.

## 1. Google: reverse, then forward DNS

> **Officially documented:**
> Run a reverse DNS lookup on the IP address from your logs. Check that the domain is `googlebot.com`, `google.com` or `googleusercontent.com`. Then run a forward DNS lookup on that domain name and check that it resolves to the same IP address. For large-scale checks, Google recommends matching IPs against its published IP lists instead.
>
> Sources: [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)

```sh
host 66.249.66.1                              # → crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com          # → 66.249.66.1  (must match)
```

Google publishes the lists as JSON files per crawler category (common crawlers, special-case crawlers, user-triggered fetchers), linked from its verification page.

## 2. OpenAI and Perplexity: published IP ranges

> **Officially documented:**
> OpenAI publishes the IP addresses for each crawler as JSON: `openai.com/searchbot.json` (OAI-SearchBot), `openai.com/gptbot.json` (GPTBot) and `openai.com/chatgpt-user.json` (ChatGPT-User).
>
> Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

> **Officially documented:**
> Perplexity publishes `www.perplexity.com/perplexitybot.json` for PerplexityBot and `www.perplexity.com/perplexity-user.json` for Perplexity-User.
>
> Sources: [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

Each file lists CIDR prefixes. A request is verified when its user-agent names the crawler **and** its IP falls inside one of that crawler's prefixes.

## 3. Anthropic

> **Officially documented:**
> Anthropic's crawler help article links a list of IP addresses its crawlers use. A source IP on that list indicates the crawler is coming from Anthropic. It also warns that blocking by IP "may not work correctly or persistently guarantee an opt-out". IP blocks stop its crawlers from reading robots.txt, so use robots.txt to opt out.
>
> Sources: [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

## 4. Baidu: reverse DNS

> **Officially documented:**
> Baidu recommends a reverse DNS lookup (`host`, `nslookup` or `dig -x` on the IP). The hostname of Baiduspider is `*.baidu.com` or `*.baidu.jp`; Baidu calls any other hostname fake.
>
> Sources: [Baidu: FAQs of Baiduspider](https://help.baidu.com/question?prod_id=99&class=0&id=3001)

```sh
host 123.125.66.120   # → Baiduspider-123-125-66-120.crawl.baidu.com
```

Baidu's FAQ stops at the reverse lookup. Adding the forward lookup used for Google costs nothing and also catches a spoofed PTR record.

## 5. 360 and Toutiao: published IP ranges

> **Officially documented:**
> 360 Search says its crawler does not yet support `nslookup` checks; match the IP against the ranges it publishes instead. Its user agent ends in `360Spider`.
>
> Sources: [360 Search: 360蜘蛛IP](https://www.so.com/help/spider_ip.html)

> **Officially documented:**
> Toutiao Search names its crawler `Bytespider` and listed six /24 IP ranges in November 2019.
>
> Sources: [Toutiao Search: 头条搜索spider介绍](https://www.toutiao.com/i6759451726496940557/)

That list is from 2019 and has no machine-readable version, so treat a mismatch as "unverified", not as proof of a fake.

## Verify

- Spot-check: pick a request from each claimed crawler and run the check by hand.
- In reports, separate *claimed* from *verified* crawler traffic. Unverified requests with a crawler's user-agent are a different population and shouldn't be counted as that crawler.

## Rollback

Verification is read-only. If you added firewall rules based on it, remove them and fall back to robots.txt rules ([Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/)).

## Common mistakes

- Trusting the user-agent string alone.
- Caching IP lists forever. The files carry a `creationTime` and change over time; refresh them on a schedule.
- Blocking unverified "Googlebot" traffic by user-agent string. It blocks nothing once the impostor changes its string. Meanwhile, IP-based blocking of real crawlers can stop them from reading your robots.txt.

## Sources

- [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Baidu: FAQs of Baiduspider](https://help.baidu.com/question?prod_id=99&class=0&id=3001)
- [360 Search: 360蜘蛛IP](https://www.so.com/help/spider_ip.html)
- [Toutiao Search: 头条搜索spider介绍](https://www.toutiao.com/i6759451726496940557/)