# GEO best practices for technical and SEO leads

> AI search can only cite pages it can reach, read and quote. Check indexing, crawler tokens, CDN and firewall rules, snippet controls and server rendering, then confirm verified crawler visits in the logs. Turn the checks into automated tests so a deploy or a CDN default change cannot silently undo them.

- URL: https://geo.casa/best-practices/technical-lead/
- Guide · Officially documented
- Reviewed: 2026-10-28

This guide is for the person who owns crawlability: a technical SEO, a platform engineer or a web lead. It turns the six-step [AI search access audit](https://geo.casa/playbooks/ai-search-access-audit/) into a programme you can own. Expect two or three days for the first full pass and a few hours a month afterwards. You need Search Console, Bing Webmaster Tools, CDN or WAF settings, and raw access logs.

## The short version

1. Confirm key pages are indexed and eligible for snippets in Google and Bing.
2. Set robots.txt per crawler token, separating search from training.
3. Check that the CDN, WAF and hosting let the same crawlers through.
4. Remove unintended snippet limits and AI exclusion settings.
5. Make sure the main content is in the server HTML.
6. Verify real crawler visits in the logs, including Chinese search engines if that market matters.
7. Automate these checks and alert on regressions.

*Each layer can block the next. Work top to bottom.*

## 1. Indexing and snippet eligibility

Every AI search product here sits on top of a search index. If a page is not indexed, nothing downstream matters.

> **Officially documented:**
> To appear as a supporting link in AI Overviews or AI Mode, a page "must be indexed and eligible to be shown in Google Search with a snippet". There are no additional technical requirements.
>
> Sources: [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

- Use URL Inspection on a sample of templates, not only the home page ([URL Inspection tool](https://geo.casa/measurement/url-inspection/)).
- Review the Page indexing report for whole templates dropping out ([Page indexing report](https://geo.casa/measurement/page-indexing-report/), [Diagnose a page not indexed by Google](https://geo.casa/playbooks/page-not-indexed/)).
- Do the same in Bing Webmaster Tools, where the AI Performance report also lives ([Bing AI Performance report](https://geo.casa/measurement/bing-ai-performance/)).

## 2. robots.txt per token

Each AI company now runs several bots with different jobs. Your policy should say which purposes you allow, then map each purpose to tokens.

> **Officially documented:**
> OpenAI states that the OAI-SearchBot and GPTBot settings are independent. A site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Changes take about 24 hours to be reflected in search.
>
> Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

> **Officially documented:**
> Anthropic separates ClaudeBot (training), Claude-User (user-requested fetches) and Claude-SearchBot (search indexing). Blocking the latter two may reduce visibility in Claude.
>
> Sources: [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

> **Officially documented:**
> For Google, AI features are part of Search, so robots.txt rules for Googlebot are the crawl control. Google-Extended applies to some of Google's other AI systems, not to Search.
>
> Sources: [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

> **Observed in practice:**
> An August 2026 audit of robots.txt on 130 company sites found that companies making an explicit choice almost never blocked search crawlers. The authors call blocking search and training crawlers together the most common and most damaging mistake.
>
> Sources: DATA17 AI Research Lab · 解数智能: 你的公司还在 AI 的答案里吗：企业 GEO 行动指南 · 2026 (unpublished report, not publicly available; readers can't check it)

| Purpose | Tokens to review |
|---|---|
| Search indexing | Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User-triggered fetch | ChatGPT-User, Claude-User, Perplexity-User |
| Training | GPTBot, ClaudeBot, Google-Extended |
| Chinese search | Baiduspider, 360Spider, Sogou web spider, Bytespider |

Full syntax per token: [Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/) and [robots.txt](https://geo.casa/crawling-indexing/robots-txt/).

> Write the policy by purpose first, then by token. "Block AI" is not a policy.

## 3. CDN, WAF and hosting

robots.txt says what you allow. The edge decides what actually gets through. Bot-management products now ship AI-specific defaults, and those defaults change.

> **Officially documented:**
> Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" among the practices that still matter for AI features.
>
> Sources: [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

> **Officially documented:**
> Cloudflare's legacy "Block AI bots" setting blocks verified bots classified as crawling for AI training, and excludes mixed-purpose bots used for both training and search. Cloudflare says that from September 15, 2026, new domains get updated defaults: training and agent bots blocked on pages with ads, search bots allowed, and mixed search-and-training crawlers blocked under any AI-training option.
>
> Sources: [Cloudflare: Block AI Bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)

- Export the bot rules, managed rulesets and rate limits that apply to the site.
- For each search token in section 2, confirm it is allowed, not challenged.
- Check firewall logs for `403`, `429` and challenge responses to verified crawler IPs.
- Re-check after any CDN plan change, new zone or provider announcement. See [HTTP status codes and crawlers](https://geo.casa/implementation/http-status-crawlers/).

## 4. Snippet controls and AI exclusion

Templates, plugins and legal requests add snippet restrictions. They can quietly remove a page from AI answers while it still ranks in classic results.

> **Officially documented:**
> `nosnippet` prevents content from being used as a direct input for AI Overviews and AI Mode, and `max-snippet` limits how much may be used.
>
> Sources: [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)

> **Officially documented:**
> If you exclude a site with Search Console's Search generative AI control, its links and content won't appear in AI Overviews, AI Mode and related features. The control does not affect AI training (use Google-Extended) and is not a way to leave Search (use noindex).
>
> Sources: [Google Search Console Help: Search generative AI control - Search Console Help](https://support.google.com/webmasters/answer/16908024)

- Grep HTML and headers for `noindex`, `nosnippet`, `max-snippet`, `data-nosnippet` and `X-Robots-Tag`.
- Check **Settings › Search generative AI** on every Search Console property, including parent domains.
- Keep a list of pages where a restriction is deliberate, with the reason.

Details: [Snippet controls for AI Overviews and AI Mode](https://geo.casa/crawling-indexing/snippet-controls/), [Search generative AI control](https://geo.casa/ai-search/search-generative-ai-control/).

## 5. Rendering

Google renders JavaScript, but most AI crawlers do not. Content that appears only after client-side rendering may be invisible to them.

> **Officially documented:**
> Google recommends server-side or pre-rendering because "not all bots can run JavaScript".
>
> Sources: [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)

> **Observed in practice:**
> In tests published in December 2024, Vercel and MERJ found that none of the major AI crawlers they examined rendered JavaScript. They named OpenAI, Anthropic, Meta, ByteDance and Perplexity crawlers. Gemini, which uses Googlebot's infrastructure, did render JavaScript.
>
> Sources: [Vercel & MERJ: The rise of the AI crawler (17 Dec 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler)

Fetch each template with JavaScript off and diff the main text against the rendered page. Fix by template, not by URL. See [JavaScript rendering and AI crawlers](https://geo.casa/implementation/javascript-rendering-ai-crawlers/) and [JavaScript sites (Next.js, Vue and others): making sure AI crawlers can read your content](https://geo.casa/best-practices/javascript-sites/).

## 6. Verify visits in the logs

User-agent strings are easy to fake. Count only visits you can verify, by DNS or by published IP ranges.

> **Officially documented:**
> Google's crawlers can be verified with a reverse DNS lookup followed by a forward lookup, or against Google's published IP lists.
>
> Sources: [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)

> **Officially documented:**
> OpenAI publishes the IP addresses for each crawler as JSON: `openai.com/searchbot.json` (OAI-SearchBot), `openai.com/gptbot.json` (GPTBot) and `openai.com/chatgpt-user.json` (ChatGPT-User).
>
> Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

For the Chinese market, each engine documents its own method.

> **Officially documented:**
> Baidu recommends a reverse DNS lookup on the IP. The hostname of Baiduspider is `*.baidu.com` or `*.baidu.jp`; Baidu calls any other hostname fake.
>
> Sources: [Baidu: FAQs of Baiduspider](https://help.baidu.com/question?prod_id=99&class=0&id=3001)

> **Officially documented:**
> 360 Search says its crawler does not yet support `nslookup` checks; match the IP against the ranges it publishes instead. Its user agent ends in `360Spider`.
>
> Sources: [360 Search: 360蜘蛛IP](https://www.so.com/help/spider_ip.html)

> **Officially documented:**
> Toutiao Search names its crawler `Bytespider` and listed six /24 IP ranges in November 2019.
>
> Sources: [Toutiao Search: 头条搜索spider介绍](https://www.toutiao.com/i6759451726496940557/)

Build a daily job that tags each log line as verified, unverified or spoofed per token. Method and scripts: [Verify AI crawler traffic in server logs](https://geo.casa/implementation/verify-ai-crawlers/). Platform context for China: [GEO in China: Doubao, Yuanbao, Qwen, ERNIE, DeepSeek and Kimi](https://geo.casa/best-practices/china-platforms/).

> An allowed crawler with zero verified visits is a bug report, not a quiet week.

## 7. Monitoring and regression

Most access failures are regressions: a new WAF rule, a template change, a robots.txt edit in a release. The checks above should run on their own. This section is our recommended setup, not platform documentation.

- **robots.txt diff:** fetch it daily and alert on any change to rules for search tokens.
- **Header and meta test:** in CI, fail a build that adds `noindex` or `nosnippet` to an indexable template.
- **No-JS content test:** in CI, fetch key templates without JavaScript and assert that a known sentence is present.
- **Edge test:** request key URLs from outside your network with each search user agent and alert on `403`, `429` or a challenge page. A spoofed UA test does not prove real crawlers pass, so pair it with log verification.
- **Log dashboard:** verified visits per token per day, with an alert when a token drops to zero.
- **Owner and runbook:** name one owner for crawler access, and document who can change CDN bot settings.

## What not to do

- **Don't block search and training crawlers with one rule.** It is the most common costly mistake (section 2).
- **Don't trust robots.txt alone.** The CDN can override it (section 3).
- **Don't count unverified user agents.** They inflate crawler traffic ([Verify AI crawler traffic in server logs](https://geo.casa/implementation/verify-ai-crawlers/)).
- **Don't block AI bots by IP when robots.txt would do.** Anthropic notes IP blocks stop its crawlers from reading robots.txt ([Verify AI crawler traffic in server logs](https://geo.casa/implementation/verify-ai-crawlers/)).
- **Don't treat a pass as a promise of citations.** These are preconditions ([Google Search Essentials](https://geo.casa/foundations/search-essentials/)).
- **Don't build special files for AI to rank.** llms.txt is ignored by Google Search ([Claim: llms.txt improves AI search visibility](https://geo.casa/tools-protocols/llms-txt/)).

## How you'll know it's working

- All checks green for every key template, recorded with a date ([AI search access audit](https://geo.casa/playbooks/ai-search-access-audit/)).
- Verified visits from each allowed search token, steady or rising.
- Pages appearing in Search Console's generative AI report ([Generative AI performance report: AI Overviews and AI Mode in Search Console](https://geo.casa/measurement/search-console-ai-features/)) and Bing's AI Performance report.

> **Officially documented:**
> Microsoft: "AI Performance does not measure rankings, authority, performance, or importance. It simply shows which content was cited."
>
> Sources: [Microsoft Bing: AI Performance report (Bing Webmaster Tools help)](https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c)

- Citation share in a fixed prompt panel, reported as a trend ([Measuring AI visibility by sampling prompts](https://geo.casa/measurement/ai-visibility-measurement/), [For marketing leads: measuring GEO and reporting it upward](https://geo.casa/best-practices/measure-report/)).

## Evidence at a glance

| Recommendation | Evidence |
|---|---|
| Indexed and snippet-eligible | Official (Google) |
| robots.txt by purpose and token | Official (OpenAI, Anthropic, Google); observed common mistake (DATA17, 2026) |
| CDN and WAF allow search crawlers | Official (Google, Cloudflare) |
| No unintended snippet limits or AI exclusion | Official (Google) |
| Server-rendered main content | Official (Google) and observed (Vercel/MERJ, 2024) |
| Verify crawlers by DNS or IP list | Official (Google, OpenAI, Baidu, 360, Toutiao) |
| Automated regression tests | Hypothesis (our recommended practice) |

## Sources

- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- DATA17 AI Research Lab · 解数智能: 你的公司还在 AI 的答案里吗：企业 GEO 行动指南 · 2026 (unpublished report, not publicly available; readers can't check it)
- [Cloudflare: Block AI Bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- [Google Search Console Help: Search generative AI control - Search Console Help](https://support.google.com/webmasters/answer/16908024)
- [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)
- [Vercel & MERJ: The rise of the AI crawler (17 Dec 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [Baidu: FAQs of Baiduspider](https://help.baidu.com/question?prod_id=99&class=0&id=3001)
- [360 Search: 360蜘蛛IP](https://www.so.com/help/spider_ip.html)
- [Toutiao Search: 头条搜索spider介绍](https://www.toutiao.com/i6759451726496940557/)
- [Microsoft Bing: AI Performance report (Bing Webmaster Tools help)](https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c)