# AI search access audit

> Before any content work, confirm AI search systems can reach a page. Run six documented checks: indexing and snippet eligibility, robots.txt, CDN and firewall, snippet restrictions, server-rendered content, and verified crawler visits in the logs.

- URL: https://geo.casa/playbooks/ai-search-access-audit/
- Guide · Officially documented
- Reviewed: 2026-10-25

## Before you start

- Pick a handful of representative URLs (a key guide, a product page, a hub) rather than the whole site.
- You need access to Google Search Console, the live `robots.txt`, CDN or firewall settings, and server or CDN logs.

## 1. Indexed and eligible for a snippet

> **Officially documented:**
> To appear as a supporting link in AI Overviews or AI Mode, a page "must be indexed and eligible to be shown in Google Search with a snippet". There are no additional technical requirements.
>
> Sources: [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

Check with Search Console's URL Inspection tool. Not indexed means stop here and fix indexing first.

## 2. Search crawlers allowed in robots.txt

Each AI search product uses its own crawler token, and blocking it removes you from that product. Confirm the tokens you care about are allowed: Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. Details per token: [Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/).

> **Officially documented:**
> OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
>
> Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)

## 3. Not blocked by CDN, firewall or hosting

> **Officially documented:**
> Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" among the practices that still matter for AI features.
>
> Sources: [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

Bot-protection rules often return `403` or a challenge page to crawlers that robots.txt allows. Check firewall logs for blocked requests from verified crawler IPs (step 6).

## 4. No unintended exclusions or snippet restrictions

> **Officially documented:**
> `nosnippet` prevents content from being used as a direct input for AI Overviews and AI Mode, and `max-snippet` limits how much may be used.
>
> Sources: [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)

In Search Console, check **Settings › Search generative AI**. If the site is set to exclude, it won't appear in AI Overviews or AI Mode at all ([Search generative AI control](https://geo.casa/ai-search/search-generative-ai-control/)). Child properties can inherit that setting from a parent.

Then search the HTML and response headers of each URL for `noindex`, `nosnippet`, `max-snippet` and `data-nosnippet`. Templates and plugins add these more often than people expect. See [Snippet controls for AI Overviews and AI Mode](https://geo.casa/crawling-indexing/snippet-controls/).

```sh
curl -sI https://example.com/page/ | grep -i x-robots-tag
curl -s  https://example.com/page/ | grep -iE 'name="(robots|googlebot)"|data-nosnippet'
```

## 5. Main content in the server HTML

> **Officially documented:**
> Google recommends server-side or pre-rendering because "not all bots can run JavaScript".
>
> Sources: [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)

Fetch each URL without JavaScript and confirm the main text, headings and links are there. Background: [JavaScript rendering and AI crawlers](https://geo.casa/implementation/javascript-rendering-ai-crawlers/).

## 6. Crawlers actually visit

> **Officially documented:**
> Google's crawlers can be verified with a reverse DNS lookup followed by a forward lookup, or against Google's published IP lists.
>
> Sources: [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)

Count only verified visits ([Verify AI crawler traffic in server logs](https://geo.casa/implementation/verify-ai-crawlers/)). If a search crawler you allow has no verified visits, something earlier in this list is probably still blocking it.

## Record the result

| Check | Pass / fail | Notes |
|---|---|---|
| 1. Indexed, snippet-eligible | | |
| 2. robots.txt allows search crawlers | | |
| 3. CDN / firewall allows them | | |
| 4. No AI exclusion or unintended snippet rules | | |
| 5. Content in server HTML | | |
| 6. Verified crawler visits | | |

## Common mistakes

- Starting with content changes while a crawler is blocked at the firewall.
- Reading "allowed in robots.txt" as "can access". Steps 3 and 6 exist because the two often differ.
- Treating a pass as a promise of citations. These checks are preconditions. Google states that meeting requirements does not guarantee crawling, indexing or serving.

## Sources

- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- [Google Search Console Help: Search generative AI control - Search Console Help](https://support.google.com/webmasters/answer/16908024)
- [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Google: Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)