# Crawling & Indexing

> How search engines and AI crawlers discover, fetch and index pages — robots.txt, canonical URLs, hreflang and sitemaps, each checked against official documentation.

- URL: https://geo.casa/crawling-indexing/
- Hub
- Reviewed: 2026-10-26

Every AI answer that cites a web page starts with a crawler fetching it and an index storing it. For Google's AI features, the documented page-level requirement is being indexed and eligible for a snippet. The site must also be included in Search's generative AI features, the default setting ([Google AI Overviews](https://geo.casa/ai-search/google-ai-overviews/)). That makes this section the foundation for the rest of the wiki.

## Learning path

1. [robots.txt](https://geo.casa/crawling-indexing/robots-txt/) — what crawlers may fetch, and why blocking is not the same as de-indexing.
2. [Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/) — separate decisions for AI search and AI training crawlers.
3. [Canonical URL signals](https://geo.casa/crawling-indexing/canonical-signals/) — which URL represents a set of duplicates.
4. [Set up hreflang for translated pages](https://geo.casa/crawling-indexing/hreflang/) — tie translations together without making them duplicates.
5. [XML sitemaps](https://geo.casa/crawling-indexing/xml-sitemaps/) — help discovery, and what Google ignores in sitemaps.
6. [Keep a page out of Google](https://geo.casa/crawling-indexing/keep-out-of-google/) — remove a page from results without the robots.txt trap.
7. [Snippet controls for AI Overviews and AI Mode](https://geo.casa/crawling-indexing/snippet-controls/) — limit what AI Overviews and AI Mode use, without leaving Search.
8. [Google-Extended](https://geo.casa/crawling-indexing/google-extended/) — opt out of Gemini training, which does not affect Search.
9. [Crawl budget](https://geo.casa/crawling-indexing/crawl-budget/) — which sites need to think about it (most don't).
10. [Move a site to new URLs](https://geo.casa/crawling-indexing/site-moves/) — move to new URLs or a new domain without losing what you've built.
11. [Mobile-first indexing](https://geo.casa/crawling-indexing/mobile-first-indexing/) — Google indexes the mobile version; keep it complete.
12. [Claim: Duplicate content gets your site penalized](https://geo.casa/crawling-indexing/duplicate-content-penalty/) — why duplicate pages are filtered, not penalized.
13. [URL structure](https://geo.casa/crawling-indexing/url-structure/) — crawlable, readable URLs, and parameters that multiply them.

## Sources

- [Google Search Central: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Google Search Central: How to specify a canonical URL with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls)
- [Google Search Central: Tell Google about localized versions of your page](https://developers.google.com/search/docs/specialty/international/localized-versions)