# GEO best practices for documentation sites and developer products > AI tools quote docs they can fetch as plain text and match to the right version. Allow search and user-fetch crawlers, serve every page as server-rendered HTML with a Markdown twin, keep one canonical URL per version, and treat llms.txt as a cheap convenience rather than a ranking lever. - URL: https://geo.casa/best-practices/developer-docs/ - Guide · Officially documented - Reviewed: 2026-10-28 This guide is for people who run a documentation site, an API reference or a developer product. Developers now ask ChatGPT, Claude, Perplexity and coding assistants before they open your docs. The goal here is that those answers quote the current, correct page. Most of the work is a one-off engineering task of a few days, followed by habits in the release process. ## The short version 1. Let AI search crawlers and user-triggered fetchers reach the docs. 2. Put the full text of every page in the server HTML. 3. Offer a clean Markdown version of each page. 4. Give each docs version one canonical URL and make "latest" obvious. 5. Write code examples and reference pages that survive being quoted out of context. 6. Publish changes in a dated, linkable changelog and keep sitemaps honest. 7. Add llms.txt if your platform makes it cheap, and don't expect it to move rankings. ## 1. Let the right crawlers in Docs sites often sit behind the same bot protection as the app. That is the most common reason a good page never shows up in AI answers. Two kinds of bots matter: search crawlers that index pages ahead of time, and user-triggered fetchers that load a page when someone asks about it. > **Officially documented:** > OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." > > Sources: [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) > **Officially documented:** > Anthropic separates ClaudeBot (training), Claude-User (user-requested fetches) and Claude-SearchBot (search indexing). Blocking Claude-User or Claude-SearchBot may reduce visibility in Claude. > > Sources: [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) > **Officially documented:** > PerplexityBot surfaces and links websites in Perplexity search results. Perplexity-User fetches pages when users ask questions and generally ignores robots.txt rules, because a user requested the fetch. > > Sources: [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) - Allow the search tokens: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot. - Decide on training crawlers (GPTBot, ClaudeBot, Google-Extended) separately. Blocking them does not remove you from search. - Check the CDN or WAF, not only robots.txt. Token-by-token details are in [Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/); the full check is in [GEO best practices for technical and SEO leads](https://geo.casa/best-practices/technical-lead/). ## 2. Put the text in the HTML Many docs frameworks render navigation and content in the browser. A crawler that does not run JavaScript then sees an empty shell. > **Officially documented:** > Google recommends server-side or pre-rendering because "not all bots can run JavaScript". > > Sources: [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) > **Observed in practice:** > In tests published in December 2024, Vercel and MERJ found that none of the major AI crawlers they examined rendered JavaScript, including OpenAI's, Anthropic's and Perplexity's. Gemini, which uses Googlebot's infrastructure, did. > > Sources: [Vercel & MERJ: The rise of the AI crawler (17 Dec 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler) Check with `curl -s ` and search the output for a sentence from the body and a line of a code sample. Tabs, accordions and "copy" widgets should keep their content in the HTML, hidden with CSS rather than fetched on click. More in [JavaScript rendering and AI crawlers](https://geo.casa/implementation/javascript-rendering-ai-crawlers/) and [JavaScript sites (Next.js, Vue and others): making sure AI crawlers can read your content](https://geo.casa/best-practices/javascript-sites/). > If curl can't see your code sample, neither can most AI crawlers. ## 3. Offer a Markdown version of each page Markdown removes navigation, scripts and layout, so an assistant gets only the content. Several docs platforms now generate it automatically. > **Officially documented:** > Mintlify automatically hosts `llms.txt` and `llms-full.txt` at the root of a docs project. "Page links in the llms.txt file include a .md extension so AI tools can fetch the Markdown version of each page directly." > > Sources: [Mintlify: llms.txt](https://www.mintlify.com/docs/ai/llmstxt) > **Officially documented:** > GitBook serves `/llms.txt`, which lists every published page with a Markdown version, and `/llms-full.txt` with the full documentation in one file. Appending `.md` to any published page URL returns that page as Markdown. > > Sources: [GitBook: LLM-ready docs](https://gitbook.com/docs/publishing-documentation/llm-ready-docs) If you build your own site, follow the same pattern: `page-url.md` returns the source Markdown with headings and fenced code blocks intact. Keep it identical to the HTML page. A Markdown copy that drifts from the HTML is a second, stale version of your docs. Add a "Copy as Markdown" or "Open in ChatGPT/Claude" button only if it uses the same file. ## 4. One canonical URL per version Versioned docs create near-duplicate pages: `/v2/auth`, `/v3/auth`, `/latest/auth`. If an assistant quotes the wrong one, users copy deprecated code. > **Officially documented:** > Google ranks canonical signals by strength. Redirects and `rel="canonical"` are strong signals, and sitemap inclusion is weak. Combining them increases the chance your preferred URL is chosen. > > Sources: [Google Search Central: How to specify a canonical URL with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) - Pick one stable URL for the current version (for example `/docs/auth`) and point `rel="canonical"` there from `/latest/` aliases. - Keep old versions online if users need them, but say so in the page title and first line: "Version 2 (deprecated). Current: v3." - List only current-version pages in the sitemap and in llms.txt. - Put the version number and "last updated" date in visible text, not only in a dropdown. Background: [Canonical URL signals](https://geo.casa/crawling-indexing/canonical-signals/). ## 5. Write pages that survive being quoted AI answers lift a paragraph or a code block and drop the rest. This section is our editorial advice, based on how docs get quoted, not on any platform's documentation. - Make every code example complete: imports, package name, install command and version. - State prerequisites and the version near the top of each page, not only on an intro page. - Use one name per concept. If the SDK method is `createClient`, don't call it "the init call" elsewhere. - Give each error message its own heading or table row, with the exact error string. Developers paste errors into assistants. - Write a one-sentence description under each H1 that says what the page is for. The general writing guidance is in [For content editors: writing articles AI answers can quote](https://geo.casa/best-practices/content-editor/) and [Claim: content must be chunked and rewritten for AI search](https://geo.casa/content-structured-data/chunking-content/). > Docs get quoted one block at a time. Each code sample should run on its own. ## 6. Date your changes and keep sitemaps honest When an API changes, the old answer lives on in assistants until the new page is crawled again. A dated changelog with one URL per release gives crawlers and users a single place to learn what changed. > **Officially documented:** > Google ignores `` and `` values. It uses `` "if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate". > > Sources: [Google Search Central: Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) - Give each release note its own anchor or page, with the date and affected versions. - Link from the changelog entry to the updated reference page, and back. - Set `` only when the content changes, not on every build. See [XML sitemaps](https://geo.casa/crawling-indexing/xml-sitemaps/) and [XML sitemaps vs IndexNow](https://geo.casa/tools-protocols/sitemaps-vs-indexnow/). - When you deprecate something, edit the old page to say so. Don't just delete it. ## 7. llms.txt: useful, but not a ranking lever llms.txt is a proposed index file for language models. For docs it is a reasonable convenience, especially when coding tools or users load it on purpose. It is not a way to rank in AI search. > **Officially documented:** > `/llms.txt` is a proposal (llmstxt.org, 2024) for a Markdown file at a site's root. It gives a short summary of the site and links to key pages, optionally with Markdown versions. The aim is to help language models use the site at inference time. > > Sources: [llmstxt.org: The /llms.txt file](https://llmstxt.org/) > **Officially documented:** > Google Search does not use llms.txt files. Maintaining them for other services "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." > > Sources: [Google Search Central: Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) > **Observed in practice:** > A 2026 study by DATA17 checked llms.txt on 163 brand domains in one product category. Brands one AI assistant mentioned had deployed llms.txt less often (17%) than brands it never mentioned (27%). With only 18 brands mentioned, the authors read this as no measurable positive effect. > > Sources: DATA17 AI Research Lab · 解数智能: 你的公司还在 AI 的答案里吗：企业 GEO 行动指南 · 2026 (unpublished report, not publicly available; readers can't check it) If your platform generates it, leave it on. If you build it by hand, generate it from the same source as the sitemap so it can't go stale. Full evidence: [Claim: llms.txt improves AI search visibility](https://geo.casa/tools-protocols/llms-txt/). > Ship llms.txt for the developers who ask for it. Don't ship it expecting citations. ## What not to do - **Don't block all bots at the docs CDN.** Search and user-fetch crawlers are separate from training crawlers ([Configure robots.txt for AI crawlers](https://geo.casa/crawling-indexing/robots-txt-ai-crawlers/)). - **Don't serve different content to AI bots.** Keep the HTML, Markdown and llms.txt in sync from one source. Google treats creating separate content mainly to game AI answers as risky ([Scaled content abuse](https://geo.casa/content-structured-data/scaled-content-abuse/)). - **Don't let `/latest/` and `/vN/` compete.** Choose a canonical (section 4). - **Don't put examples only in images or interactive sandboxes.** Keep a text version on the page. - **Don't treat llms.txt as the GEO project.** It is one small item ([Claim: llms.txt improves AI search visibility](https://geo.casa/tools-protocols/llms-txt/)). ## How you'll know it's working - **Logs:** count verified visits from OAI-SearchBot, Claude-SearchBot, PerplexityBot and user-fetch agents to docs URLs, including `.md` URLs ([Verify AI crawler traffic in server logs](https://geo.casa/implementation/verify-ai-crawlers/)). - **Bing AI Performance:** see which docs pages Copilot and Bing AI cite ([Bing AI Performance report](https://geo.casa/measurement/bing-ai-performance/)). > **Officially documented:** > Microsoft: "AI Performance does not measure rankings, authority, performance, or importance. It simply shows which content was cited." > > Sources: [Microsoft Bing: AI Performance report (Bing Webmaster Tools help)](https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c) - **Prompt panel:** keep 20–30 real developer questions ("how do I paginate with the X SDK") and run them monthly. Check whether answers cite your docs and use the current version ([Measuring AI visibility by sampling prompts](https://geo.casa/measurement/ai-visibility-measurement/)). - **Support tickets:** tag tickets where the user followed an outdated AI answer. A falling count is the outcome that matters. ## Evidence at a glance | Recommendation | Evidence | |---|---| | Allow search and user-fetch crawlers | Official (OpenAI, Anthropic, Perplexity) | | Server-rendered HTML | Official (Google) and observed (Vercel/MERJ, 2024) | | Markdown version of each page | Official platform features (Mintlify, GitBook); effect on citations untested | | One canonical per version | Official (Google canonical signals) | | Self-contained code examples and error pages | Hypothesis (our editorial advice) | | Accurate `lastmod` and dated changelog | Official (Google sitemaps) for `lastmod`; changelog structure is our advice | | llms.txt | Official: ignored by Google Search; observed: no measurable effect in one study | ## Sources - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) - [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) - [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) - [Vercel & MERJ: The rise of the AI crawler (17 Dec 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler) - [Google Search Central: How to specify a canonical URL with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) - [llmstxt.org: The /llms.txt file](https://llmstxt.org/) - [Google Search Central: Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) - DATA17 AI Research Lab · 解数智能: 你的公司还在 AI 的答案里吗：企业 GEO 行动指南 · 2026 (unpublished report, not publicly available; readers can't check it) - [Mintlify: llms.txt](https://www.mintlify.com/docs/ai/llmstxt) - [GitBook: LLM-ready docs](https://gitbook.com/docs/publishing-documentation/llm-ready-docs) - [Google Search Central: Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) - [Microsoft Bing: AI Performance report (Bing Webmaster Tools help)](https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c)