Optimizing Robots.txt for AI SEO
Description
The robots.txt file is an exclusion protocol that tells crawlers which parts of the site to crawl or ignore.
Placed at the root of the domain, it defines access rules for each user agent, optimizes the crawl budget, and protects sensitive resources from indexing.
Why is this important for ai search?
LLMs use specific robots (GPTBot, PerplexityBot, ClaudeBot) to collect information.
A poorly configured robots.txt can block access to relevant content, preventing it from being cited in generated answers.
An optimal configuration allows models to access quality content while protecting sensitive data.
Technical details
- Robot.txt File Accessibility
- File Format
- AI Bot Specific Guidelines
- Rules for Important Sections and Pages
1. Presence and Accessibility of the robots.txt File
The robots.txt file must be present and accessible at the root of the domain. This is the first crucial step for crawlers, including those from generative AI engines, to discover and interpret your crawling directives.
- Location: The robots.txt file must be located at the root of the domain. For example, for the domain example.com, the file must be accessible via https://example.com/robots.txt.
- HTTP/HTTPS Accessibility: The file must be accessible via both HTTP and HTTPS protocols. It is recommended to ensure that the HTTPS version is the canonical version and that any HTTP requests are redirected to HTTPS.
- HTTP Status Code: The server must return an HTTP 200 OK status code when requesting the robots.txt file. A 404 Not Found or any other error code will prevent robots from crawling your site according to your instructions.
2. Robots.txt File Format (MIME Type)
The robots.txt file must be served with the correct MIME type to ensure that crawlers treat it as a plain text file.
- MIME Type: The server must return the robots.txt file with the MIME type text/plain. Any other MIME type, such as text/html, may cause robots to misinterpret or reject the file.
- Encoding: The file must be encoded in UTF-8 to ensure compatibility with all characters and avoid parsing issues.
- Minimum Content: The robots.txt file must not be empty. At a minimum, it should contain User-agent: * and Allow: / directives to indicate that all bots are allowed to crawl the entire site. This sets a clear baseline for future guidelines.
3. AI Bot Specific Guidelines
For GEO, it is imperative to include specific guidelines for artificial intelligence bots. These bots are used by generative AI engines to collect data and train their models. By explicitly managing them, you control the visibility of your content in these environments.
GPTBot: This bot is used by OpenAI to train its models, including ChatGPT. You can allow or block its access to specific sections of your site.
User-agent: GPTBot
Allow: /
User-agent: GPTBot
Disallow: /
GoogleOther: This is a generic Google crawler used by various product teams to retrieve public content. It is distinct from Googlebot, which is primarily used for indexing traditional web search.
User-agent: GoogleOther
Allow: /
User-agent: GoogleOther
Disallow: /
It is recommended that you allow these bots to crawl the sections of your site that you want to appear in generative AI responses, unless you have specific reasons to block them.
Beyond GPTBot and GoogleOther, many other robots and AI crawlers are active. For comprehensive GEO optimization, it is recommended that you explicitly manage them. If you don't want to block access to your content for AI training, a non-blocking User-agent: * is a viable alternative.
Google-Extended: This user-agent allows you to control Bard and Vertex AI's access to your content.
User-agent: Google-Extended
Allow: /
User-agent: Google-Extended
Disallow: /
CCBot: Used by Common Crawl, a non-profit organization that builds and maintains an open repository of web data.
User-agent: CCBot
Allow: /
ChatGPT-User: A user-agent linked to ChatGPT, distinct from GPTBot.
User-agent: ChatGPT-User
Allow: /
OAI-SearchBot: Another bot from OpenAI.
User-agent: OAI-SearchBot
Allow: /
PerplexityBot / Perplexity-User: Bots used by Perplexity AI.
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
ClaudeBot / Claude-SearchBot : Bots used by Anthropic for their Claude model.
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: * : If you want to allow crawling by all bots by default, including those not explicitly listed, you can use the generic User-agent: * directive.
User-agent: *
Allow: /
It is crucial not to accidentally block important bots with a User-agent: * if you have specific Disallow directives for other user-agents. The directives are read sequentially and the most specific directive applies.
4. Important Section and Editorial Page Rules
To maximize your content's visibility in generative AI engines, it's essential to ensure that important sections and editorial pages on your site are accessible to crawlers. This includes blog posts, product pages, service pages, and any other pages containing valuable information that you want to appear in AI responses.
- Allow Directive: Use the Allow directive to specify the paths that crawlers are allowed to crawl. By default, if no Disallow directive is present for a path, it is considered allowed. However, it is good practice to explicitly specify important paths, especially if you have more general Disallow directives.
- Avoiding Unintentional Blocks: Carefully check that your Disallow directives do not unintentionally block important sections of content. Disallow directives take precedence over Allow directives if they are more specific.
- Sitemaps: Although robots.txt is not an indexing tool, it is common to include a link to your XML sitemap. This helps bots discover all pages on your site, including those that might not be found by crawling.
Sitemap: https://www.example.com/sitemap.xml
By allowing these sections to be crawled, you enable AI engines to understand and integrate your relevant content into their knowledge bases, increasing your chances of being cited or referenced in their answers.
Resources & Useful links:
Be the answer in AI search!
Boostez votre visibilité dans les résultats de recherche IA
ChatGPT, Perplexity, Gemini, Mistal, Claude...
