Robots.txt and AI: the complete guide to getting cited by ChatGPT, Gemini and Perplexity

Your robots.txt was written for Google, but it now decides whether ChatGPT, Claude or Perplexity can cite you.

One in five publishers tracked by the IPTC blocks OAI-SearchBot, the crawler behind ChatGPT search. Most of them never meant to. They were trying to block AI training. They shut the door on AI answers.

Robots.txt has become the first GEO setting on any site. And the rules are moving: on October 30, 2026, Anthropic, OpenAI, Google and Microsoft adopted a shared signal to opt out of AI training.

In this guide:

• the up-to-date list of AI crawlers to allow or block;
• the 6 reading rules that trip up even SEO teams;
• three ready-to-copy robots.txt files;
• a checklist to make sure everything works.

Key takeaways

• One crawler per purpose. Every AI company runs separate crawlers for training, for its search index and for fetching a page on demand. Block the wrong one and you drop out of AI answers.
• The golden rule. Never block Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot or PerplexityBot. Training (GPTBot, ClaudeBot, Google-Extended...) is a case-by-case decision.
• Blocking GPTBot does not remove you from ChatGPT. OAI-SearchBot decides whether you appear in ChatGPT Search.
• A new shared token. Since October 30, 2026, __usage_ai-training expresses a training opt-out to Anthropic, OpenAI, Google and Microsoft. Rollouts run into 2027.
• The Gemini trap. At Google, opting out of training can also remove you from the Gemini app, but not from AI Overviews.
• The file is not enough. A firewall, a CDN, a server error or JavaScript-rendered content can block AI crawlers even with a perfect robots.txt.

Why robots.txt now needs to be optimized for AI

Because AI engines can only cite what they are allowed to read. ChatGPT, Claude, Perplexity and Copilot send their own crawlers to find the pages they cite. And those crawlers always read robots.txt first.

The problem: many sites block "AI" wholesale. Across the 504 publishers tracked by the IPTC Metawatch observatory, AI search crawlers are blocked almost as often as training crawlers.

These sites thought they were protecting their content. By blocking OAI-SearchBot or Claude-SearchBot, they mostly disappeared from ChatGPT and Claude answers.

Brand websites do no better. Across more than 3,000 sites analyzed by BotRank's GEO Scan between October 2025 and November 2026:

• 6.5% had no accessible robots.txt;
• 8.3% had a robots.txt that failed our AI crawler directives check.

The 3 families of AI crawlers

You no longer block "AI": you choose uses. Every AI company now splits its crawlers into three families. Your robots.txt should treat them differently.

1. Indexing crawlers build the index the engine draws its sources from. Blocking them takes you out of citations.
2. On-demand agents read a specific page when a user asks for it. OpenAI and Perplexity warn that robots.txt may not apply to these visits.
3. Training crawlers collect content for future models. Blocking them does not remove you from real-time answers, except from the Gemini app at Google.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

They are OpenAI's three crawlers, and they are independent:

• GPTBot collects content for training;
• OAI-SearchBot indexes the pages ChatGPT Search can cite;
• ChatGPT-User reads a page when a user asks for it in ChatGPT.

OpenAI says it explicitly: you can allow OAI-SearchBot while blocking GPTBot. The same logic applies at Anthropic with ClaudeBot, Claude-SearchBot and Claude-User.

List of AI crawlers in 2026: which ones to allow or block?

Always allow indexing crawlers and on-demand agents. Make a deliberate call on training crawlers. The tokens below are the ones each company documents as of November 3, 2026. Version numbers change; the token stays.

Indexing crawlers: allow

On-demand agents: allow

Blocking these agents achieves little: most of them can ignore the rule. The ones that honor it only come when a user asks for your page.

Training crawlers: decide

How a crawler reads your robots.txt: the 6 rules to master

Most accidental blocks come from a misunderstood reading rule, not a typo. The format has been set since 2022 by RFC 9309. For the basics of the file, see also our robots.txt technical documentation.

1. One file per host. It lives at the root (https://www.example.com/robots.txt) and every subdomain needs its own. UTF-8, under 500 KiB: Google ignores the rest.
2. A crawler reads a single group. It keeps the group that names it and ignores all the others, including User-agent: *. This is the number one trap.
3. Crawler names ignore case, paths do not. gptbot equals GPTBot, but /Blog/ and /blog/ are two different paths.
4. The longest rule wins. Between Disallow: / and Allow: /press/, Allow applies to /press/. When both are the same length, Allow wins.
5. Only two wildcards. * matches any characters, $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF.
6. The HTTP code matters as much as the content. A 404 means "everything is allowed", a 5xx means "everything is disallowed". The file stays cached for up to 24 hours.

Annotated example

# 1. Shared group: both crawlers read these rulesUser-agent: OAI-SearchBotUser-agent: Claude-SearchBotDisallow: /cart/            # exclusion repeated here (rule 2)Allow: / # 2. GPTBot only reads this group: /cart/ no longer applies to itUser-agent: GPTBotDisallow: /Allow: /press/              # longer than "/": it wins (rule 4) # 3. All other crawlersUser-agent: *Disallow: /cart/Disallow: /*?sort=          # wildcard (rule 5) # 4. Applies to the whole fileSitemap: https://www.example.com/sitemap.xml

Robots.txt examples optimized for AI

The right robots.txt depends on one question: is your content a product you sell, or a way to make your brand known? A brand benefits from being known by AI models, even when they answer without searching the web. A media outlet has more reasons to opt out of training.

Profile 1: maximum visibility

The safest setup is also the simplest. With no crawler-specific group, no AI crawler can be excluded by mistake.

User-agent: *Disallow: /admin/Disallow: /cart/Disallow: /*?sort= Sitemap: https://www.example.com/sitemap.xml

Profile 2: cited in answers, no training

You stay visible in ChatGPT, Claude, Perplexity, Copilot, Vibe and AI Overviews. Training crawlers are refused one by one, then through the shared token.

# --- Indexing and AI agents: allowed ---User-agent: GooglebotUser-agent: BingbotUser-agent: ApplebotUser-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: PerplexityBotUser-agent: Perplexity-UserUser-agent: meta-webindexerUser-agent: MistralAI-IndexUser-agent: MistralAI-UserDisallow: /admin/Disallow: /cart/Allow: / # --- Training: refused crawler by crawler ---User-agent: GPTBotUser-agent: ClaudeBotUser-agent: CCBotUser-agent: meta-externalagentUser-agent: MistralAI-TrainingUser-agent: Applebot-ExtendedDisallow: / # --- Shared token (rolling out in 2026-2027) ---User-Agent: __usage_ai-trainingDisallow: / # --- Google: one token for Gemini training AND the Gemini app ---# "Disallow: /" = training opt-out, but out of the Gemini app.# "Allow: /" = present in Gemini, training accepted.User-agent: Google-ExtendedDisallow: / # --- All other crawlers ---User-agent: *Disallow: /admin/Disallow: /cart/ Sitemap: https://www.example.com/sitemap.xml

Profile 3: selective protection

You accept training on your brand pages, but not on your high-value content. The opt-out is applied path by path.

# Training opt-out limited to high-value contentUser-Agent: __usage_ai-trainingDisallow: /premium/Disallow: /research/ User-agent: GPTBotUser-agent: ClaudeBotUser-agent: CCBotUser-agent: meta-externalagentUser-agent: MistralAI-TrainingUser-agent: Applebot-ExtendedUser-agent: Google-ExtendedDisallow: /premium/Disallow: /research/Disallow: /admin/ User-agent: *Disallow: /admin/ Sitemap: https://www.example.com/sitemap.xml

__usage_ai-training: the new shared token against AI training

Since October 30, 2026, a single rule is enough to opt out of training with Anthropic, OpenAI, Google and Microsoft. The __usage_ai-training token comes from an IETF draft written by Kevin Kelley (Anthropic). It is not a standard yet.

It is written like a crawler name but names a use:

User-Agent: __usage_ai-trainingDisallow: /Allow: /press/

This group changes nothing about crawling: crawlers keep indexing your pages. Conversely, a User-agent: * rule does not count as a training opt-out.

The Gemini trap

• What is not affected: AI Overviews and AI Mode, which rely on Googlebot.
• To stay in the Gemini app: add a User-agent: Google-Extended group with Allow: /. You will then also accept Gemini training: Google ties both uses together.

Beyond robots.txt: Google, Bing, Cloudflare and llms.txt

Robots.txt works per site and per path. For Google and Bing AI answers, the finest controls live elsewhere.

Google: check your Search Console setting

Since August 31, 2026, every site can exclude itself from Google Search's generative AI features (Settings, "Search generative AI"). By default, the site is included. The setting covers the property and its child properties (which can override it). It touches neither the Gemini app nor training.

For GEO, it is a setting to check, never one to switch on by reflex.

Bing: training and AI answers finally separated

Bing lets you opt out of training while staying in Copilot:

<meta name="Bingbot" content="ai-training:disallow">

Bing applies these preferences starting February 1, 2027 (Bing announcement).

Cloudflare: check what it publishes for you

Cloudflare can write your robots.txt for you:

• "Disallow AI Training" opts out of training and lets search crawlers through;
• the "Block" options block the crawlers concerned, search included.

After every change, open the file actually served online.

What about llms.txt?

llms.txt neither blocks nor allows anything: it is a reading map offered to agents. Google states that no AI-specific file is needed to appear in AI Overviews. Publish one if you like, once your robots.txt is in order.

8 mistakes that make you invisible to AI

Most sites missing from AI answers never decided to be. They copied an overly broad list or let a tool decide for them.

1. Copying a "block all AI" list. These lists mix training and search. You target GPTBot and also block OAI-SearchBot and PerplexityBot.
2. Forgetting that the * group no longer applies. A crawler with its own group ignores the general rules. Repeat your exclusions in every group.
3. Letting the file return an error. A 5xx error on /robots.txt means a full block for crawlers that follow RFC 9309.
4. Letting the firewall contradict the file. A CDN or bot protection can return a 403 to an allowed crawler. Google reminds site owners that crawling must be allowed by robots.txt and by the whole infrastructure.
5. Targeting obsolete names. anthropic-ai, Claude-Web or FacebookBot no longer protect anything.
6. Blocking a page to deindex it. A crawler that cannot read the page cannot see its noindex tag either. Leave it crawlable and use the tag.
7. Loading content with JavaScript. Several AI crawlers read HTML without running JavaScript. Client-side text may stay invisible to them (see our analysis of technical audits in the AI era).
8. Believing robots.txt erases the past. A rule only applies going forward. It removes nothing from a model already trained.

How to test your robots.txt for AI in 5 steps

A well-written robots.txt proves nothing until crawlers actually reach your pages. Run this check again after every release.

1. Open the file served online

Open https://www.yoursite.com/robots.txt in a browser. Expected: a 200 code, plain UTF-8 text, under 500 KiB. Do the same for every public subdomain.

2. Test the rules crawler by crawler

The robots.txt report in Search Console flags fetch errors. For each indexing crawler, check that a key URL (homepage, product page, article) is allowed.

3. Simulate a crawler visit

This command reveals a firewall that filters by user-agent. It does not detect IP-based filtering.

curl -I -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.yoursite.com/key-page/

A 403 response signals a block. If you filter by IP, allow the ranges published by each company (for example openai.com/searchbot.json).

4. Read your server logs

Look for visits from OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User. They should receive 200 responses. Allow 24 hours after a change: OpenAI, Meta and Google may keep the old file in cache.

5. Measure the effect on your citations

Record your citations per AI engine before the change, then a few weeks after. BotRank's GEO Scan checks AI crawler access page by page. Multi-LLM visibility tracking then shows the real effect, engine by engine.

The summary checklist

• The file returns a 200, in UTF-8, under 500 KiB
• Every public subdomain has its own robots.txt
• No indexing crawler is blocked on your key pages
• General exclusions are repeated in every group
• The Sitemap: line points to an up-to-date sitemap
• The CDN does not return a 403 to allowed crawlers
• Search Console: "Search generative AI" is set to "Include"
• No AI protection relies on noarchive after March 1, 2027

FAQ: robots.txt and artificial intelligence

What is a robots.txt file?

It is a text file placed at the root of a website. It tells crawlers, from search engines and AI companies alike, which pages they may visit. Its format is standardized by RFC 9309. It expresses a request: reputable crawlers honor it, but it does not technically block anything.

Where is the robots.txt file located?

Always at the root of the domain: https://www.yoursite.com/robots.txt. Each subdomain has its own. On WordPress, the file can be generated automatically. Yoast SEO and Rank Math let you edit it without server access.

Should you block AI crawlers?

Not indexing crawlers: they are what make you citable in ChatGPT, Claude, Perplexity or Copilot. The only real decision is about training. A brand usually benefits from allowing it. A media outlet or a paid-content publisher has more reasons to refuse it.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is only used for training. Your presence in ChatGPT Search depends on OAI-SearchBot. OpenAI states that the two settings are independent.

What is ClaudeBot?

It is Anthropic's crawler that collects content to train Claude models. Blocking it does not remove you from Claude answers: Claude's web search goes through Claude-SearchBot, and on-demand reads go through Claude-User.

Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews and AI Mode rely on Googlebot. Google-Extended controls Gemini training and grounding in the Gemini app and Vertex AI.

How do I opt out of AI Overviews without leaving Google?

With the "Search generative AI" setting in Search Console, open to every site since August 31, 2026. Google states that it is not used as a ranking signal for the rest of Search.

Do all AI companies honor robots.txt?

The major companies commit to it for their indexing and training crawlers. User-triggered fetches (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher) may not apply it. For a strict block, only server-side control is reliable.

How long does a change take to apply?

Allow about 24 hours for the file itself. OpenAI, Meta and Google cache it for up to a day. The effect on citations can take several weeks, while pages are crawled again.

Does llms.txt replace robots.txt?

No. llms.txt offers agents a reading map, without allowing or forbidding anything. Permissions stay in robots.txt.

Main sources

• RFC 9309, Robots Exclusion Protocol
• IETF draft on the __usage_ai-training token
• Google, shared AI training control announcement (October 30, 2026)
• Google, Search generative AI control
• Google, AI features and your website
• OpenAI, Overview of OpenAI Crawlers
• Anthropic, Claude's crawlers
• Microsoft Bing, new training and grounding preferences
• IPTC Metawatch, AI preferences and opt-out signals

Documentation from other companies: Perplexity, Meta, Mistral AI, Cloudflare.