robots.txt

robots.txt is a text file at the root of a website that tells crawlers, AI crawlers included, which parts of the site they may or may not crawl.
Category:
SEO (Search Engine Optimization)
Last update:
November 15, 2026

Definition

robots.txt is a plain-text file served at the root of a domain (example.com/robots.txt) that tells crawlers which URLs they may fetch. It follows the Robots Exclusion Protocol, used informally since 1994 and standardized by the IETF as RFC 9309 in September 2022. The file is organized in groups: one or more User-agent lines naming the crawlers concerned, followed by Allow and Disallow rules on URL paths. It can also declare the location of the XML sitemap.

A few rules decide how the file is read. Crawler names are case-insensitive, paths are case-sensitive. The most specific rule, meaning the longest matching path, wins, and Allow wins a tie. A group that names a crawler replaces the User-agent: * group for that crawler instead of adding to it. The HTTP code matters too: a 4xx response means everything is allowed, a 5xx response means everything is blocked, and crawlers may keep the file in cache for up to 24 hours. Finally, robots.txt controls crawling, not indexing: a blocked URL can still appear in Google without a description. To keep a page out of results, use a noindex directive on a crawlable page.

Since 2023, the file has also become the main control point for AI crawlers. Providers publish separate user-agents for model training, for search and for fetches triggered by a user, so a site can block training while remaining citable in ChatGPT, Claude or Perplexity answers. The reverse mistake is common: blocking OAI-SearchBot or PerplexityBot by accident removes a site from those engines' answers. robots.txt remains a request rather than access control: user-triggered fetchers may not apply it, unscrupulous scrapers ignore it, and Web Bot Auth is emerging to verify who is really crawling. The IETF is also working on AI usage preferences that could be expressed in the file, and Google has announced support for an AI training preference planned for early 2027.

robots.txt works together with llms.txt, which suggests pages to read, and with the sitemap, which lists URLs to discover. BotRank's technical audits check that the file is reachable and that AI search crawlers are not blocked by mistake. For a full walkthrough by provider, read our robots.txt and AI guide.

Official documentation

Examples

After a site migration, the robots.txt of the staging environment, which contains User-agent: * followed by Disallow: /, is pushed to production by mistake. Googlebot and the AI search crawlers stop fetching the site, and within a few days its pages disappear from the sources cited in AI answers. Restoring the right file and monitoring crawler visits in the server logs is the first fix.

Complete example: an annotated robots.txt file

# 1. Search crawlers of AI engines: same exclusions as everyoneUser-agent: OAI-SearchBotUser-agent: Claude-SearchBotUser-agent: PerplexityBotDisallow: /cart/Disallow: /*?sort= # 2. A training crawler: everything blocked except the press areaUser-agent: GPTBotDisallow: /Allow: /press/ # 3. All other crawlersUser-agent: *Disallow: /cart/Disallow: /*?sort= Sitemap: https://www.example.com/sitemap.xml

The first group repeats the exclusions of the general group, because a crawler that has its own group no longer reads User-agent: *. In the second group, Allow: /press/ is longer than Disallow: /, so it wins for that folder. The * wildcard matches any sequence of characters, and $ marks the end of a URL (Disallow: /*.pdf$ blocks every PDF). The Sitemap line applies to the whole file, whatever the group.

Frequently Asked Questions

Does a Disallow rule remove a page from Google?

No. robots.txt prevents crawling, not indexing. A blocked URL can still appear in results, without a description, if other pages link to it. To remove a page, leave it crawlable and add a noindex directive, which the crawler must be able to read.

Should I block AI crawlers in robots.txt?

Distinguish two cases. Blocking training crawlers such as GPTBot or ClaudeBot is a choice about the use of your content and does not affect citations. Blocking search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot removes your pages from the answers of those engines. Most brands that want AI visibility allow search crawlers and decide case by case on training.

Do all bots respect robots.txt?

The declared crawlers of major providers do, but the protocol is voluntary. Fetchers triggered by a user request may not apply it, as OpenAI and Perplexity state for some of their agents, and malicious scrapers ignore it. For real access control, rely on your CDN or firewall rules and on cryptographic verification such as Web Bot Auth.

How do I check my robots.txt?

Open example.com/robots.txt and check that it returns a 200 code in plain text. Google Search Console shows how Google read the file, server logs show which crawlers actually visit, and a technical audit such as BotRank's flags AI crawlers that are blocked by mistake.