llms.txt
Definition
llms.txt is a plain-text file placed at the root of a website (for example botrank.ai/llms.txt) that gives large language models and AI crawlers a curated, markdown-formatted map of the site's most important pages. Unlike a sitemap.xml, which lists every URL for search engine indexing, llms.txt is written for AI systems that need to understand what a site is about and where to find authoritative answers, in a format that fits inside a limited context window.
A well-built llms.txt typically opens with a short description of the brand or product, then groups links by topic under markdown headings, such as documentation, product pages, or a glossary. Each link is paired with a one-line description, so an AI system can decide which pages are worth fetching without having to crawl the entire site first.
llms.txt was proposed in 2024 and is not yet an official web standard recognized by every AI provider, and there is no public confirmation that ChatGPT, Perplexity or Google's AI systems currently use it to influence citations. It should be treated as a low-cost, forward-looking addition to a site's AI search visibility setup, not a replacement for solid content, clean HTML structure and schema markup.
llms.txt works best alongside proper crawler access: an AI system still needs permission to fetch a page, described by AI crawlers, before an llms.txt entry is of any use. BotRank's technical audits check for the presence and structure of an llms.txt file as part of a site's AI-readiness review.
Examples
A SaaS company adds an llms.txt file listing its top 15 documentation pages and its pricing page, each with a one-line summary, so an AI assistant answering a product question can go straight to the right page instead of guessing from a generic crawl.
Frequently Asked Questions
Is llms.txt the same as robots.txt or sitemap.xml?
No. robots.txt controls crawler permissions and sitemap.xml lists URLs for indexing; llms.txt is a curated, human-and-AI-readable summary meant to help language models quickly understand what a site offers and which pages matter most.
Do I need llms.txt to appear in ChatGPT or AI Overviews?
No confirmed AI system currently requires it to cite a site, and none has publicly confirmed using it as a ranking input. It is a low-effort addition, not a prerequisite for AI visibility.
What should an llms.txt file include?
A short description of the site or brand, followed by markdown-formatted sections grouping links by topic (docs, product, glossary, etc.), each with a one-line description.
Does llms.txt replace schema markup?
No, they serve different purposes. Schema markup structures data for machines at the page level; llms.txt gives a site-level index. The two are complementary.
