Web Bot Auth
Definition
Web Bot Auth is a cryptographic authentication system that lets a crawler or AI agent prove its identity to a website using a digitally signed HTTP request, instead of relying on a user agent string or IP range that anyone can fake. It is built on HTTP Message Signatures (IETF RFC 9421), with a web-specific specification published in 2025 and adopted by Cloudflare as part of its Verified Bots program starting in mid-2025.
Technically, a bot publishes a public key at a standard, well-known location on its own domain, then signs every request it sends with the matching private key. The receiving server, or the CDN in front of it, verifies that signature against the published key and can check when it was created and when it expires, which prevents both spoofing (an impostor claiming to be GPTBot or ClaudeBot) and replay attacks (reusing an old, captured signature).
This matters directly for AI crawlers management: without cryptographic proof, a site operator managing access through robots.txt or a llms.txt file has to trust that a request claiming to be an AI crawler actually is one, which is exactly the weakness scrapers and impersonators exploit. Web Bot Auth closes that gap by making bot identity provable rather than declared.
For sites investing in AI visibility, this shifts the access-control conversation from blanket allow or block rules toward selective, verifiable trust: a publisher can confidently allow a verified ChatGPT or Perplexity crawler through while still blocking an unverified scraper claiming the same identity. Checking whether crawler access rules on a site are built around this kind of verifiable trust is part of what BotRank's technical audits cover.
Examples
A publisher notices scraper traffic spiking under a user agent string that claims to be an AI search crawler, while its real traffic from that same AI engine is verified separately through Cloudflare's Verified Bots program. Web Bot Auth lets the publisher tell the two apart with cryptographic certainty instead of guessing from the header alone.
Frequently Asked Questions
Why can't a site just trust a crawler's user agent string?
A user agent string is just a line of text a request sends, and anyone can set it to claim to be GPTBot, ClaudeBot, or any other crawler, with no verification required. This makes user agent strings unreliable for distinguishing a real AI crawler from a scraper impersonating one, which is the exact gap Web Bot Auth is designed to close with cryptographic proof.
What standard does Web Bot Auth build on?
It builds on HTTP Message Signatures, standardized by the IETF as RFC 9421, with a web-specific Web Bot Auth draft specification published in 2025. Cloudflare integrated it into its Verified Bots program in mid-2025, and the IETF has been discussing forming a dedicated working group to further standardize the approach.
Does adopting Web Bot Auth mean allowing every AI crawler through?
No. Verifying a bot's identity and deciding whether to allow it access are two separate steps. Web Bot Auth only proves that a request genuinely comes from the crawler it claims to be; a site owner still decides, through robots.txt and similar rules, which verified crawlers it wants to allow and which to block.
How does Web Bot Auth relate to llms.txt?
They solve different problems. <a href="/glossary/llms-txt">llms.txt</a> tells AI crawlers how to read and prioritize a site's content once they arrive. Web Bot Auth proves that the crawler making the request is genuinely who it claims to be before that content is ever served, which makes it a security and trust layer rather than a content-guidance one.
