What is web-bot-auth?

The problem web-bot-auth solves is familiar to anyone who has run a large site: any request can claim to be Googlebot or GPTBot, because a user-agent is free text anyone can fake. Until now, verification leaned on IP lists and reverse DNS checks, manual upkeep that breaks silently when a range rotates.

The protocol, developed under Cloudflare and Google’s lead on top of RFC 9421, works differently: the bot operator holds a private key and signs every HTTP request with dedicated headers. The site fetches the operator’s public key from a well-known address and verifies the signature. Faking an identity requires the private key, which makes it practically impossible.

A live example: OpenAI’s ChatGPT agent signs every request it sends today, and sites behind Cloudflare get that verification automatically, without a line of code.

How it relates to SEO and GEO

AI visibility starts with the right bots getting in. As sites block bots more aggressively, the risk grows that blanket blocking also hits the bots feeding answers in ChatGPT and Gemini, including the fetchers that arrive a moment before a citation. Web-bot-auth enables precise policy: admit verified AI-engine bots, block impersonators, and stop losing citations to blind blocking. Anyone running technical SEO should know this layer the way they know robots.txt.

The difference between web-bot-auth and robots.txt

robots.txt is a polite request that relies on mutual respect, and the standard itself states it is not an authorization mechanism. Web-bot-auth is cryptographic proof of identity. The first answers “what may you do,” the second “who are you at all.” A serious site will need both.