What is web-bot-auth?
Nati Elimelech·Updated 04.09.2026 web-bot-auth is an emerging protocol at the IETF: bots cryptographically sign requests, sites verify the sender.
The problem web-bot-auth solves is familiar to anyone who has run a large site: any request can claim to be Googlebot or GPTBot, because the user-agent is free text anyone can fake. Until now verification leaned on IP lists and reverse DNS checks, manual maintenance that breaks silently when a range changes. web-bot-auth replaces that layer with a signature: a private key held by the bot operator, a public key visible to everyone, and a signature on requests that the site verifies against the public key.
How web-bot-auth works
The protocol originated at Cloudflare, and the current drafts are co-authored with Google, on top of RFC 9421, a published IETF standard for signing HTTP messages. The bot operator signs requests in three headers: Signature-Agent, which points at the operator’s key directory, Signature-Input, which states what was signed and when the signature expires, and Signature itself.
Signature-Agent: sig1="https://example-bot.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1753500000;expires=1753500300;keyid="...";tag="web-bot-auth"
Signature: sig1=:BASE64_SIGNATURE:The site reads the address from Signature-Agent, fetches the public key from it at the fixed path /.well-known/http-message-signatures-directory, and verifies the signature. Typing a header no longer proves anything: impersonation requires the operator’s private key.
The format is still settling. The August 2026 draft uses the dictionary form above, while Cloudflare’s current verifier documentation still expects the older structured-string form, for example "https://chatgpt.com", and rejects the newer dictionary syntax. That is normal for an active draft, but it matters if you are writing rules by hand.
web-bot-auth is already working in the field before it is an RFC: the IETF working group was chartered in October 2025, and the core protocol is now a working-group Internet-Draft. The public key directories are already up too. A GET to /.well-known/http-message-signatures-directory returns public-key JSON today on both chatgpt.com and agent.bot.goog. The full adoption picture, company by company, is in the web-bot-auth guide.
How web-bot-auth relates to SEO and GEO
As sites block bots more aggressively, the risk grows that blanket blocking also hits the bots that bring citations, including user-triggered fetchers. web-bot-auth enables precise policy instead of all-or-nothing: admit signed bots, and block impostors that send the same user-agent without a signature.
The difference between web-bot-auth and robots.txt
robots.txt is a polite request: RFC 9309, the standard that defines it, states outright that its rules are not a form of access authorization. web-bot-auth is cryptographic proof of identity. robots.txt answers “what may you do”. web-bot-auth answers “who are you in the first place”. robots.txt cannot identify an impostor, and web-bot-auth does not say where the bot may go.
Questions about web-bot-auth
Should I block bots that do not sign their requests?▼
Blocking bots that do not sign is a bad rule today, because almost no crawler signs. Googlebot, Bingbot, GPTBot and OAI-SearchBot send unsigned requests, and a sign-or-block rule would shut out most search and AI crawling.
A signature is useful as an allow rule, not a block rule: prefer a signed bot, and keep verifying the rest against the companies' official IP lists and a reverse DNS check.
Is web-bot-auth an official standard yet?▼
web-bot-auth is not an RFC yet: the IETF working group was chartered in October 2025, and on September 1, 2026 the core protocol became a Web Bot Auth working-group Internet-Draft. What is final is the foundation, RFC 9421, the HTTP message signatures standard the protocol rests on.
Meanwhile adoption in the field is running ahead of standardization: the major CDNs already verify signatures in production.
Who uses web-bot-auth today?▼
On the signing side: ChatGPT's browser in agent mode, which runs in OpenAI's cloud and identifies as chatgpt.com, Google-Agent, a subset of whose requests Google signs experimentally, and Amazon's Bedrock AgentCore, in preview and off by default. On the verification side: Cloudflare, Akamai, Vercel, HUMAN, and AWS WAF on CloudFront distributions, among others.
What is not happening yet: today this is a protocol for agents, not for search and training crawlers.
Can a web-bot-auth signature be faked?▼
Faking a web-bot-auth signature without the private key is not possible.
But the protocol is not airtight: under the draft, guarding against a signed request being reused (the nonce check) is entirely the receiving site's job, and the site is allowed to skip it. Cloudflare, for example, skips it and relies on the signature expiring quickly. And a leaked key requires rotation, like any key.
Do I need to implement web-bot-auth on my site?▼
Implementing web-bot-auth yourself is still unnecessary in most cases: if the site sits behind Cloudflare, Akamai or Vercel, verification already happens in the infrastructure with no code on your side. It is worth knowing the protocol and tracking adoption.