web-bot-auth: The Bot Authentication Protocol Explained

If you manage a website of any real size, you have probably spent time verifying bots. Not the interesting kind of time. The kind where you maintain IP allowlists that go stale, run reverse DNS lookups against ranges that change without notice, and trust user-agent strings that anyone can fake in one line of code.

Key facts:

• web-bot-auth is an IETF protocol for cryptographic bot identity, built on RFC 9421
• Co-authored by Cloudflare and Google, with an IETF working group chartered in October 2025
• OpenAI signs Operator requests with it; Google is experimenting with it for Google-Agent
• Cloudflare, AWS WAF, and Vercel already verify signatures server-side
• It does not replace robots.txt, IP verification, or reverse DNS. It adds a cryptographic verification option alongside them
• Standards-track specs are due to IESG by April 30, 2026

web-bot-auth offers an alternative: a single cryptographic signature on every HTTP request. The bot signs. You verify. Identity established, without maintaining IP lists or running DNS lookups.

The protocol is an IETF draft co-authored by Cloudflare and Google, built on RFC 9421 (HTTP Message Signatures). OpenAI signs Operator requests with it. Google is experimenting with it for Google-Agent. Cloudflare, AWS, and Vercel verify signatures on the server side.

This guide covers how the protocol works, who built it, who is adopting it, and what it means for your site.

The ops tax of bot verification

The IETF charter for webbotauth states plainly that current bot verification methods (IP allowlisting, user-agent strings, and shared API keys) have “significant limitations regarding security, scalability, and manageability.” That is diplomatic. Here is what those limitations look like in practice.

User-agent strings are self-reported. Any HTTP client can claim to be Googlebot, GPTBot, or anything else. There is no verification in the string itself. If you make access decisions based on a user-agent header, you are trusting the visitor’s word for it.

IP allowlists are brittle ops work. Google publishes its crawler IP ranges. So does Bing, so does OpenAI. But those ranges change. Cloud providers share IP space across customers. Maintaining current allowlists across multiple bot operators is ongoing manual labor that scales poorly and breaks silently when a range rotates.

Shared secrets do not scale. You could distribute API keys to every bot operator you want to allow. That works for one or two integrations. It does not work when hundreds of operators need to identify themselves to millions of websites.

web-bot-auth solves all three with asymmetric cryptography. The bot operator holds a private key, signs every request, and publishes the corresponding public key. You verify the signature. If it checks out, the identity is confirmed.

Three headers, one proof

web-bot-auth is a profile built on top of RFC 9421 (HTTP Message Signatures), a general-purpose HTTP signing standard published in February 2024. RFC 9421 defines how to sign arbitrary HTTP message components. The web-bot-auth architecture draft constrains it for bot identity: only asymmetric keys (“Implementations MUST NOT use shared HMAC”), mandatory domain binding, and a fixed tag so servers know the signature is a bot identity claim.

A signed request carries two required headers and one recommended header:

• Signature-Input — declares what was signed and the signature parameters
• Signature — the base64-encoded cryptographic signature
• Signature-Agent — (recommended) a URL pointing to the bot operator’s public key directory

Here is what they look like on a real request:

GET /page HTTP/1.1
Host: example.com
Signature-Input: sig1=("@authority" "signature-agent");keyid="OGMwMzFh...";alg="ed25519";created=1711180800;expires=1711267200;nonce="a1b2c3d4";tag="web-bot-auth"
Signature: sig1=:dGhpcyBpcyBhbiBleGFtcGxlIHNpZ25hdHVyZQ==:
Signature-Agent: agent1="https://example-bot.com"

Required signature parameters:

• created and expires — timestamps bounding the signature’s validity (max 24 hours per spec, though Cloudflare recommends ~1 minute)
• keyid — identifies which public key to use for verification
• tag — must be the literal string web-bot-auth, which is how servers identify bot identity signatures vs other uses of RFC 9421
• At least @authority (the target domain) or @target-uri (the full request URI) must be signed, binding the signature to the destination so it cannot be replayed against a different site

The spec does not mandate a single algorithm. It references the IANA HTTP Message Signatures Algorithm registry, and the test vectors demonstrate both rsa-pss-sha512 and ed25519. In practice, Ed25519 dominates early implementations. Cloudflare’s verifier supports only Ed25519.

How verification works

When a server receives a request with these headers, the verification flow is:

1. Parse Signature, Signature-Input, and Signature-Agent headers. Return 400 if parsing fails.
2. Check that the tag parameter equals web-bot-auth.
3. Look up the keyid in the local key store.
4. If the key is unknown, fetch the bot’s public key directory from the URL in Signature-Agent — a JWKS file hosted at /.well-known/http-message-signatures-directory.
5. Verify the signature using the public key and specified algorithm.
6. Enforce any origin-specific policies (nonce, expiration, rate limits).

Key directory and metadata

• The key directory is a JSON Web Key Set (RFC 7517) served at /.well-known/http-message-signatures-directory
• The directory response itself should be signed using a different tag (http-message-signatures-directory) to prevent confusion with request signatures
• Multiple keys can coexist for rotation
• The Signature Agent Card is a separate spec with 14 optional metadata fields, including rfc9309-product-token and rfc9309-compliance (bridging cryptographic identity to robots.txt) and a trigger field that distinguishes between fetcher (user-initiated) and crawler (autonomous)

Implementation gotcha

The signature value encoding is base64, not base64url. Using base64url causes verification failure. Stytch’s implementation guide documents this and other practical pitfalls.

Who is adopting it

Who built it

• Architecture draft (v05, March 2, 2026): authored by Thibault Meunier (Cloudflare) and Sandor Major (Google)
• Registry draft: adds Maxime Guerreiro (Cloudflare) and Ulas Kirazci (Amazon) as co-authors
• IETF webbotauth Working Group: formally chartered under the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef
• Milestones: April 30, 2026 (standards-track specs due to IESG), August 31, 2026 (operational best practices)
• Reference implementation: TypeScript and Rust, Apache 2.0 licensed
• Cloudflare’s original blog post (May 15, 2025) introduced two proposals: HTTP Message Signatures and request mTLS. The signatures approach has seen far more traction.

Bots signing their requests

• OpenAI signs Operator requests using web-bot-auth with Ed25519. The Signature-Agent points to https://chatgpt.com, and the public keys are hosted at chatgpt.com/.well-known/http-message-signatures-directory. Their implementation signs @authority, @method, @path, and signature-agent, more components than the spec’s minimum.
• Google is “experimenting with the web-bot-auth protocol, using the agent.bot.goog identity” for Google-Agent, a new user-triggered fetcher associated with Project Mariner. “Experimenting” is the exact word Google chose. But Sandor Major of Google co-authoring the IETF draft tells you this is more than a casual experiment. Google-Agent is classified as a user-triggered fetcher that generally ignores robots.txt, has separate IP ranges, and uses the .goog TLD for its identity, separate from the google.com domains used by Googlebot. For a complete list of Google-Agent IP addresses, see our Googlebot IP list.
• Shopify takes a unique approach. In addition to signing its own requests with Signature-Agent: https://shopify.com, Shopify lets merchants create signing keys for third-party tools like SEO crawlers and accessibility auditors. This is the reverse of the typical flow where only bot operators sign their own requests.

Servers verifying signatures

• Cloudflare verifies web-bot-auth signatures in its WAF and Bot Management. Their implementation supports Ed25519 only, does not validate nonces, and recommends roughly one-minute expiration windows. More restrictive than the spec on algorithms, more relaxed on nonce checking.
• Vercel supports web-bot-auth as a verification method in its Bot Protection product alongside IP and reverse DNS checks. ChatGPT Operator is specifically listed as a verified bot.
• AWS WAF added web-bot-auth verification in November 2025. It automatically allows verified AI agent traffic by default at no additional cost, but only on CloudFront distributions, not ALB or API Gateway.

Worth noting: agentic commerce

web-bot-auth has found adoption beyond bot management. Visa’s Trusted Agent Protocol (TAP) and Mastercard’s Agent Pay both use web-bot-auth as their agent authentication layer, according to Cloudflare. The same cryptographic identity that verifies a search crawler now verifies AI agents making purchases.

Where web-bot-auth fits

web-bot-auth and robots.txt operate on different axes:

• robots.txt (RFC 9309): crawl policy. Advisory directives about which paths a bot should access. The RFC states explicitly that its rules “are not a form of access authorization.”
• web-bot-auth: identity verification. Cryptographic proof of who is making the request.

These are complementary. You need both.

RFC 9309’s security section recommends that users “employ a valid security measure relevant to the application layer” and specifically cites HTTP Authentication as an example. web-bot-auth is that security measure.

Here is the emerging standards stack, layer by layer:

The IETF’s aipref working group, authored by Gary Illyes of Google and Martin Thomson of Mozilla, fills the preference gap: defining Content-Usage headers and robots.txt extensions for expressing how content may be used by AI systems. Its milestone is August 2026. The webbotauth charter explicitly excludes “defining a vocabulary for the intents of bots.” That is aipref’s job.

What to do now

• Most site owners will not implement web-bot-auth verification themselves. Your CDN or WAF handles it.
• Cloudflare, AWS WAF (CloudFront only), and Vercel already support signature verification.
• Check whether your provider supports it.
• My recommendation: understand the protocol, but wait before making infrastructure changes. This is still an emerging standard. Watch whether adoption broadens enough to become the default before investing ops time.

The open questions

The technology works. Here is what remains unresolved.

The “whitelisted web” concern. The W3C TPAC 2025 session on the future of the open web noted that web-bot-auth carries “the implication that those clients that don’t (authenticate) may be blocked.” If the default becomes “block unknown, allow signed,” only well-resourced entities that can manage signing infrastructure get through. A commenter on Hacker News put it more bluntly: “key generation is free, so being a well-behaved unknown bot is the same as being an unidentified bot.” The system can only function as a whitelist, not a trust mechanism for unknowns. This is a genuine design constraint acknowledged by the charter’s non-goal of detecting non-participating bots.

Centralization risk. Hammad Tariq, writing after attending IETF 124 in Montreal, argues the governance model risks recreating “the certificate authority problem through deployment patterns.” His concern is not the cryptography (he calls the technical implementation “sound”) but that practical deployment may concentrate through a handful of CDN providers. He also flags the absence of Certificate Transparency-style logs, which would prevent “secret exclusions” and “opaque business deals affecting technical access.” CT logs were a major improvement for the CA ecosystem. Their absence here is a concrete, fixable gap.

Commercialization creep. Tariq identifies a related concern: companies like TollBit and Skyfire are already building billing infrastructure on top of bot identity, conflating identity verification with paywall mechanics. Once you can cryptographically prove who accessed what, the leap to metered billing is short.

Implementation immaturity. Cloudflare’s verifier does not validate nonces and does not guard against replay attacks using a nonce database, relying instead on short expiration windows. The Signature-Agent header format has changed between draft versions, meaning earlier implementations may need updating. Normal growing pains for an emerging standard, but they underscore that this is still early.

Only cooperative actors participate. Malicious bots will not sign their requests. The charter explicitly acknowledges this: detecting non-participating bots is a non-goal. web-bot-auth helps you distinguish verified bots from unknown automation, but you still need traditional bot management for everything else.

My take: the cryptographic foundation is solid, the adoption momentum is real, and the ops simplification is significant. The question worth watching is whether the verification ecosystem stays open, with federated, auditable directories and competitive policy layers, or consolidates through a few gatekeepers. Certificate Transparency logs for bot identity would be a good start.

IP allowlists and reverse DNS lookups are not going away, but they are no longer the only option. web-bot-auth adds a cryptographic layer that simplifies bot verification significantly. The bot proves who it is. You verify the proof. The technology is here, the IETF deadline is April 2026, and the major players are already building on it. The question now is how the trust ecosystem around it takes shape.

Resources

IETF Specs

• Architecture draft (v05) — the core protocol
• Directory draft (v05) — key discovery via JWKS
• Registry draft (v01) — Signature Agent Cards
• Working group charter — scope, milestones, governance
• RFC 9421 — HTTP Message Signatures (the foundation layer)
• aipref draft — AI usage preferences (complementary standard)

Implementation

• Cloudflare reference implementation — TypeScript + Rust, Apache 2.0
• Cloudflare developer docs — verifier requirements and test endpoint
• Stytch implementation guide — complete Node.js walkthrough

Adoption

• Cloudflare blog (May 2025) — original announcement
• Cloudflare agent registry blog (Oct 2025) — registry format introduction
• Cloudflare agentic commerce blog (Oct 2025) — Visa TAP + Mastercard Agent Pay
• OpenAI Operator signing — Castle.io analysis of signed requests
• Vercel Bot Protection — web-bot-auth verification support
• Shopify crawler access keys — merchant-created signing keys
• AWS WAF support — verification for CloudFront distributions
• Visa Trusted Agent Protocol — commerce extension

Google-Agent

• Google user-triggered fetchers docs — definition and web-bot-auth experiment
• Google-Agent IP ranges — dedicated JSON file
• Project Mariner — the AI agent behind Google-Agent
• SE Roundtable coverage — rollout timeline
• Googlebot IP list — complete IP database including Google-Agent

Criticism & Context

• Hacker News discussion — community debate on Cloudflare blog
• Hammad Tariq — centralization risk — IETF 124 analysis
• Anita Srinivasan — Tech Policy Press — legal/policy perspective
• W3C TPAC 2025 session — Future of the Open Web