web-bot-auth: The Bot Authentication Protocol Explained

If you manage a website of any real size, you have probably spent time verifying bots. Not the interesting kind of time. The kind where you maintain IP allowlists that go stale, run reverse DNS lookups against ranges that change without notice, and trust user-agent strings that anyone can fake in one line of code.

Key facts:

• web-bot-auth is an IETF protocol for cryptographic bot identity, built on RFC 9421
• Co-authored by Cloudflare and Google, with an IETF working group chartered in October 2025
• Signing requests with it today:
  • OpenAI (ChatGPT Work’s Cloud browser)
  • Google (Google-Agent, experimental, a subset of requests)
  • Shopify (merchant-created keys for third-party crawlers)
  • Amazon (Bedrock AgentCore Browser)
• Verifying signatures today:
  • Cloudflare (WAF and Bot Management, plus the reference implementation)
  • Akamai (App & API Protector)
  • Vercel (Bot Protection)
  • HUMAN (Sightline and AgenticTrust)
  • AWS WAF (CloudFront distributions only)
• It does not replace robots.txt^? What is robots.txt? ✕ robots.txt is a text file that says which URLs crawlers may fetch. robots.txt controls crawling, not indexing. Full definition of robots.txt ← , IP verification, or reverse DNS. It adds a cryptographic verification option alongside them
• The working group’s April 2026 standards-track milestone passed with no adopted drafts; the operational best-practices milestone is August 31, 2026. At IETF 126 in July 2026 the room polled 22 to 6 in favour of the HTTP Message Signatures direction. In August 2026 the protocol draft was revised with Standards Track intent and absorbed the key-directory draft, still as an individual submission

web-bot-auth offers an alternative: a single cryptographic signature on every HTTP request. The bot signs. You verify. Identity established, without maintaining IP lists or running DNS lookups.

The protocol is an IETF draft co-authored by Cloudflare and Google, built on RFC 9421 (HTTP Message Signatures). OpenAI signs ChatGPT agent requests with it. Google is experimenting with it for Google-Agent. Cloudflare, Akamai, AWS, Vercel, and HUMAN verify signatures on the server side.

This guide covers how the protocol works, who built it, who is adopting it, and what it means for your site.

The ops tax of bot verification

The IETF charter for webbotauth states plainly that current bot verification methods (IP allowlisting, user-agent strings, and shared API keys) have “significant limitations regarding security, scalability, and manageability.” That is diplomatic. Here is what those limitations look like in practice.

User-agent strings are self-reported. Any HTTP client can claim to be Googlebot, GPTBot, or anything else. There is no verification in the string itself. If you make access decisions based on a user-agent header, you are trusting the visitor’s word for it.

IP allowlists are brittle ops work. Google publishes its crawler IP ranges. So does Bing, so does OpenAI. But those ranges change. Cloud providers share IP space across customers. Maintaining current allowlists across multiple bot operators is ongoing manual labor that scales poorly and breaks silently when a range rotates.

Shared secrets do not scale. You could distribute API keys to every bot operator you want to allow. That works for one or two integrations. It does not work when hundreds of operators need to identify themselves to millions of websites.

web-bot-auth solves all three with asymmetric cryptography. The bot operator holds a private key, signs every request, and publishes the corresponding public key. You verify the signature. If it checks out, the identity is confirmed.

Three headers, one proof

web-bot-auth is a profile built on top of RFC 9421 (HTTP Message Signatures), a general-purpose HTTP signing standard published in February 2024. RFC 9421 defines how to sign arbitrary HTTP message components. The web-bot-auth architecture draft constrains it for bot identity: only asymmetric keys (“Implementations MUST NOT use shared HMAC”), mandatory domain binding, and a fixed tag so servers know the signature is a bot identity claim.

A signed request carries two required headers and one recommended header:

• Signature-Input — declares what was signed and the signature parameters
• Signature — the base64-encoded cryptographic signature
• Signature-Agent — (recommended) a URL pointing to the bot operator’s public key directory

Here is what they look like in the current IETF draft:

GET /page HTTP/1.1
Host: example.com
Signature-Input: sig1=("@authority" "signature-agent");keyid="OGMwMzFh...";alg="ed25519";created=1711180800;expires=1711267200;nonce="a1b2c3d4";tag="web-bot-auth"
Signature: sig1=:dGhpcyBpcyBhbiBleGFtcGxlIHNpZ25hdHVyZQ==:
Signature-Agent: agent1="https://example-bot.com"

There is a live compatibility gap here. The August 2026 draft defines Signature-Agent as a dictionary structured header, like agent1="https://example-bot.com", and treats the older bare structured string as legacy. Cloudflare’s verifier documentation, last updated July 1, 2026, still implements that older string form, like "https://signature-agent.test", and says requests using the later dictionary form fail verification. If you are implementing this today, follow your verifier’s documented header shape alongside the newest draft text.

Required signature parameters:

• created and expires — timestamps bounding the signature’s validity (the spec recommends no more than 24 hours; Cloudflare recommends ~1 minute)
• keyid — identifies which public key to use for verification
• tag — must be the literal string web-bot-auth, which is how servers identify bot identity signatures vs other uses of RFC 9421
• At least @authority (the target domain) or @target-uri (the full request URI) must be signed, binding the signature to the destination so it cannot be replayed against a different site

The spec does not mandate a single algorithm. It references the IANA HTTP Message Signatures Algorithm registry, and the test vectors demonstrate both rsa-pss-sha512 and ed25519. In practice, Ed25519 dominates early implementations. Cloudflare’s verifier supports only Ed25519.

How verification works

When a server receives a request with these headers, the verification flow is:

1. Parse Signature, Signature-Input, and Signature-Agent headers. Return 400 if parsing fails.
2. Check that the tag parameter equals web-bot-auth.
3. Look up the keyid in the local key store.
4. If the key is unknown, fetch the bot’s public key directory from the URL in Signature-Agent — a JWKS file hosted at /.well-known/http-message-signatures-directory.
5. Verify the signature using the public key and specified algorithm.
6. Enforce any origin-specific policies (nonce, expiration, rate limits).

Key directory and metadata

• The key directory is a JSON Web Key Set (RFC 7517) served at /.well-known/http-message-signatures-directory. It started life as a separate directory draft; the August 2026 revision of the protocol draft folded it into the main document
• The directory response itself should be signed using a different tag (http-message-signatures-directory) to prevent confusion with request signatures
• Multiple keys can coexist for rotation
• Since the August 2026 revision, the Signature-Agent URL, not the key, is the bot’s identifier: “Key lookup MUST be keyed on the (URL, key) pair, not on the key alone”. A bot’s identity now survives key rotation, and a key alone earns no trust without the URL that published it
• The Signature Agent Card is a separate spec with a set of optional metadata fields, including rfc9309-product-token and rfc9309-compliance (bridging cryptographic identity to robots.txt) and a trigger field that distinguishes between fetcher (user-initiated) and crawler (autonomous)

Implementation gotcha

The signature value encoding is base64, not base64url. Using base64url causes verification failure. Stytch’s implementation guide documents this and other practical pitfalls.

Who is adopting it

Who built it

• Protocol draft (formerly the architecture draft, renamed in June 2026): authored by Thibault Meunier (Cloudflare) and Sandor Major (Google). The August 2026 revision (-01) is a substantial one: intended status changed from Informational to Standards Track, and the separate key-directory draft was folded into the main document. Still an individual submission, but the authors are now openly aiming at a standard
• Registry draft: adds Maxime Guerreiro (Cloudflare) and Ulas Kirazci (Amazon) as co-authors
• IETF webbotauth Working Group: formally chartered under the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef
• Milestones: the April 30, 2026 target for standards-track specs passed with no working-group-adopted drafts. Everything is still individual submissions, and the operational best-practices milestone (August 31, 2026) is next
• IETF 126 in Vienna (July 22, 2026) polled the room on building the identifying approach on HTTP Message Signatures: 22 in favour, 6 against, 5 no opinion, out of roughly 75 people. Objections centred on key discovery, centralization risk, and discriminatory effects. The chairs are exploring a recharter to formally adopt crawler best practices, with a possible virtual interim before IETF 127 in November
• Reference implementation: TypeScript and Rust, Apache 2.0 licensed
• Cloudflare’s original blog post (May 15, 2025) introduced two proposals: HTTP Message Signatures and request mTLS. The signatures approach has seen far more traction.

Bots signing their requests

• OpenAI signs ChatGPT Work’s Cloud browser requests using web-bot-auth with Ed25519. The product has now been renamed twice: Operator became ChatGPT agent, which became Cloud browser. OpenAI’s own allowlisting doc is the primary reference, and it specifies that Signature-Agent must match "https://chatgpt.com" exactly, quotation marks included, with the public keys hosted at chatgpt.com/.well-known/http-message-signatures-directory. If you are writing a rule by hand, the quotes are the part people get wrong. Their implementation signs @authority, @method, @path, and signature-agent, more components than the spec’s minimum. The rest of OpenAI’s bot fleet is a different story: GPTBot, OAI-SearchBot, and ChatGPT-User still verify via published IP lists, and ChatGPT Atlas browsing traffic is not signed at all.
• Google is “experimenting with the Web Bot Auth protocol” using the https://agent.bot.goog identity for Google-Agent, the user-triggered fetcher now behind Gemini Agent (Project Mariner, its original home, was folded into Gemini in May 2026). “Experimenting” is the exact word Google chose, and it means it: only a subset of Google-Agent requests are signed, with IP ranges and reverse DNS still the documented fallback. But Sandor Major of Google co-authoring the IETF draft tells you this is more than a casual experiment. Google-Agent is classified as a user-triggered fetcher that generally ignores robots.txt, has separate IP ranges, and uses the .goog TLD for its identity, separate from the google.com domains used by Googlebot. For a complete list of Google-Agent IP addresses, see our Googlebot IP list.
• Shopify takes a unique approach: it lets merchants create signing keys for third-party tools like SEO crawlers and accessibility auditors, managed from the Shopify admin. This is the reverse of the typical flow where only bot operators sign their own requests.
• Amazon signs on the agent side too: Bedrock AgentCore Browser supports web-bot-auth signatures for its automated browsing traffic.

Servers verifying signatures

• Cloudflare verifies web-bot-auth signatures in its WAF and Bot Management. Their implementation supports Ed25519 only, does not validate nonces, and recommends roughly one-minute expiration windows. It also still follows the pre-August Signature-Agent structured-string format and rejects the newer dictionary form from the -01 draft. More restrictive than the spec on algorithms, more relaxed on nonce checking, and not yet aligned with the newest header syntax.
• Vercel supports web-bot-auth as a verification method in its Bot Protection product alongside IP and reverse DNS checks. OpenAI’s Cloud browser is still listed there under its original identifier, chatgpt-operator, which has survived two product renames. Search your CDN’s bot list for the old name if the new one returns nothing.
• AWS WAF added web-bot-auth verification in November 2025. It automatically allows verified AI agent traffic by default at no additional cost, but only on CloudFront distributions, not ALB or API Gateway.
• Akamai supports web-bot-auth in its bot and abuse solutions, including bot visibility and mitigation in App & API Protector.
• HUMAN verifies signatures in both Sightline, where signed agents show up under Known Bots & Crawlers, and AgenticTrust, which adds a per-agent permissions layer on top of the identity. It is the newest verifier of the group, and the only one that lets you set what a verified agent is allowed to do once you have confirmed who it is.

Cloudflare also built a product category on top of the protocol: Signed Agents, launched with ChatGPT agent, Block’s Goose, Browserbase, and Anchor Browser as the first cohort of user-directed agents authenticating via web-bot-auth. In July 2026 the separate category was merged into Verified Bots, with metadata distinguishing direct from intermediary access.

In August 2026 Cloudflare promoted the protocol from a bot-management feature to the identity pillar of its agentic-internet launch week: “Web Bot Auth lets a bot cryptographically identify itself to any site it visits, so publishers can decide who they welcome and who they don’t.” The same week, its new Agent Readiness diagnostics scored sites on Web Bot Auth support as one of the advanced-integration checks, and Cloudflare Wallets named it as the base of its identity layer: “Web Bot Auth already allows agents to register their identity via a keypair. IDs attached to Cloudflare Wallets allow this keypair to become human-readable.”

The launch messaging groups web-bot-auth with PACT (Private Access Control Tokens) in its list of open standards. PACT, announced back in June 2026, is a separate mechanism solving the complementary problem: web-bot-auth proves which bot is calling, while PACT lets a site that already knows a human is behind a session issue anonymous tokens vouching for that, and it is designed so sites cannot use the tokens to track the person across the web. The names committed to developing it:

• Mozilla (Firefox)
• Google (Chrome)
• Microsoft (Edge)
• Shopify

Worth noting: agentic commerce

web-bot-auth has found adoption beyond bot management. Visa’s Trusted Agent Protocol (TAP) and Mastercard’s Agent Pay both use web-bot-auth as their agent authentication layer, according to Cloudflare. The same cryptographic identity that verifies a search crawler now verifies AI agents making purchases.

Where web-bot-auth fits

web-bot-auth and robots.txt operate on different axes:

• robots.txt (RFC 9309): crawl policy. Advisory directives about which paths a bot should access. The RFC states explicitly that its rules “are not a form of access authorization.”
• web-bot-auth: identity verification. Cryptographic proof of who is making the request.

These are complementary. You need both.

RFC 9309’s security section recommends that users “employ a valid security measure relevant to the application layer” and specifically cites HTTP Authentication as an example. web-bot-auth is that security measure.

Here is the emerging standards stack, layer by layer:

The IETF’s aipref working group fills the preference gap: defining Content-Usage headers and robots.txt extensions for expressing how content may be used by AI systems. Its attach draft, authored by Gary Illyes of Google and Martin Thomson of Mozilla, expired in May 2026 without a revision, while the companion vocabulary draft kept moving. The group’s milestone is August 2026. The webbotauth charter explicitly excludes “defining a vocabulary for the intents of bots.” That is aipref’s job.

What to do now

• Most site owners will not implement web-bot-auth verification themselves. Your CDN or WAF handles it.
• Cloudflare, Akamai, HUMAN, AWS WAF (CloudFront only), and Vercel already support signature verification.
• Check whether your provider supports it.
• My recommendation: understand the protocol, but wait before making infrastructure changes. This is still an emerging standard. Watch whether adoption broadens enough to become the default before investing ops time.

The open questions

The technology works. Here is what remains unresolved.

The “whitelisted web” concern. Mark Nottingham’s session proposal for the W3C TPAC 2025 breakout on the future of the open web noted that web-bot-auth carries “the implication that those clients that don’t (authenticate) may be blocked.” If the default becomes “block unknown, allow signed,” only well-resourced entities that can manage signing infrastructure get through. A commenter on Hacker News put it more bluntly: “key generation is free, so being a well-behaved unknown bot is the same as being an unidentified bot.” The system can only function as a whitelist, not a trust mechanism for unknowns. This is a genuine design constraint acknowledged by the charter’s non-goal of detecting non-participating bots.

Centralization risk. Hammad Tariq, writing after reviewing the IETF 124 sessions from Montreal, argues the governance model risks recreating “the certificate authority problem through deployment patterns.” His concern is not the cryptography (he calls RFC 9421 “solid engineering”) but that practical deployment may concentrate through a handful of CDN providers. He also flags the absence of Certificate Transparency-style logs, without which there is no way to audit who gets excluded, or why. CT logs were a major improvement for the CA ecosystem. Their absence here is a concrete, fixable gap.

Commercialization creep. Tariq identifies a related concern: companies like TollBit and Skyfire are already building billing infrastructure on top of bot identity, conflating identity verification with paywall mechanics. Once you can cryptographically prove who accessed what, the leap to metered billing is short.

Implementation immaturity. Cloudflare’s verifier does not validate nonces and does not guard against replay attacks using a nonce database, relying instead on short expiration windows. The core drafts themselves were renamed in June 2026 and restructured again in August 2026, when the directory spec was folded into the protocol draft and Signature-Agent moved to dictionary syntax. Earlier implementations and integrations may need updating. Normal growing pains for an emerging standard, but they underscore that this is still early.

Only cooperative actors participate. Malicious bots will not sign their requests. The charter explicitly acknowledges this: detecting non-participating bots is a non-goal. web-bot-auth helps you distinguish verified bots from unknown automation, but you still need traditional bot management for everything else.

Two of those objections now have drafts aimed at them, both landing on the datatracker on July 19, 2026, three days before the IETF 126 session. Anonymous Bot Authentication from Eric Rescorla and Richard Barnes, first posted in April and revised that day, issues anonymous credentials through trusted “Anchors” so a site learns that a request came from an authenticated bot within an approved set without learning which bot it was, and can still rate-limit each agent. Its stated motivation is the discrimination problem itself: bots that monitor discriminatory hiring practices or track government activity are precisely the ones a site might block if it could identify them. Hosted Key Directories, new that day, goes after key discovery by letting operators delegate directory hosting instead of each running their own. Neither is working-group adopted, and the Rescorla draft carries an explicit warning that it “has not yet seen significant security analysis.”

My take: the cryptographic foundation is solid, the adoption momentum is real, and the ops simplification is significant. The question worth watching is whether the verification ecosystem stays open, with federated, auditable directories and competitive policy layers, or consolidates through a few gatekeepers. Certificate Transparency logs for bot identity would be a good start.

IP allowlists and reverse DNS lookups are not going away, but they are no longer the only option. web-bot-auth adds a cryptographic layer that simplifies bot verification significantly. The bot proves who it is. You verify the proof. The technology is here, the IETF working group is active even after its first milestone slipped, and the major players are already building on it. The question now is how the trust ecosystem around it takes shape.

Resources

IETF Specs

• Protocol draft — the core protocol (formerly the architecture draft; Standards Track intent and key discovery folded in since August 2026)
• Directory draft — key discovery via JWKS (merged into the protocol draft, August 2026)
• Registry draft — Signature Agent Cards
• Working group charter — scope, milestones, governance
• RFC 9421 — HTTP Message Signatures (the foundation layer)
• aipref attach draft — AI usage preferences (complementary standard; expired May 2026) and its active vocabulary draft
• JAFAR draft — a JSON format for publishing bot IP ranges (Gary Illyes)
• Crawler best practices draft — operational guidance for crawler operators
• Anonymous bot auth draft — authorization and rate limiting without identity (Eric Rescorla, Richard Barnes)
• Hosted key directories draft — delegating key directory hosting instead of self-hosting

Implementation

• Cloudflare reference implementation — TypeScript + Rust, Apache 2.0
• Cloudflare developer docs — verifier requirements and test endpoint
• Stytch implementation guide — complete Node.js walkthrough

Adoption

• Cloudflare blog (May 2025) — original announcement
• Cloudflare agent registry blog (Oct 2025) — registry format introduction
• Cloudflare agentic commerce blog (Oct 2025) — Visa TAP + Mastercard Agent Pay
• OpenAI Operator signing — Castle.io analysis of signed requests
• Vercel Bot Protection — web-bot-auth verification support
• Shopify crawler access keys — merchant-created signing keys
• AWS WAF support — verification for CloudFront distributions
• Amazon Bedrock AgentCore Browser — agent-side signing
• Akamai web-bot-auth support — verification in App & API Protector
• OpenAI Cloud browser allowlisting — the signing side from the source, with per-vendor allowlist identifiers
• IETF 126 webbotauth session — agenda, minutes, and the direction poll
• Cloudflare Signed Agents — user-directed agent category (merged into Verified Bots, July 2026)
• Cloudflare agentic-internet post (Aug 2026) — Web Bot Auth as the identity layer of the launch week
• Cloudflare Agent Readiness / AEO (Aug 2026) — Web Bot Auth as a scored advanced-integration check
• Cloudflare Wallets (Aug 2026) — human-readable IDs on top of web-bot-auth keypairs
• PACT press release (June 2026) — Private Access Control Tokens, with Mozilla, Google, Microsoft, and Shopify
• Visa Trusted Agent Protocol — commerce extension

Google-Agent

• Google’s Web Bot Auth documentation — the experiment’s official reference
• Google user-triggered fetchers docs — Google-Agent definition
• Google-Agent IP ranges — dedicated JSON file
• SE Roundtable coverage — rollout timeline
• Googlebot IP list — complete IP database including Google-Agent

Criticism & Context

• Hacker News discussion — community debate on Cloudflare blog
• Hammad Tariq — centralization risk — IETF 124 analysis
• Anita Srinivasan — Tech Policy Press — legal/policy perspective
• W3C TPAC 2025 session proposal — Future of the Open Web (Mark Nottingham)