← Back to the blog

Who signs? 16 AI agent operators publish Web Bot Auth keys

Which AI crawlers and agents can a website actually verify today, and which ones still give it nothing but a User-Agent string?

The short answer

On 18 November 2026 we asked 141 hosts run by AI crawler and agent operators for a Web Bot Auth key directory. 16 served one. Google, Meta and OpenAI are among them; Anthropic, Perplexity and Apple are not, at the hosts we checked.

Web Bot Auth lets an agent sign its HTTP requests (RFC 9421) and publish the public keys at /.well-known/http-message-signatures-directory on its own domain. A website can then check that a request claiming to be, say, ExaSearchBot really came from Exa, whatever IP address it arrives from.

The 16

OperatorDirectory hostKeys
Google (Google-Agent)agent.bot.goog5
Metawww.meta.com5
Ahrefsahrefs.com2
Browserbasewww.browserbase.com2
Parallelshapbot.parallel.ai2
OpenAI (ChatGPT agent)chatgpt.com1
DuckDuckGoassistbot.duckduckgo.com1
Exacrawler.exa.ai1
Hyperbrowserwww.hyperbrowser.ai1
ibouibou.io1
Kernelwww.kernel.sh1
Klaviyowww.klaviyo.com1
Known Agentsknownagents.com1
Riverriver.ai1
Shopifywww.shopify.com1 (a single key, not a key set)
You.comyou.com1

Every key is Ed25519.

What we found missing

No directory at the hosts we checked for Anthropic (anthropic.com, claude.ai, claude.com), Perplexity, Apple, Mistral, Common Crawl or Anchor. That doesn't mean they never sign: a signer can name a directory elsewhere in its own Signature-Agent header, which a probe like this can't see.

How we counted

  • The list: every host named in the ai.robots.txt list (robots.json, 18 November), plus 54 hosts of agent, browser-agent and search operators we know of. 141 hosts in all.
  • The probe: one HTTPS GET per host to the well-known path, with the directory media type in Accept, redirects not followed (the draft says a verifier must not follow them), from one location in Europe.
  • A hit: status 200, the directory media type, and a key in it.

Otto van der Schaaf's count on 19 October used a wider list, including Cloudflare Radar's bot directory, and found 55 of 150 operators with keys (modpagespeed.com). Our number is lower because our list leans on ai.robots.txt, where far fewer operators sign. Read both as counts over their lists, not shares of all agents.

What a signature does and doesn't tell you

A valid signature proves the request came from whoever holds the key for that directory. It doesn't tell you who that operator is beyond its domain, how well it has been checked, or how its agents behave on other sites.

We'll run this census again every month and publish what changed.