Last updated October 7, 2026
AgenticWebCensusBot
AgenticWebCensusBot is the probe of the Agentic Web Census, the observatory that measures how ready the web is for AI agents. It visits a site only when somebody asks us to measure one of its pages, makes a handful of requests, and signs every one of them so that the site can tell it is us.
When it visits
Only when somebody measures a page on this site, for free or by buying a report, and once more when a bought report is written. It does not crawl, it does not follow links and it never comes back on its own.
The census's monthly figures do not come from this bot: they come from HTTP Archive's public crawl.
What it asks for
Six requests, all at once, to the origin of the page being measured. It follows up to five redirects, each one signed for its own host.
- The page itself.
- /robots.txt, to report which AI agents it lets in.
- /llms.txt and /sitemap.xml, to report whether they exist.
- /.well-known/ard.json, to report whether the site describes itself to agent registries.
- /agentic-web-census-probe-404, a path nobody has, to see how the server answers a missing page.
How to recognise it
Its User-Agent is AgenticWebCensusBot/1 (+https://agenticwebcensus.com/about). It does not pretend to be a browser, it does not run JavaScript and it does not try to solve a challenge.
Every request carries a Web Bot Auth signature: HTTP Message Signatures with an Ed25519 key, the header Signature-Agent: "https://agenticwebcensus.com", and the public key published at https://agenticwebcensus.com/.well-known/http-message-signatures-directory.
What it keeps
The status of each answer, a short list of headers that the report reads (content type, caching, server, the technology it announces and the Cloudflare challenge header), and the first 512 KiB of the page and of robots.txt. It keeps no cookies.
Stopping it or writing to us
You can turn it away by its User-Agent at your server or at your CDN. For anything about it, write to [email protected].