Agentic Web CensusCheck your site
EN

AI agent · September 2026

Claude-Web in robots.txt: who blocks it

1.5% of the 166,024 sites in the census name Claude-Web in their robots.txt, and 71.2% of those block it.

1.5%
Sites naming it

Sites naming it

Sample
n=166,024
Coverage
100.0%

The September 2026 crawl, parsed for a robots.txt rule naming this user-agent, on the same 1% sample of the corpus as every other figure on this site — 166,024 sites. It used to be measured on the whole corpus and this note used to say so; it moved to the monthly sample when the axis gained a series, so it now carries the same margin as everything else.

1.1%
Sites blocking it

Sites blocking it

Sample
n=166,024
Coverage
100.0%
Name it
Name it: 1.5%
1.5%
Block it
Block it: 1.1%
1.1%
0%25%50%75%100%
Of the 166,024 sites in the census, 1.5% name Claude-Web and 1.1% forbid it. Both bars are the same base, so the gap between them is the sites that named it to let it in.

What Claude-Web actually does

Run by Anthropic. An older token for fetching a page on a user’s behalf. Blocking it means that when someone asks about your page, the assistant cannot open it.

What to write in your robots.txt

Put the rule in its own group. A Disallow under User-agent: * predates these agents and several of them do not treat it as applying to them — which is also why this census does not count it.

User-agent: Claude-Web
Disallow: /
robots.txtTurns Claude-Web away. Blocking it means that when someone asks about your page, the assistant cannot open it.
User-agent: Claude-Web
Allow: /
robots.txtLets it through, said out loud rather than by saying nothing. Either way it is a published preference and not a wall: robots.txt enforces nothing.