Which AI agents the web blocks

Agent	What it does	Sites naming it	Sites blocking it	Blocking, month by month
GPTBot GPTBot Operator OpenAI Of those naming it, blocked 86.0% Sample n=166,024 Crawls to train OpenAI models. Blocking it does not remove you from ChatGPT answers — that is OAI-SearchBot and ChatGPT-User, which are separate tokens.	Trains models	10.0%	8.6%	▲ +0.2 pp
ClaudeBot ClaudeBot Operator Anthropic Of those naming it, blocked 86.8% Sample n=166,024 Crawls to train Anthropic models.	Trains models	9.2%	8.0%	▲ +0.5 pp
CCBot CCBot Operator Common Crawl Of those naming it, blocked 94.3% Sample n=166,024 Builds the public Common Crawl corpus, which many model trainers use. Blocking it is an indirect way of opting out of several models at once.	Trains models	8.5%	8.0%	= +0.1 pp
Google-Extended Google-Extended Operator Google Of those naming it, blocked 90.0% Sample n=166,024 Controls whether your content is used to train Google’s generative models. It does NOT affect Google Search: blocking it leaves your search ranking alone.	Trains models	8.4%	7.5%	▲ +0.3 pp
Bytespider Bytespider Operator ByteDance Of those naming it, blocked 96.6% Sample n=166,024 Crawls to train ByteDance models.	Trains models	8.0%	7.7%	= +0.1 pp
Amazonbot Amazonbot Operator Amazon Of those naming it, blocked 96.8% Sample n=166,024 Crawls for Amazon’s assistant answers.	Indexes for answers	7.9%	7.7%	▲ +0.2 pp
meta-externalagent meta-externalagent Operator Meta Of those naming it, blocked 92.9% Sample n=166,024 Crawls to train Meta models.	Trains models	7.6%	7.1%	▲ +0.5 pp
Applebot-Extended Applebot-Extended Operator Apple Of those naming it, blocked 94.3% Sample n=166,024 Controls whether your content trains Apple’s generative models. Separate from Applebot, which indexes for Siri and Spotlight and is not affected.	Trains models	7.5%	7.1%	▲ +0.2 pp
anthropic-ai anthropic-ai Operator Anthropic Of those naming it, blocked 85.4% Sample n=166,024 An older token for the same crawling. Still widely written in robots.txt files, so it is still worth counting.	Trains models	3.6%	3.1%	= −0.0 pp
cohere-ai cohere-ai Operator Cohere Of those naming it, blocked 90.5% Sample n=166,024 Crawls to train Cohere models.	Trains models	2.9%	2.6%	= −0.1 pp
PerplexityBot PerplexityBot Operator Perplexity Of those naming it, blocked 54.1% Sample n=166,024 Indexes pages so Perplexity can cite them with a link back.	Indexes for answers	2.9%	1.6%	= −0.0 pp
YouBot YouBot Operator You.com Of those naming it, blocked 94.9% Sample n=166,024 Indexes pages so You.com can cite them.	Indexes for answers	2.6%	2.5%	= −0.1 pp
ChatGPT-User ChatGPT-User Operator OpenAI Of those naming it, blocked 65.0% Sample n=166,024 Fetches a page because a person asked ChatGPT about it. Blocking it breaks that link for a reader who was already interested.	Fetches on request	2.5%	1.6%	▼ −0.1 pp
OAI-SearchBot OAI-SearchBot Operator OpenAI Of those naming it, blocked 54.0% Sample n=166,024 Indexes pages so ChatGPT can cite them. This is the one to leave open if you want to appear in its answers.	Indexes for answers	2.3%	1.2%	▲ +0.3 pp
Claude-Web Claude-Web Operator Anthropic Of those naming it, blocked 71.2% Sample n=166,024 An older token for fetching a page on a user’s behalf.	Fetches on request	1.5%	1.1%	= −0.1 pp
Claude-User Claude-User Operator Anthropic Of those naming it, blocked 64.8% Sample n=166,024 Fetches a page because a person asked Claude about it.	Fetches on request	0.9%	0.6%	= +0.1 pp
Perplexity-User Perplexity-User Operator Perplexity Of those naming it, blocked 60.7% Sample n=166,024 Fetches a page because a person asked about it.	Fetches on request	0.8%	0.5%	= +0.1 pp