Trends · Agents that can reach the site
Agents that can reach the site across the web
What share of the AI agents we track can reach the site. It has gone down since May 2026, by more than the margin allows us to call a wobble — though last month on its own did not move.
Excellent95.5%
Agents that can reach the site in September 2026
= −0.1 pp96.4%95.5%
What this measures
Who moved
| Segment | August 2026 | September 2026 | Movement |
|---|---|---|---|
| .click | 99.3% | 76.4% | ▼ −23.0 pp |
| Vuetify | 97.2% | 83.8% | ▼ −13.5 pp |
| Substack | 98.4% | 85.7% | ▼ −12.7 pp |
| Mustache 4 | 88.7% | 77.6% | ▼ −11.0 pp |
| PubSubJS | 84.8% | 95.7% | ▲ +10.9 pp |
| Wunderkind | 86.5% | 75.9% | ▼ −10.6 pp |
| SweetAlert2 11 | 83.3% | 93.6% | ▲ +10.3 pp |
| MediaWiki 1 | 87.3% | 77.0% | ▼ −10.3 pp |
| Bootstrap 0 | 71.0% | 81.3% | ▲ +10.3 pp |
| Integral Ad Science | 82.6% | 72.7% | ▼ −10.0 pp |
| MediaWiki | 87.3% | 77.6% | ▼ −9.7 pp |
| Recent Posts Widget With Thumbnails 7 | 88.4% | 97.9% | ▲ +9.5 pp |
| Font Awesome 3 | 80.5% | 71.1% | ▼ −9.4 pp |
| Uploadcare | 87.5% | 96.4% | ▲ +9.0 pp |
| Tengine | 81.9% | 90.7% | ▲ +8.8 pp |
How to move it on your own site
4 · Reachable
multiplies the rest
Your robots.txt does not turn away the AI agents you want reaching you.
This one multiplies the rest of your score instead of adding to it, because it is the precondition: an agent that is disallowed never fetches the page, so how readable the page is stops mattering. It is measured per agent and gradually — turning away one of the seventeen we track costs you a seventeenth, turning away all of them costs you everything. And this is the one place where a low score may be exactly what you want: if you have decided that AI crawlers should not read your site, a low number here is that decision working, not a defect. What it is not is readiness.
What to do
- Look at your robots.txt before assuming anything: many site builders and security plugins add AI crawler blocks by default, and the person who runs the site often has no idea. Squarespace sites in our census are reachable by only 35% of the agents we track.
- Decide agent by agent, because they do different jobs: one crawls to train a model, another fetches a page because a person just asked about it. Turning away the second one costs you the referral traffic that comes with the answer.
- Remove the Disallow for the agents you want reaching you, or narrow it to the paths that actually need protecting instead of the whole site.
- Remember what a robots.txt is: a published preference that well-behaved crawlers honour. It enforces nothing, so blocking is not protection. If you need enforcement, that is a different job.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /basket/
Disallow: /account/Ten minutes of editing, once the decision is made. The cheapest thing on this list to change, and the one that can cost you the most if it is wrong.