Trends · Sites with an AI crawler policy
Sites with an AI crawler policy across the web
The share of sites where a robots.txt names an AI crawler, either to allow it or to block it. It does NOT feed the index: deciding is not the same as being ready, and scoring it would make a value judgement out of a measurement.. It has gone up since May 2026, and again last month. Both steps clear their margin.
Poor11.1%
Sites with an AI crawler policy in September 2026
▲ +0.2 pp9.1%11.1%
What this measures
| Dimension | Weight in the index |
|---|---|
| 1 · Read | 53% of the score |
| 2 · Guide | 27% of the score |
| 3 · Act | 20% of the score |
| 4 · Reachable | multiplies the rest |
Who moved
| Segment | August 2026 | September 2026 | Movement |
|---|---|---|---|
| .click | 1.6% | 50.8% | ▲ +49.2 pp |
| Vuetify | 6.4% | 34.5% | ▲ +28.1 pp |
| Browser-Update.org | 10.7% | 30.2% | ▲ +19.4 pp |
| Recent Posts Widget With Thumbnails | 22.0% | 4.0% | ▼ −18.0 pp |
| .site | 36.7% | 19.8% | ▼ −16.9 pp |
| Rudderstack | 14.6% | 30.3% | ▲ +15.8 pp |
| CakePHP | 26.2% | 11.6% | ▼ −14.6 pp |
| Apple Sign-in | 49.6% | 35.8% | ▼ −13.8 pp |
| .cc | 22.9% | 9.4% | ▼ −13.6 pp |
| PubSubJS | 23.7% | 10.2% | ▼ −13.4 pp |
| Knockout.js | 45.8% | 58.9% | ▲ +13.1 pp |
| TripleLift | 55.0% | 68.0% | ▲ +13.1 pp |
| .tw | 7.3% | 20.3% | ▲ +13.0 pp |
| comScore | 47.2% | 60.2% | ▲ +12.9 pp |
| Tengine | 25.2% | 13.1% | ▼ −12.1 pp |