Measured, and not part of the index

These are signals, not scores. They are kept apart on purpose: mixed in with the dimensions, a low one reads as a bad grade for something the index never counted.

3.8%
Sites with an AI crawler policy

Sites with an AI crawler policy

Sample
n=104
Coverage
100.0%

Not a Lighthouse audit: it is read from the robots.txt of the sampled sites, checking whether the file names any of the AI crawlers we track. It measures HAVING AN EXPLICIT POLICY and not permitting: blocking AI crawlers is a legitimate decision, and penalising it would be a value judgement dressed up as a measurement. What is a sign of maturity is having decided.

Sites with an AI crawler policy: 4 months, no significant change= −2.5 pp
28.7%
Sites serving an llms.txt

Sites serving an llms.txt

Sample
n=104
Coverage
100.0%

The same llms-txt audit as the figure below, read differently: this one counts SOMETHING being served at /llms.txt, valid or not. Read that literally, because the distance between the two figures is large and it is the interesting part: of the pages that serve something, fewer than two in five serve a file that parses. Our reading — and we cannot verify it without fetching those files ourselves, which we do not do — is that a good share of the rest are soft 404s: a server answering 200 with an HTML error page. So the figure to cite is the valid one below, and this one is best read as an upper bound. Pages whose audit errored are excluded from both, and the coverage says how many that was.

Sites serving an llms.txt: 4 months, down▼ −7.6 pp

What else we measure here

3.8%
Sites with an AI crawler policy

Sites with an AI crawler policy

Sample
n=104
Coverage
100.0%
98.6%
Agents that can reach the site

Agents that can reach the site

Sample
n=104
Coverage
100.0%
28.7%
Sites serving an llms.txt

Sites serving an llms.txt

Sample
n=104
Coverage
100.0%
7.4%
Sites with a valid llms.txt

Sites with a valid llms.txt

Sample
n=104
Coverage
100.0%

How this was measured