Agentic Web CensusCheck your site
EN

Trends · May 2026 – September 2026

The agentic web is getting better, and we can prove it

The index went from 15.2% in May 2026 to 21.0% in September 2026 across 5 monthly measurements. Every single step of that clears its own margin, which is what makes it a trend and not a wobble.

21.6%14.6%AWCS · May: 15.2%AWCS · June: 17.4%AWCS · July: 19.9%AWCS · August: 20.1%AWCS · September: 21.0%AWCSMayJuneJulyAugustSeptember
The AWCS, May 2026 to September 2026. The last step was up 0.9%.

The dimensions underneath

38.1%0.4%Sites with an AI crawler policy · May: 9.1%Sites with an AI crawler policy · June: 9.6%Sites with an AI crawler policy · July: 10.2%Sites with an AI crawler policy · August: 11.0%Sites with an AI crawler policy · September: 11.1%Pages an agent can parse · May: 27.8%Pages an agent can parse · June: 29.4%Pages an agent can parse · July: 30.3%Pages an agent can parse · August: 30.9%Pages an agent can parse · September: 32.2%Sites serving an llms.txt · May: 33.3%Sites serving an llms.txt · June: 33.0%Sites serving an llms.txt · July: 34.1%Sites serving an llms.txt · August: 34.3%Sites serving an llms.txt · September: 34.9%Sites with a valid llms.txt · May: 3.5%Sites with a valid llms.txt · June: 8.8%Sites with a valid llms.txt · July: 13.6%Sites with a valid llms.txt · August: 13.9%Sites with a valid llms.txt · September: 14.8%Sites with an AI crawler policyPages an agent can parseSites serving an llms.txtSites with a valid llms.txtMayJuneJulyAugustSeptember
The dimensions that make up the index, in percent of sites. Same months, same sample. Sites serving an llms.txt is not part of the index — it is here because having a file and having a valid one are different problems.

Every figure, and how much of the web moved with it

How to read these figures

A difference smaller than the wider of the two margins is not a small movement, it is a tie: with a different sample of pages it could have gone the other way. We label those = and we do not describe them as up or down. It is why the last column matters — a figure can move while almost none of the segments underneath it do.Every figure here comes from a sample, not from counting every site — so it carries a margin: how much it could move if we had happened to sample different pages. A figure measured on a thousand pages has a wide margin; one measured on a hundred thousand has a narrow one. It matters for one thing: two figures that differ by less than the wider of their margins are not one above the other, they are tied. That is why nothing here has a position number.
FigureSeptember 2026Last monthSegments that moved
AWCS21.0%▲ +0.9336 of 1,067
Pages an agent can parse32.2%▲ +1.3 pp531 of 1,067
Sites with an AI crawler policy11.1%▲ +0.2 pp257 of 822
Sites with a valid llms.txt14.8%▲ +0.9 pp356 of 1,094
Sites serving an llms.txt34.9%▲ +0.7 pp564 of 1,094
Sites exposing tools7,237▲ +63294 of 1,067
Google's category score58.4%▲ +0.9250 of 701
Layout stability88.1%▲ +0.51 of 1
Agents that can reach the site95.5%= −0.1 pp122 of 1,095
Forms an agent can fill98.6%= +0.2 pp0 of 1
Valid tool schemas98.5%= −0.1 pp0 of 1