AI agent access
21.7% of sites have no robots.txt with rules. Among those that do, 93.3% have no rule for AI agents, and 3.1% explicitly allow at least one. The chart shows the percentage of sites blocking each agent by name (Disallow: /).
- Bytespider 2.6%
- GPTBot 2.5%
- Amazonbot 2.3%
- meta-externalagent 2.3%
- ClaudeBot 2.2%
- CCBot 2.2%
- Google-Extended 2.0%
- Applebot-Extended 2.0%
- anthropic-ai 1.9%
- ChatGPT-User 0.3%
- PerplexityBot 0.2%
- OAI-SearchBot 0.1%
Training agents: Bytespider, GPTBot, Amazonbot, meta-externalagent, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, anthropic-ai. Search and user-request agents: ChatGPT-User, PerplexityBot, OAI-SearchBot.
How many agents each site blocks
Among sites that block at least one agent. The average is 6.7 agents per site.
- 1 agent 13.6%
- 2 to 5 agents 20.1%
- 6 to 11 agents 65.2%
- All 12 agents 1.1%
11.5% of sites that block also stop at least one search or user-request agent. The other 88.5% only block training agents.
Blocking by platform
- PrestaShop 5.8%
- None detected 4.1%
- Joomla 2.8%
- WordPress 2.2%
- WooCommerce 2.2%
- Shopify 2.0%
- Wix 0.0%
None of the 156 Wix sites block AI agents. On sites with no detected platform (custom code or unidentified platforms), blocking is 4.1%. PrestaShop, Joomla and Wix have 131 to 195 sites in the sample, so the differences are indicative.