European websites get one human visitor back for every 179 AI bot visits — and their 'do not scrape' file is ignored three times as often

TollBit's State of the Bots report, covering the first half of 2026 across 3,906 publishers of which 456 are European, found the median European site was scraped four times more heavily than the median North American one. It received one human referral visit from AI apps for every 179 bot visits, a ratio more than three times worse than in North America, and its robots.txt instructions not to scrape were ignored nearly three times more often. About 15% of identified AI page fetchers in Europe reached URLs the site had marked disallowed. Page fetchers are the agents that load a single page in real time because a user asked a question — distinct from training or search crawlers. Three of them, ChatGPT-User, Bytespider and Youbot, each reached disallowed pages on close to half the European sites that had named them by hand in robots.txt. European publishers are also less defended: only 9% block Claude-User and 13% block Perplexity-User, against 26% each in North America. Scraping on European sites was about 20% higher in June than in January. OpenAI's own documentation argues robots.txt may not apply to ChatGPT-User because a user-initiated fetch is not autonomous crawling; Anthropic rejects that reading for its own agents.

✓ Verified · 4 sources

Related stories