European websites get one human visitor back for every 179 AI bot visits — and their 'do not scrape' file is ignored three times as often
TollBit's State of the Bots report, covering the first half of 2026 across 3,906 publishers of which 456 are European, found the median European site was scraped four times more heavily than the median North American one. It received one human referral visit from AI apps for every 179 bot visits, a ratio more than three times worse than in North America, and its robots.txt instructions not to scrape were ignored nearly three times more often. About 15% of identified AI page fetchers in Europe reached URLs the site had marked disallowed. Page fetchers are the agents that load a single page in real time because a user asked a question — distinct from training or search crawlers. Three of them, ChatGPT-User, Bytespider and Youbot, each reached disallowed pages on close to half the European sites that had named them by hand in robots.txt. European publishers are also less defended: only 9% block Claude-User and 13% block Perplexity-User, against 26% each in North America. Scraping on European sites was about 20% higher in June than in January. OpenAI's own documentation argues robots.txt may not apply to ChatGPT-User because a user-initiated fetch is not autonomous crawling; Anthropic rejects that reading for its own agents.
Related stories
Norway moves to ban AI glasses from playgrounds and changing rooms.2026-10-06Under oath, four AI firms would not say if AI could be catastrophic.
2026-10-06Anthropic handed a user's diary entry to the police.
2026-10-05Two senators want prison time for bosses whose AI agents hack.
2026-10-05Musk's xAI got America's first AI nudify ban frozen in court.
2026-10-05