ChatGPT's Fetch Bot Bypassed robots.txt on 54% of Its Scrapes, TollBit Finds

New data from the content-licensing firm TollBit puts numbers on something the major AI operators have been saying in their documentation for months: robots.txt does not reliably stop the bots that fetch a page live at the moment a user asks a question. In the first half of 2026, roughly 15% of AI scrapes bypassed an active disallow instruction, a rate TollBit reports for European and North American sites alike.

The report, “The Bad Bots”, covers Q1 and Q2 2026 and draws on identified bot traffic across 3,906 publishers, 456 of them European. It matters because much of the bypassing is not coming from training crawlers that a site might reasonably want to block for licensing reasons. A large share comes from the agent fetchers that retrieve a page in real time to answer a live query, and for many sites those are exactly the requests that lead to a citation.

Which bots bypass most?

TollBit defines a bypass as a scrape of content the publisher had explicitly disallowed that bot from taking, and counts any such request regardless of what the operator’s documentation claims about its own compliance.

On that measure ChatGPT-User had the highest bypass rate, ignoring explicit robots.txt instructions on 54% of all its scrapes. Bytespider followed at 48% and PerplexityBot at 42%. Measured by reach rather than rate, ChatGPT-User, Bytespider and Youbot each bypassed robots.txt on nearly half of all European sites that had explicitly disallowed them.

Two of those three are not agent fetchers, which matters for how you read the number. Bytespider is ByteDance’s training crawler, and PerplexityBot serves both indexing and live retrieval. The bypassing is concentrated in a handful of agents, but it is not confined to the tier whose operators publicly reserve the right to ignore the file.

Why doesn’t robots.txt stop these bots?

For the agent tier, because the operators have decided it does not have to. OpenAI’s crawler documentation states that ChatGPT-User visits a page when a ChatGPT user asks a question, and that “because these actions are initiated by a user, robots.txt rules may not apply.” The same page states that OpenAI’s robots.txt tags govern OAI-SearchBot and GPTBot, having removed ChatGPT-User from that list in a December 2025 revision. OAI-SearchBot, not ChatGPT-User, is the agent that decides whether a site appears in ChatGPT’s search results, and it still honours robots.txt.

Perplexity documents the same position for Perplexity-User, which it says generally ignores the file for the same user-initiated reason. Anthropic takes the stricter line and states all three of its bots respect robots.txt. The split is why our AI crawler reference treats agent fetchers as a separate control problem from training and search-indexing crawlers: for the agent tier, a robots.txt entry is a request the operator may decline to honour, not an enforced rule.

Europe blocks the old crawlers harder and the new agents barely at all

The regional split is the more useful finding, and it runs in opposite directions depending on which tier you look at.

For the established training and indexing crawlers, European sites block more often than North American ones: 63% of European sites disallow CCBot against 46% in North America, 60% disallow GPTBot against 41%, 53% ClaudeBot against 43%, 50% PerplexityBot against 40%, and 46% OAI-SearchBot against 34%.

For the newer real-time agents that fetch a page live to answer a prompt, the ordering reverses and North America pulls ahead, often by more than double. Only 9% of European sites disallow Claude-User against 26% in North America, and 13% disallow Perplexity-User against 26%. Most of the newest agents sit in single digits across Europe. ChatGPT-User is the exception, blocked at a rate closer to the established crawlers, most likely because it is the most widely recognised.

So European publishers are not lax about AI access in general. They are ahead on the tier robots.txt does control and behind on the tier it does not, which is the worse way round to have it. If you run a European site and have audited your robots.txt only for GPTBot and CCBot, the agent user-agents are the gap.

What this means for site owners

If your intent is to keep agent fetchers off specific content, robots.txt alone will not do it. The reliable control is at the server or CDN layer: firewall rules, IP or user-agent blocking, or a bot-management product that can act on the request before it reaches your content. Cloudflare’s three-way crawler categories, which gate Search, Agent and Training bots independently at the network layer, are one way to enforce a decision that robots.txt only advises.

Two caveats keep this from being a blanket recommendation. First, blocking agent fetchers is a trade-off, not a free win: the same request that reads your page live is often the one that produces a citation in ChatGPT or Perplexity, so blocking it can cost you visibility in those answers rather than protecting anything valuable. Decide per content type, and keep the well-behaved search-indexing crawlers such as OAI-SearchBot allowed if you want to be cited. Second, robots.txt remains the right tool for the licensing decision it was always meant to express, and most training crawlers still honour it, but do not assume that holds indefinitely. TollBit’s quarter-on-quarter figures show GPTBot, CCBot and DuckAssistBot all bypassing more often in Q2 than in Q1, which the company reads as an early sign the behaviour may be spreading beyond the few agents that dominate it today. Check your server logs rather than your robots.txt to find out what actually arrived.

How much weight to put on the numbers

These are TollBit’s own figures, drawn from the publishers on its platform rather than an independent audit of the open web, so read them as a large sample rather than a census. They also cover only identified bots, which TollBit notes means the true scale is likely higher.

On the regional gap specifically, other datasets disagree. Cloudflare told Digiday its Radar data does not show European publishers being hit harder than North American ones in absolute terms, though European sites do see a higher share of their total requests come from bots. DataDome also reports no consistent Europe-North America gap, and points to enormous variance between individual publishers. TollBit’s own cofounder and an INMA researcher both offered the same speculative explanation, that Europe’s language mix draws more scraping, and the researcher separately noted the composition of TollBit’s customer base could account for some of it. The bypass measurements are the sturdier part of the report; the regional causation is not settled.

Sources

• State of the Bots, 2026 Q1 & Q2: The Bad Bots — TollBit
• Bots documentation — OpenAI
• OpenAI Says Robots.txt May Not Apply To ChatGPT’s Fetch Bot — Search Engine Journal
• 15% of AI page fetchers in Europe reached disallowed URLs, TollBit finds — PPC Land
• European publishers are getting hit harder by AI bot scraping, report finds — Digiday