Some AI Agents Reach Pages Sites Told Them To Skip

ChatGPT’s page-fetching bot is disallowed by extra websites than every other AI bot of its sort. It additionally reached disallowed pages on extra websites than every other bot. OpenAI says robots.txt guidelines could not apply to it as a result of an individual requested for the web page.

TollBit’s newest State of the Bots report has the numbers for the primary half of 2026. Right here’s what else the information reveals about how these crawlers behave and what it means to your website.

The place The Bypasses Land

Within the European websites mentioned within the report, about 15% of recognized AI page-fetchers reached URLs that the websites had marked as disallowed.

This occurs largely with just a few particular brokers. For instance, ChatGPT-Consumer, Bytespider, and Youbot every accessed disallowed pages on practically half of the European websites that had explicitly listed them. Amongst these, ChatGPT-Consumer reached essentially the most websites.

Websites Did Disallow It

Lots of the newer page-fetching brokers are hardly blocked in any respect. Solely 9% of European web sites disallow Claude-Consumer, in comparison with 26% in North America. Perplexity-Consumer sits at 13% versus 26%.

Many of the latest brokers have disallow charges within the single digits throughout Europe, however ChatGPT-Consumer stands out as an exception.

What OpenAI Says About The Rule

OpenAI’s crawler documentation says ChatGPT-Consumer visits a web page when a ChatGPT consumer asks a query, and that as a result of these actions are initiated by a consumer, robots.txt guidelines could not apply.

Perplexity says Perplexity-Consumer typically ignores the file for a similar motive, however Anthropic has a distinct view and states that each one three of its bots respect it, as we reported in February. TollBit treats any request to a disallowed URL as a bypass, no matter what the operator claims.

Why This Issues

A disallow line for ChatGPT-Consumer is a request that OpenAI’s documentation says could not apply.

It’s necessary to have a look at a distinct side right here. In keeping with OpenAI’s documentation, the agent liable for deciding if a website reveals up in ChatGPT search outcomes known as OAI-SearchBot, not ChatGPT-Consumer. Websites that block each brokers to forestall AI visitors have traded away the visibility half of that deal and saved a fetching management that carries a carve-out.

Server logs or CDN information present what truly arrived. The file solely reveals what you requested for.

Trying Forward

Cloudflare is making some updates to the way it manages its crawler controls, transferring the choice to the community layer. With regards to the bots it acknowledges, compliance is now not left as much as the crawler itself. Ranging from September 15, new domains added to Cloudflare may have their Coaching and Agent crawlers blocked by default on pages with advertisements, whereas Search crawlers stay allowed.

Whether or not the user-initiated loophole survives is the open query. It rests on the argument that requesting a web page differs from a crawler taking it, and now all main assistants fetch pages this fashion.

Featured Picture: Internet Vector/Shutterstock