ChatGPT Now Crawls 3.6x More Than Googlebot

This publish was sponsored by Alli AI. The opinions expressed on this article are the sponsor’s personal. 

Everybody assumes Googlebot is the dominant crawler hitting their web site. That assumption is now improper.

We analyzed 24,411,048 proxy requests throughout 78,000+ pages on 69 buyer web sites on Alli AI’s crawler enablement platform over a 55-day interval (January to March 2026). OpenAI’s ChatGPT-Consumer crawler made 3.6x extra requests than Googlebot throughout our information pattern. And that’s not even counting GPTBot, OpenAI’s separate coaching crawler.

A be aware on methodology: Crawler identification used person agent string matching, verified in opposition to printed IP ranges. Request metrics are measured on the proxy/CDN layer. The dataset covers 69 web sites throughout a wide range of industries and sizes, predominantly WordPress-based. Full methodology is detailed on the finish.

Discovering 1: AI Crawlers Now Outpace Google 3.6x & ChatGPT Leads the Pack

After we ranked each recognized crawler by request quantity, the outcomes had been unambiguous:

ChatGPT-Consumer made extra requests than Googlebot, Amazonbot, and Bingbot mixed.

Grouped by function, AI-related crawlers (ChatGPT-Consumer, GPTBot, ClaudeBot, Amazonbot, Applebot, Bytespider, PerplexityBot, CCBot) made 213,477 requests versus 59,353 for conventional search crawlers (Googlebot, Bingbot, YandexBot). AI crawlers at the moment are making 3.6x extra requests than conventional search crawlers throughout our community.

Discovering 2: OpenAI Makes use of 2 Crawlers (And Most Websites Don’t Know the Distinction)

OpenAI operates two distinct crawlers with very totally different functions.

ChatGPT-Consumer is the retrieval crawler. It fetches pages in actual time when customers ask ChatGPT questions that require up-to-date net data. This determines whether or not your content material seems in ChatGPT’s solutions.

GPTBot is the coaching crawler. It collects information to enhance OpenAI’s fashions. Many websites block GPTBot through robots.txt however not ChatGPT-Consumer, or vice versa, with out understanding the distinct penalties of every.

Mixed, OpenAI’s crawlers made 142,225 requests: 3.8x Googlebot’s quantity.

The robots.txt directives are separate:

Consumer-agent: GPTBot      # Coaching crawler — feeds OpenAI's fashions
Consumer-agent: ChatGPT-Consumer # Retrieval crawler — fetches pages for ChatGPT solutions

Discovering 3: AI Crawlers Are Quicker & Extra Dependable, However Their Quantity Provides Up

AI crawlers are considerably extra environment friendly per request:

Two possible causes. First, AI retrieval crawlers are fetching particular pages in response to person queries, not exhaustively discovering web site structure. They know what they need, they seize it, they usually go away. Second, whereas all crawlers on our infrastructure obtain pre-rendered responses, Googlebot’s broader crawl sample means it requests a wider vary of URLs, together with stale paths from sitemaps and its personal legacy index, which provides latency from redirect chains and error dealing with that retrieval crawlers keep away from fully.

However there’s a catch: whereas every particular person request is light-weight, the sheer quantity means combination server load is substantial. ChatGPT-Consumer at 11ms × 133,361 requests remains to be an actual infrastructure value, simply distributed in a different way than Googlebot’s fewer, heavier requests.

Discovering 4: Googlebot Sees a Completely different (Worse) Model of Your Web site

Googlebot’s 96.3% success fee versus near-perfect charges for AI crawlers reveals an necessary structural distinction.

Googlebot obtained 624 blocked responses (403) and 480 not discovered errors (404), accounting for 3% of its requests. In the meantime, ChatGPT-Consumer achieved 99.99% success. PerplexityBot hit an ideal 100%.

Why the hole? The almost certainly clarification is index age and crawl habits, not web site misconfiguration.

Googlebot maintains an enormous legacy index constructed over years of steady crawling. It routinely re-requests URLs it already is aware of about — together with pages which have since been deleted (404s) or restructured (403s). That is regular habits for a search engine sustaining an index of this scale, nevertheless it means a significant proportion of Googlebot’s requests are directed at URLs that not exist.

AI crawlers don’t carry that baggage. ChatGPT-Consumer fetches particular pages in response to real-time person queries, focusing on content material that’s at the moment related and linked. That’s a structural benefit that produces near-perfect success charges.

Business Reviews Verify AI Crawling Surged 15x in 2025

These findings align with broader business developments. Cloudflare’s 2025 evaluation reported ChatGPT-User requests surging 2,825% YoY, with AI “person motion” crawling rising greater than 15x over the course of 2025. Akamai recognized OpenAI as the single largest AI bot operator, accounting for 42.4% of all AI bot requests. Vercel’s evaluation of nextjs.org confirmed that none of the major AI crawlers currently render JavaScript.

Our information exhibits this crossover could already be taking place on the web site degree for properties that actively allow AI crawler entry.

Your New web optimization Technique: How To Audit, Clear Up & Optimize For AI Crawlers

1. Audit your robots.txt for AI crawlers in the present day

Most robots.txt recordsdata had been written for a Googlebot-first world. At minimal, have specific directives for ChatGPT-Consumer, GPTBot, ClaudeBot, Amazonbot, PerplexityBot, Applebot, Bytespider, CCBot, and Google-Prolonged.

Our suggestion: Most companies profit from permitting each retrieval crawlers (ChatGPT-Consumer, PerplexityBot, ClaudeBot) and coaching crawlers (GPTBot, CCBot, Bytespider), coaching information is what teaches these fashions about your model, merchandise, and experience. Blocking coaching crawlers in the present day means AI fashions be taught much less about you tomorrow, which reduces your possibilities of being cited in AI-generated solutions down the road.

The exception: when you’ve got content material you particularly want to guard from mannequin coaching (proprietary analysis, gated content material), use granular Disallow guidelines for these paths moderately than blanket blocks.

2. Clear up stale URLs in Google Search Console

Our information exhibits Googlebot hits a 3% error fee, principally 403s and 404s, whereas AI crawlers obtain near-perfect success charges. That hole possible displays Googlebot re-crawling legacy URLs that not exist. However these failed requests nonetheless eat the crawl finances.

Audit your GSC crawl stats for recurring 404s and 403s. Arrange correct redirects for restructured URLs and submit up to date sitemaps.

3. Deal with AI crawler accessibility as a definite web optimization channel

Rating in ChatGPT’s solutions, Perplexity’s outcomes, and Claude’s responses is rising as a definite visibility channel. In case your content material isn’t accessible to those crawlers, notably when you’re working JavaScript-heavy frameworks, you’re invisible in AI search.

We’ve printed a live dashboard displaying how AI crawler visitors breaks down throughout an actual web site: which platforms are visiting, how typically, and their share of complete visitors; if you wish to see what this appears to be like like in follow.

4. Plan for quantity, not simply particular person request weight

AI crawlers ship mild, quick requests, however they ship many of them. ChatGPT-Consumer alone accounted for greater than 133,000 requests in 55 days. The combination server load from AI crawlers is now possible exceeding your Googlebot load. Make certain your internet hosting and CDN can deal with it, the low per request response occasions in our information mirror the truth that Alli AI serves pre-rendered static HTML from the CDN edge, which is precisely the type of structure that absorbs this quantity with out taxing your origin server.

Methodology

This evaluation is predicated on 24,411,048 HTTP proxy requests processed via Alli AI’s crawler enablement platform between January 14 and March 9, 2026, masking 69 buyer web sites.

Crawler identification used person agent string matching, verified in opposition to printed IP ranges. For OpenAI crawlers particularly, each request was cross-referenced in opposition to OpenAI’s printed CIDR ranges. This confirmed 100% of GPTBot requests and 99.76% of ChatGPT-Consumer requests originated from OpenAI’s infrastructure. The remaining 0.24% (requests from spoofed person brokers) had been excluded.

Limitations: The dataset is scoped to Alli AI prospects who’ve opted into crawler enablement. Crawlers that don’t self-identify through person agent usually are not captured. Response time measurements are on the proxy layer, not the origin server.

About Alli AI

Alli AI supplies server-side rendering infrastructure for AI and search engine crawlers. This evaluation was produced utilizing information from our proxy infrastructure to assist the web optimization neighborhood higher perceive the evolving crawler panorama.

Wish to see this information in motion? See the breakdown firsthand by visiting our AI visibility dashboard.

Picture Credit

Featured Picture: Picture by Alli AI. Used with permission.

In-Put up Iamges: Photos by Alli AI. Used with permission.