97% Of llms.txt Files Got No Requests, Ahrefs Data Shows

Ahrefs analyzed logs from 137,000 domains and located 97% of llms.txt recordsdata bought zero requests. No bots, no people.

The analysis used Ahrefs information to establish person brokers fetching recordsdata. Round 28% of 137,000 domains publish an llms.txt file, however since Ahrefs’ prospects are extra technical, the precise adoption on the broader internet is probably going decrease.

Of roughly 38,000 domains with legitimate recordsdata, solely about 1,100 acquired any site visitors.

Of recordsdata with requests, 96% got here from bots, principally non-AI. AI retrieval bots linked to ChatGPT and Perplexity made up 1%.

Who Fetches llms.txt Information

website positioning audit instruments had 21% requests, then unidentified bots (14%), internet crawlers like Googlebot (13%), and tech profiling instruments like BuiltWith (11%).

AI bots, throughout 4 classes, made up 19% of requests. AI is the biggest section, however the breakdown differs from most llms.txt advocates’ expectations.

Coding brokers despatched 10% of requests, coaching crawlers 5%, assistants 2%. Claude-Code and GPTBot had been the highest particular person bots.

Slackbot alone fetched llms.txt recordsdata extra typically than PerplexityBot did.

The Trade Learning Itself

The report discovered 12% of requests from instruments that audit, scan, or research llms.txt recordsdata moderately than use them.

GEO and AEO readiness instruments despatched 5% of requests; devoted scanners and validators despatched 3%, greater than AI retrieval bots and assistants mixed. Analysis bots despatched 2%, with the biggest analysis crawler figuring out as a immediate injection survey.

An ecosystem has developed round scoring and cataloging a file format earlier than a big viewers seems.

No AI Bot Appears For Information That Don’t Exist

Requests for /llms.txt paths with 404 errors drew no AI site visitors. People hitting these 404s appear to be individuals typing the URL into browsers, possible checking rivals.

The Chrome Lighthouse llms.txt audit, which reignited the llms.txt debate in Could, generated about 22 requests throughout the dataset, roughly 1 in 1,000.

Why This Issues

The info strains up with what Google’s John Mueller has said about llms.txt for over a yr. Lily Ray pressed Mueller on the hole between Google Search’s dismissal and Chrome’s Lighthouse audit. He stated llms.txt is “not performed for search” and referred to as it a “momentary crutch, maybe to avoid wasting tokens” for AI coding instruments.

The info exhibits the file’s viewers is coding brokers and coaching crawlers, not AI search and retrieval bots that may generate citations.

We reported on the break up between Google Search and Lighthouse documentation in Could. SE Rating’s earlier analysis of 300,000 domains confirmed no connection between having llms.txt and AI quotation frequency. Ahrefs’ information factors to 1 potential cause: the bots most instantly tied to reside AI retrieval barely requested these recordsdata in Could.

Trying Forward

The immediate injection discovering is price watching. Ahrefs discovered a crawler learning llms.txt as a immediate injection threat, since brokers belief ingested content material. Websites auto-generating these recordsdata by way of CMS ought to evaluation their content material.

Each determine on this report is a ceiling. Ahrefs measured requests, not whether or not bots acted on what they fetched.

Featured Picture: sdecoret/Shutterstock