97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows via @sejournal, @MattGSouthern

Ahrefs analyzed logs from 137,000 domains and found 97% of llms.txt files got zero requests. No bots, no humans.

The analysis used Ahrefs data to identify user agents fetching files. Around 28% of 137,000 domains publish an llms.txt file, but since Ahrefs’ customers are more technical, the actual adoption on the broader web is likely lower.

Of roughly 38,000 domains with valid files, only about 1,100 received any traffic.

Of files with requests, 96% came from bots, mostly non-AI. AI retrieval bots linked to ChatGPT and Perplexity made up 1%.

Who Fetches llms.txt Files

SEO audit tools had 21% requests, then unidentified bots (14%), web crawlers like Googlebot (13%), and tech profiling tools like BuiltWith (11%).

AI bots, across four categories, made up 19% of requests. AI is the largest segment, but the breakdown differs from most llms.txt advocates’ expectations.

Coding agents sent 10% of requests, training crawlers 5%, assistants 2%. Claude-Code and GPTBot were the top individual bots.

Slackbot alone fetched llms.txt files more often than PerplexityBot did.

The Industry Studying Itself

The report found 12% of requests from tools that audit, scan, or study llms.txt files rather than use them.

GEO and AEO readiness tools sent 5% of requests; dedicated scanners and validators sent 3%, more than AI retrieval bots and assistants combined. Research bots sent 2%, with the largest research crawler identifying as a prompt injection survey.

An ecosystem has developed around scoring and cataloging a file format before a significant audience appears.

No AI Bot Looks For Files That Don’t Exist

Requests for /llms.txt paths with 404 errors drew no AI traffic. Humans hitting those 404s seem to be people typing the URL into browsers, likely checking competitors.

The Chrome Lighthouse llms.txt audit, which reignited the llms.txt debate in May, generated about 22 requests across the dataset, roughly 1 in 1,000.

Why This Matters

The data lines up with what Google’s John Mueller has said about llms.txt for over a year. Lily Ray pressed Mueller on the gap between Google Search’s dismissal and Chrome’s Lighthouse audit. He said llms.txt is “not done for search” and called it a “temporary crutch, perhaps to save some tokens” for AI coding tools.

The data shows the file’s audience is coding agents and training crawlers, not AI search and retrieval bots that would generate citations.

We reported on the split between Google Search and Lighthouse documentation in May. SE Ranking’s earlier analysis of 300,000 domains showed no connection between having llms.txt and AI citation frequency. Ahrefs’ data points to one possible reason: the bots most directly tied to live AI retrieval barely requested these files in May.

Looking Ahead

The prompt injection finding is worth watching. Ahrefs found a crawler studying llms.txt as a prompt injection risk, since agents trust ingested content. Sites auto-generating these files via CMS should review their content.

Every figure in this report is a ceiling. Ahrefs measured requests, not whether bots acted on what they fetched.

Featured Image: sdecoret/Shutterstock