llms.txt is a dud: the GEO advice you can safely ignore

Short answer: no, llms.txt does not work. Not "it works a bit", not "it can't hurt". The major AI companies have never said they use it, server log studies show their crawlers barely request it, and Google's own search liaison has compared it to the keywords meta tag, the most famously ignored piece of metadata in web history. If an agency has put llms.txt at the top of your AI visibility plan, that tells you more about the agency than about AI search.

We run AI-heavy content and outbound systems for UK businesses every week, through products like Contentwell and Bidwell and through client builds, so we have a commercial interest in AI search working well. That is exactly why we will not sell you this particular file.

"A file nobody fetches is not a strategy. It is a checkbox pretending to be one."

Dean Cookson, founder, Operosus

What is llms.txt supposed to do?

The idea sounds reasonable on paper. You place a markdown file at yoursite.com/llms.txt that summarises your site and points language models at your most important pages, a sort of curated reading list for AI. The pitch is that ChatGPT, Claude, Gemini and Perplexity will read it, understand you better, and cite you more often in their answers.

The proposal came from the developer community in 2024 and spread fast. Plugins appeared for every CMS, generators popped up everywhere, and "add llms.txt" landed on thousands of SEO audit checklists, usually billed as an easy win for generative engine optimisation (the tactics that do have evidence behind them are in our GEO guide for SMBs).

There is one problem. For the file to do anything, AI crawlers have to fetch it and AI companies have to act on it. Neither is happening.

Do the AI companies actually use it?

No, and they have not been shy about it. OpenAI, Anthropic and Google have published plenty of documentation about how their crawlers work and how to control them, all of it built on robots.txt, the standard that has existed since the 1990s. None of them lists llms.txt as something their systems read.

Google's John Mueller put it bluntly in a Reddit discussion in April 2025: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag." For anyone who missed that era, the keywords meta tag was the field site owners stuffed with whatever they wanted to rank for, until search engines stopped trusting self-declared claims and ignored it entirely. Mueller's point is the same here: if a bot has to crawl your real pages anyway to verify what your llms.txt claims, the file adds nothing.

What does the log data show?

This is the part that should end the conversation. Two independent studies have now looked at real server logs and real traffic, and both came back empty-handed.

OtterlyAI monitored AI bot traffic on a test site for 90 days after implementing llms.txt correctly at the root. Out of more than 62,100 AI bot visits, just 84 requests touched the file. That is roughly 0.1% of AI bot traffic. Their average ordinary content page received around 265 AI bot visits over the same window, so a normal blog post got about three times more attention from AI crawlers than the file specifically designed for them. Their conclusion: the file "did not correlate with any noticeable uptick in overall AI bot activity or shift in crawling patterns".

A separate Search Engine Land study tracked 10 websites across finance, B2B SaaS, ecommerce, insurance and pet care, comparing 90 days before and after adding llms.txt. Eight of the ten sites saw no measurable change in AI traffic. Two sites grew, but the analysis attributed the growth to things like new comparison content, downloadable templates and press coverage, not the file. One site actually declined.

So the people who would need to honour the standard say they do not use it, and the logs confirm their bots are not even looking. There is no version of "it can't hurt" that survives contact with that evidence. It costs time, it crowds out work that matters, and it teaches you to measure activity instead of results.

Why did it catch on anyway?

Because it is easy, and easy sells. AI search genuinely is reshaping how buyers find suppliers, and business owners are rightly asking what to do about it. "Install this file" is a much more comfortable answer than "improve your content, your structure and your authority", because it takes ten minutes and produces a visible deliverable. It is the same dynamic that kept agencies billing for keyword meta tags years after they stopped mattering.

There is also a softer version of the pitch worth addressing: that llms.txt is harmless future-proofing, in case the standard gets adopted later. Maybe. But standards on the web get adopted when the big platforms commit to them, and after two years not one has. If that changes, adding the file takes an afternoon. You lose nothing by waiting for evidence.

What should you do instead?

The unglamorous truth is that the things that make you visible to AI assistants overlap heavily with the things that make you visible to Google, because the assistants lean on the same crawled and indexed web.

Start with access. Check your robots.txt and your firewall rules, because plenty of sites are blocking GPTBot, ClaudeBot and PerplexityBot without realising it, sometimes via a CDN setting nobody remembers switching on. You cannot be cited by a system you have locked out.

Then make your content readable without JavaScript gymnastics. If the substance of a page only appears after client-side rendering, you are gambling on every crawler executing your code. Server-rendered HTML with clear headings and direct answers near the top is the safe bet, and it is how we build every content system at Operosus.

Then publish things worth citing. Assistants reach for sources when a page answers a specific question plainly, or contains something that exists nowhere else: your pricing, your process, your data, your honest opinion. The Princeton GEO study measured which tactics move citations, and they are all evidence tactics, not configuration files. The Search Engine Land study above is instructive here, because the sites that gained AI traffic did it with genuinely useful new content, not configuration files.

Finally, look at your own logs. This is the habit that separates evidence from folklore. Your server logs will tell you exactly which AI bots visit, what they fetch and how often, and that data should drive what you invest in next. It is the same principle we apply across our own products: when Bidwell drafts a tender response or Contentwell plans an article, the decisions trace back to real source material, not assumptions.

AI search optimisation is real work. It is mostly the old work, done properly, plus a willingness to check what the machines are actually doing rather than what a checklist says they should do. Skip the magic file. Spend the time on something a bot will actually read.