In March we added an llms.txt file to a client's site. It was on every SEO checklist doing the rounds, it took twenty minutes, and it felt like the responsible thing to do in a year when a growing share of traffic arrives through an AI assistant rather than a search results page.
Ninety days later we went back to the access logs to see what had come of it. In that window the file was requested exactly zero times by any AI crawler. The only hits it ever got were from us, checking that it deployed. Over the same period the same site served roughly fourteen thousand requests to crawlers operated by AI companies — every single one of them for ordinary HTML pages.
That gap is worth understanding, because the advice to "add an llms.txt" has hardened into received wisdom without anyone stopping to check whether the thing on the other end is listening.
What llms.txt Is Supposed to Be
The idea is straightforward and, on paper, sensible. You put a Markdown file at /llms.txt that describes your site to a language model: what it is, which pages matter, where the canonical documentation lives. Think of it as a sitemap written for a reader rather than a parser.
The pitch is that an assistant answering a question about your product should not have to guess which of your four hundred URLs is the real pricing page. Hand it a curated index and you get better answers and fewer mangled citations.
Nothing about that reasoning is wrong. The problem is entirely in the second half of the sentence — the part where someone reads the file.
A Standard Requires a Consumer
robots.txt is a standard because Googlebot fetches it before it crawls you, and because that behaviour is documented, tested, and enforced by the company that operates the crawler. sitemap.xml is a standard for the same reason. The file format is the least interesting part; the commitment on the other side is what makes it real.
llms.txt has the file format and not the commitment. As of this writing no major AI vendor has published documentation saying their crawler fetches it, and — more to the point — none of them appear in our logs asking for it. A convention that no client implements is a proposal, not a standard.
This is easy enough to verify on your own infrastructure. If you have access to raw logs, one line tells you whether anything has ever asked:
grep -c "GET /llms.txt" access.logRun the same count against robots.txt for comparison. On the client site above, the second number was in the hundreds. The first was zero.
What the Crawlers Actually Request
The AI crawlers hitting your site are not looking for a special file. They are doing what crawlers have always done: fetching HTML, following links, and reading robots.txt on the way in.
If you want to see which ones are visiting you, the user-agent strings are distinctive enough to grep for directly:
grep -oE "GPTBot|ClaudeBot|Claude-User|PerplexityBot|Google-Extended|CCBot|Bytespider|meta-externalagent" access.log \
| sort | uniq -c | sort -rnTwo useful things fall out of that command. The first is a real picture of who is reading your site, which is often quite different from what people assume — smaller crawlers frequently outnumber the ones that get written about. The second is a reminder that all of this traffic is arriving at your ordinary pages, through your ordinary server, governed by your ordinary crawl rules.
The Distinction Almost Everyone Gets Wrong
There are two categories of AI crawler and they deserve very different treatment.
Training crawlers collect content in bulk to build future models. GPTBot and CCBot are the familiar examples. Whether you allow them is a business decision about your content, and there is a defensible answer in either direction.
Retrieval fetchers are a different animal. They fire when a person has already asked a question and the assistant is going out to read a page in order to answer it. Blocking one of these does not protect your content from anything — it removes you from an answer a real potential customer is reading right now, and hands the citation to a competitor who left the door open.
The mistake we see most often is a blanket block written in a hurry, which catches both. If you want to opt out of training while staying visible in live answers, you have to name the agents individually:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /Note that the same caveat from ordinary SEO applies here: robots.txt is a request, not a fence. Crawlers that respect it will respect it, and crawlers that do not were never going to. If you need enforcement rather than etiquette, that belongs at the server or CDN layer, not in a text file.
The One Case for Keeping an llms.txt
None of this makes the file harmful. It costs a few kilobytes and it breaks nothing, and if a major vendor does adopt the convention next year, having one already in place is a small head start.
But keep it only under one condition: generate it from the same source as your sitemap, as part of whatever build or deploy step you already run. A file that is produced automatically stays honest for free.
A hand-maintained one does not. Six months after launch it describes a product tier you retired, links to two pages that now 404, and confidently states a price that changed in April. If anything ever does start reading it, that is the version it reads. A stale llms.txt is meaningfully worse than no llms.txt, because it replaces "no information" with "wrong information", stated in your own voice.
What Actually Makes a Site Readable to a Model
Here is the slightly deflating part. Everything that helps a language model understand your site is something we have been telling clients to do for fifteen years.
Put the content in the HTML. A page that renders its text through JavaScript after load is a page that some crawlers will record as empty, exactly as it was a decade ago with search. Use one <h1> that says what the page is. Use real heading levels underneath it rather than styled <div> elements. Put a visible date on anything time-sensitive, so a model quoting you can tell whether it is quoting something current.
We can see this play out across our own properties. The static, server-delivered pages get quoted back to us with their wording intact. The JavaScript-heavy sections get summarised vaguely, when they get picked up at all. No index file changed that; the markup did.
Optimise for What Actually Arrives
The appeal of llms.txt is that it is a discrete task. You can add it on a Tuesday, tick it off, and feel prepared for a shift in how people find things. Auditing your markup and deciding a real position on training crawlers is slower and harder to declare finished.
But the logs are not ambiguous. Ninety days, zero requests for the curated file, fourteen thousand requests for the pages themselves. Whatever attention these crawlers have to give your site, they are spending all of it on your actual HTML.
So spend your effort there. Make the pages clean, make the important ones reachable, decide deliberately which crawlers you want and write that down in the file they demonstrably read. Keep the llms.txt if you like — generated, not hand-written — and treat it as a bet on next year rather than a fix for this one.