Maintaining an llms.txt that nobody's reading
This site had an llms.txt that went stale in just over a week. Easy enough to fix, but what's the point of maintaining them if they're not being read?
On an idle whim, I created an llms.txt for this blog. I told myself, what could possibly go wrong?
And then I noticed: the contents of the file were already rotten after being live for a week and a half.
The file was static, and didn't take into account recent updates to the site. It still described a category structure I had retired and pointed at posts that were no longer relevant, among other things.
Fixing it took five minutes, give or take. But then it occurred to me. Are LLMs of any stripe actually reading the file? And if not, is the maintenance overhead really necessary?
What the file is meant to be
The pitch is simple. llms.txt is a short markdown file you put at the root of your site, like robots.txt but aimed at language models rather than search crawlers.
Proposed by Jeremy Howard in 2024, the format is meant to tell an AI reader what the site is, point at the pages that matter, and add whatever context you want it to carry when it summarises or cites you. A polite, machine-readable front door.
It's a nice idea. The trouble is the gap between the idea and what's happening in the wild.
Is anyone reading it?
Short answer: No, not really.
No major AI company has committed to reading it in production. Google spokespeople have said so repeatedly. Gary Illyes confirmed at Search Central Live that Google does not support llms.txt and is not planning to, and John Mueller likened it to the old keywords meta tag. Ouch!
Finally, the mighty G issued a comprehensive advisory about optimizing for GenAI on search as recently as last month.
The other players – OpenAI, Anthropic, Meta, Mistral – have so far not confirmed they check for them. And you don't even have to take their word for it. John Mueller (him again) has noted that you can see in your own server logs that the AI crawlers don't bother requesting the file at all. Hard to ignore something more thoroughly than that.
So, if you are shipping an llms.txt to climb an imaginary AI ranking, you should stop and reconsider. In the current moment, it's about as practical as giving a best man speech while gargling with jelly.
Let's make one anyway
I knew the verdict before I started. The pattern of statements (and non-statements) on the topic is difficult to ignore. But I went and made an llms.txt file anyway.
The point was to walk the steps myself. There's lessons to be learned from doing a pointless thing like gargling with jelly, whereas someone telling me “you're just wasting your time” doesn't land in the same way.
I wanted to write the file, wire it into the build, and work out exactly how it was pointless and where, quietly, it might provide value.
This blog is, more than anything, a record of doing the thing to see what the thing teaches.
The trap is taking it too seriously
There is one danger with a file like this, which is to over-invest in it.
The worst pattern would be to generate a literal markdown copy of every single page on your domain. If those copies are indexable, you'll have manufactured duplicate content at scale, which is a real problem for your SEO. Don't do that.
The problem I encountered was smaller but the same shape. The original iteration of my llms.txt listed a bunch of individual posts by hand. That's a stupid thing to do, because as I continue to publish posts, the static file did not keep pace.
In effect, I had cobbled together a substandard copy of my own sitemap. D'oh!
What I actually did
I cut things back a bit.
The file now has two layers. A stable layer that will only be updated when the site changes shape; the purpose, the category structure, the author and attribution rules, the line clarifying that two of my three “authors” are AI personas and not real people.
And a pointer layer that hands any reader the live indexes the site already maintains; the sitemap, the RSS feed, the category and author pages. Those are solid, they can't fall out of date. The file points at them rather than badly duplicating them.
A snippet of my new and improved (and easier to maintain) llms.txt file.
Then a small guardrail. A short script in CI pulls every link out of the file and checks it still resolves, so a renamed slug or a dead page fails loudly instead of sitting there quietly wrong. It validates, it does not regenerate.
The point was to build an evergreen file that's correct by default and easy to maintain (on the working assumption that almost nobody human or machine is reading it anyway). Publishing a post, meanwhile, is precisely the thing that shouldn't require touching this file.
And that was the takeaway from this experiment: to stop treating llms.txt as a second sitemap. I already knew it wouldn't bring traffic or citations, but now I know what information to leave in and what to leave out so it doesn't accidentally mislead with rotten content.
The verdict
llms.txt in 2026 is a courtesy note slipped under a door, which most of the big labs are declining to read.
Don't create one expecting visibility, citations or ranking, since the evidence says you won't get any of them.
Plus, if you do it the lazy way, you'll end up with a duplicate-content problem for your trouble.
The durable stuff is good content that's well-structured, current, and easy to crawl and easy to trust. Contentful's own write-up lands in the same place from the strategy side, an experiment to keep near the bottom of the list, not a substitute for fundamentals.
Write one only if you're curious enough to conduct an experiment, like I am, and it doesn't eat into precious time you could be spending on other things.
