Maintaining an llms.txt that nobody's reading
This site had an llms.txt that went stale in just over a week. Easy enough to fix, but what's the point of maintaining them if they're not being read?
On an idle whim, I created an llms.txt for this blog. I told myself, what could possibly go wrong?
And then I noticed: the contents of the file were already rotten after being live for a week and a half.
The file was static, and didn't take into account recent updates to the site. It still described a category structure I had retired and pointed at posts that were no longer relevant, among other things.
Fixing it took five minutes, give or take. But then it occurred to me. Are LLMs of any stripe actually reading the file? And if not, is the maintenance overhead really necessary?
What the file is meant to be
The pitch is simple. llms.txt is a short markdown file you put at the root of your site, like robots.txt but aimed at language models rather than search crawlers.
Proposed by Jeremy Howard in 2024, the format is meant to tell an AI reader what the site is, point at the pages that matter, and add whatever context you want it to carry when it summarises or cites you. A polite, machine-readable front door.
It's a nice idea. The trouble is the gap between the idea and what's happening in the wild.
Is anyone reading it?
Short answer: No, not really.
No major AI company has committed to reading it in production. Google spokespeople have said so repeatedly. Gary Illyes confirmed at Search Central Live that Google does not support llms.txt and is not planning to, and John Mueller likened it to the old keywords meta tag. Ouch!
More recently, the mighty G issued a comprehensive advisory about optimising for GenAI on search that also pooh-poohed the llms.txt format.
The other players – OpenAI, Anthropic, Meta, Mistral – still haven't confirmed they check for them. And that's easy enough to verify. John Mueller (him again) has noted that you can see in your own server logs that the AI crawlers don't bother requesting the file at all. Hard to ignore something more thoroughly than that.
So, if you are shipping an llms.txt to climb an imaginary AI ranking, you should stop and reconsider. In the current moment, it's about as practical as giving a best man speech while gargling with jelly.
Let's make one anyway
I knew the verdict before I started. The pattern of statements (and non-statements) on the topic is difficult to ignore. But I went and made an llms.txt file anyway.
The point was to walk the steps myself. There are lessons to be learned from doing a pointless thing like gargling with jelly, whereas someone telling me “you're just wasting your time” doesn't land in the same way.
I wanted to write the file, wire it into the build, and work out exactly how it was pointless and where, quietly, it might provide value.
This blog is, more than anything, a record of doing the thing to see what the thing teaches.
The trap is taking it too seriously
There is one danger with a file like this, though, which is to over-invest in it.
The worst pattern would be to generate a literal markdown copy of every single page on your domain. If those copies are indexable, you'll have manufactured duplicate content at scale, which is a real problem for your SEO. Don't do that.
The problem I encountered was smaller but the same shape. The original iteration of my llms.txt listed a bunch of individual posts by hand. That's a stupid thing to do, because as I continued to publish posts, the static file didn't keep pace.
In effect, I 'd cobbled together a substandard copy of my own sitemap. D'oh!
What I actually did
So I cut things back a bit.
The file now has two layers. First, a stable layer that'll only be updated when the site changes shape: the purpose, the category structure, the author and attribution rules, and the line clarifying that two of my three “authors” are AI personas and not real people.
Next, a pointer layer that hands any reader the live indexes the site already maintains: the sitemap, the RSS feed, the category and author pages. Those are solid so they can't fall out of date. The file points at them rather than badly duplicating them.
A snippet of my new and improved (and easier to maintain) llms.txt file.
Then a small guardrail. A short script in CI pulls every link out of the file and checks it still resolves, so a renamed slug or a dead page breaks the build instead of going stale unnoticed. It validates; it doesn't regenerate.
The point was to build an evergreen file that's correct by default and easy to maintain (on the working assumption that almost nobody human or machine is reading it anyway). Publishing a post, meanwhile, is precisely the action that shouldn't require touching this file.
And that was the takeaway from this experiment: to stop treating llms.txt as a second sitemap. I already knew it wouldn't bring traffic or citations, but now I know what information to leave in and what to leave out so it doesn't accidentally mislead with rotten content.
The verdict
llms.txt in 2026 is a courtesy note slipped under a door, which most of the big labs are declining to read.
Don't create one expecting visibility, citations or ranking, since the evidence says you won't get any of them.
Plus, if you do it the lazy way, you'll end up with a duplicate-content problem for your trouble.
The guidance from Contentful is that llms.txt is an experiment to keep near the bottom of the list, not a substitute for fundamentals. Which is exactly what I've done here.
The durable stuff, meanwhile, is good content that's well-structured, current, and easy to crawl and easy to trust.
