Do I need an llms.txt file?
What llms.txt is, what Google actually said about it, the claim everyone repeats that does not mean what they think, and how to check it on your own logs.
Key takeaways
- llms.txt is a proposal from September 2024, not a standard, and no major AI provider has confirmed consuming it.
- Google's John Mueller said in April 2025 that server logs show AI services do not even request the file, and compared it to the keywords meta tag.
- The AI labs publishing their own llms.txt is about their developer documentation and says nothing about whether their crawlers read yours.
- Check your robots file for the AI crawlers first, GPTBot and ClaudeBot and PerplexityBot among them, because that is what actually gates whether assistants can read you.
On this page11 sections
Almost certainly not yet, and definitely not instead of the things that do work. llms.txt is a proposed file that hands AI systems a clean map of your site. It is cheap to add and harmless to have. It is also, as far as anyone can demonstrate, not being read.
What is an llms.txt file?
A markdown file at the root of your domain listing your important pages with a line of context each. The idea is that a model reading your site gets a curated index instead of guessing from navigation and scripts. It was proposed by Jeremy Howard in September 2024.
It is a proposal, not a standard. No standards body has adopted it and no major model provider has committed to consuming it. That distinction matters enormously when somebody tells you it is required.
What has Google said about llms.txt?
This is the part usually left out. Google's John Mueller addressed it directly in April 2025:
"AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag."
The server log point is the one that should settle it for you. This is not a claim about whether the file is a good idea. It is an observation that the crawlers are not requesting it, which you can verify on your own logs in about a minute.
The keywords meta tag comparison is pointed, and fair. That tag was also a way for a site owner to declare what their site was about, and it also died because a declaration you write about yourself is the least trustworthy signal available to a machine that can just read the page.
Do OpenAI and Anthropic actually read it?
You will be told that OpenAI, Anthropic and Google all publish llms.txt files, therefore it works. Both halves are true and the conclusion does not follow.
They publish llms.txt files for their own developer documentation. That is a convenience for developers pointing a coding assistant at an API reference. It says nothing whatsoever about whether their web crawlers read yours. Publishing a format and consuming a format are different acts, and this particular conflation is why the topic is so confidently misreported.
If you see someone assert that a named assistant reads llms.txt during retrieval, ask for the announcement. There isn't one.
How is llms.txt different from robots.txt?
robots.txt says who may crawl what. It is a permission file, it is honoured widely, and it has actual consequences.
llms.txt says here is what matters and why. It is a hint file, honouring it is optional, and at present nobody appears to be taking the hint.
They solve different problems, so one never replaces the other, and a site that gets llms.txt right while blocking crawlers in robots.txt has achieved nothing at all.
What should I do instead, today?
Open your robots file and check whether you are blocking AI crawlers. Plenty of sites do without realising, sometimes because a hosting provider or a copied template added the rule. Look for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. If assistants cannot read your pages, a curated index of those pages helps nobody.
While you are there, one correction worth having. Google-Extended is not the switch for Google's AI answers. Google's crawler documentation says it governs whether your content trains future Gemini models and that it does not affect inclusion in Google Search or act as a ranking signal. AI Overviews are built from the ordinary index.
How do I check my own server logs?
Mueller's argument is falsifiable, which is rare in this field, so you can settle it for your own site rather than taking his word or mine.
Open your server or CDN access logs, filter for requests to the path, and look at what user agents appear. On most hosts that is a search box and thirty seconds. If assistant crawlers are fetching your llms.txt regularly, you have evidence that beats every blog post written about this, including this one.
Do the same for a page you actually care about, and compare. If GPTBot pulls your articles every week and has never once requested llms.txt, that is your answer, measured on your own traffic.
What does a usable llms.txt look like?
Keep it short. Your name, one sentence on what you do, then your genuinely important pages grouped under plain headings, each with a short description of what that page answers.
If it runs past a screen or two you are listing pages that do not deserve it, and you have turned a curated index back into the sitemap it was supposed to improve on.
So should I add one or not?
Add it if it takes twenty minutes, you already have pages worth listing, and you understand you are placing a small bet on a proposal rather than fixing anything. There is a reasonable argument that being early costs nothing.
Skip it if you are choosing between this and writing the pages themselves. The file is an index, and an index of nothing is nothing. Every hour spent on it is an hour not spent on the only thing that has ever reliably worked, which is having pages worth citing.
If this changes, and it might, it will change with an announcement from a named provider. Until then, treat anyone selling llms.txt as an AI visibility strategy with the scepticism you would apply to anyone selling meta keywords in 2009.
Common questions
Where does the llms.txt file go?
At the root of the domain, alongside robots.txt, so it resolves at yoursite.com/llms.txt. The proposal also allows one on a subpath for a specific section. It is a plain markdown file, so no build step or plugin is required.
Will adding llms.txt hurt my site?
No. It is an ordinary text file that nothing currently reads, so the downside is the twenty minutes you spend on it and the risk of it going stale. A file listing pages that no longer exist is worse than no file, so if you add one, add it to whatever checklist you use when you change your site structure.
Is my host blocking AI crawlers without telling me?
It happens, and it is worth checking rather than assuming. Some hosting and CDN providers ship bot protection that blocks AI user agents by default, and some site templates carry a copied robots file with those rules already in it. Fetch your own robots.txt and read it, then check your access logs for which crawlers actually reach you.
Related reading
For what actually determines whether assistants quote you, see What is AEO and does my small team need it?. To find out where you currently stand, see Does ChatGPT mention my business?.
Sources
Give your company a marketing team this week.
Munshi plans, writes, publishes, and tracks your blog and socials.
Start your workspace