The short version
- The llms.txt proposal moved to version 2 on 10 August 2026. It adds three things: standard links so an agent can find the file and the Markdown version of a page, Markdown versions of the pages themselves, and files per subdirectory.
- It is still a proposal, not a standard. The Chrome audit that checks it only flags a server error; a missing file is marked not applicable, because providing it remains optional.
- We applied it on dotsland.com, and we publish here what we measured: our articles and expertise pages now exist in Markdown, three to five times lighter than their HTML page. We also publish what our server logs show: the llms.txt index is still almost ignored, but our Markdown versions are read by OpenAI’s and Anthropic’s crawlers, on top of the HTML pages (measurement updated on 25 October 2026).
- Google does nothing with it. Its guide to generative AI features says so: its search uses neither llms.txt nor Markdown versions. The file is for other agents.
- The most misunderstood rule is the subdirectory one: a file only covers the pages under its path. Copying another site’s structure often means publishing files that cover nothing.
Run the test. Type your site’s address followed by /llms.txt. Then take one of your articles and add index.md to the end of its address. On our site, the first returns an index of 72 links, and the second returns the text of the article, with no menu, no footer, no script. If yours returns a 404 twice, this article tells you what version 2 changes, and what it does not.
What llms.txt is, in three ideas
A Markdown index, for agents. The proposal, published by Jeremy Howard in September 2024, consists of placing at the root of a site a /llms.txt file that presents the site in a few lines and points to its useful content. The format is simple and fixed: a title, a summary, sections, and links with a short description. The starting idea has not changed: a web page is made for a human, wrapped in navigation and code, and an agent wastes time and context cleaning it up.
In practice, an llms.txt file looks like this:
# Example Workshop
> Solid wood furniture. Buying and care guides.
## Guides
- [Choosing a wood species](https://example.com/guides/species/index.md): oak, walnut, ash, what changes in daily use
- [Caring for oiled furniture](https://example.com/guides/care/index.md): frequency, products, common mistakesA title, a one-line summary, then links grouped by section, each with a few words saying what it contains. The agent reads these few lines and picks the page to open.
It has spread through technical documentation. Version 2 opens with that observation, which is its author’s: thousands of sites publish a file, documentation platforms generate one automatically, and the major AI labs publish one for their own developer documentation. We cannot verify the number of sites. We checked that last point: OpenAI, Anthropic and Google each serve one for their documentation.
Version 2 starts from real usage. Where the first version, in 2024, predicted that agents would read websites, the second describes what they do: look at the index, then follow the useful links. Hence its new features, all aimed at making that path more direct.
Where it comes from, in four dates
- 3 September 2024. Jeremy Howard publishes the proposal on llmstxt.org.
- The following two years. According to version 2, the file settles into technical documentation: documentation platforms generate it automatically, and the major AI labs publish one for their developer documentation.
- 10 August 2026. Version 2 is released, rewritten from those two years of use.
- 22 August 2026. We publish our citability audit, where we write that no public evidence shows the file gets a site cited. That is still true.
- 24 September 2026. We apply version 2 on dotsland.com, a month and a half after its release.
Two years between the proposal and its first overhaul, then a few weeks for a site like ours to fall behind. A file set up once and forgotten ages quickly: the question is not only whether you have one, but which version it follows.
What version 2 changes
| Question | Version 1 (2024) | Version 2 (10 August 2026) |
|---|---|---|
| How does an agent find the file or the Markdown version of a page? | It applies an address rule, but nothing in the page signals the Markdown version | Through standard link relations: describedby to the llms.txt that covers the page, alternate with type text/markdown to its Markdown version. As a tag in the page, or as an HTTP header |
| Where is the Markdown version? | page.html.md, and index.html.md for an address with no file name |
A second accepted form, page.md, and index.md alongside index.html.md |
| What does an llms.txt in a subdirectory mean? | Allowed, with no rule | It covers the pages under its path, and the most specific file applies |
| And the “Optional” section? | Told tools what they could leave out | Still allowed, but with no mechanical effect: the context-expansion tool is no longer part of the proposal |
Why a Markdown version of each page
It is the most concrete new feature, and the least explained. An agent does not read a page the way you do: it fetches its entire code, menu, footer, scripts and consent banner included, then has to extract the text. The proposal puts it this way: an HTML page “wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise”. The Markdown version is the text alone, with its structure: headings, lists, tables, links.
- Less noise. The agent reads what you wrote, not your menu, and is less likely to mix your point with the rest of the page.
- Less weight. An agent works within a limited reading window, and every useless element costs it time and money. On our site, a page weighs three to five times less in Markdown than in HTML.
- More pages read for the same effort. An agent that consults several pages to answer a question reads more of them when they are light.
The limit is the same as for the file: it helps an agent already visiting you, it does not make it come.
The blind spot: what the file does not do
Version 2 states that Chrome’s audit tool, Lighthouse, checks for an llms.txt as part of its agentic browsing audits. That is accurate, and it is often quoted as proof that the file has become essential. Lighthouse’s own documentation says something else when you read it in full: the audit only flags a server error when retrieving the file. If the file does not exist, the audit is marked not applicable, “as providing the file is optional at the moment”. Lighthouse itself describes it as an emerging convention.
In other words, nothing has changed since what we wrote in August about being cited by AI: no public evidence shows that the file triggers citations. What version 2 improves is the experience of an agent that is already looking for something on your site. Not the likelihood that it comes.
We measured it on our own site. From 31 August to 24 September 2026, our server logs show four reads of our llms.txt by an AI crawler whose IP address we verified (the name a crawler gives itself can be faked, its address much less so: 130 fake GPTBots came through our site in six weeks): three by OpenAI’s search crawler, one by GPTBot. None by a crawler presenting itself as Anthropic’s or Perplexity’s. Most other reads came from our own checking tools. As for our Markdown versions, put online on 24 September, no agent had read them yet that day. We will measure again in a month.
Update of 25 October 2026 We measured again, as announced, from 30 September to 25 October, counting only crawlers whose IP address appears in the lists published by OpenAI and Anthropic. Our llms.txt files were read only four times, twice by GPTBot and twice by OpenAI’s search crawler, and still never by Anthropic’s. Our Markdown versions, on the other hand, are being read: 72 pages by ClaudeBot, 46 by GPTBot, 22 by OpenAI’s search crawler. These crawlers find them through the link placed in each page, not through the index: ClaudeBot never opened our llms.txt. And they read them on top of the HTML page, not instead of it: GPTBot and OpenAI’s search crawler read each of these pages in both formats. Reading is not citing: nothing yet shows whether these reads lead to citations.
And Google? Its guide to generative AI features, updated on 10 July 2026, is blunt: its search uses neither llms.txt, nor AI text files, nor Markdown versions. Such a file, it writes, will neither harm nor help your presence in Google Search. For AI Overviews and AI Mode, the lever is still search engine optimisation.
The mistake we can see coming Copying another site’s structure. The subdirectory rule says a file covers the pages under its path. A site whose articles live under /2026/ benefits from a /2026/llms.txt. On a site whose articles sit at the root of the address, like ours, a /blog/llms.txt would cover no article at all. Before creating a file per directory, look at where your addresses actually live.
The method, and what we did on our own site
We applied version 2 on dotsland.com in September 2026. The figures below were taken on 24 September, on the live site.
- Keep the index short. Our French
/llms.txtweighs 28 KB for 72 links, the English one 21 KB. That is the order of magnitude version 2 recommends: small enough to fit in an agent’s context, with the detail living behind the links. 69 of our 72 links now point to the Markdown version of the page rather than its HTML. - Publish the Markdown version of the pages that matter. Our articles and expertise pages respond at
index.md, astext/markdown. The article on product order in categories weighs 87 KB in HTML and 24 KB in Markdown, 3.6 times less; the e-commerce expertise page goes from 64 KB to 12 KB, 5.1 times less. These versions are served with anoindexheader, which asks search engines not to index them alongside the HTML page. - Set both link relations, as an HTTP header and in the page. Each article declares its Markdown version and the llms.txt that covers it. The most-specific-file rule is respected: an expertise page points to
/expertise/llms.txt, an English article to/en/llms.txt. - Create subdirectory files only where addresses have subdirectories. On our site, only the expertise pages sit under a path,
/expertise/and/en/expertise/: those are the only two subdirectory files we created.
A five-minute self-diagnosis: open /llms.txt on your site, count the links and check that none returns an error. Then look at the address of three of your most important pages: do they sit under a shared directory, or at the root? The answer tells you whether subdirectory files make sense for you. Finally, if you have access to your server logs, look at who reads your llms.txt: it is the only way to know whether it is used, and by whom.
The options, and what they are worth
| Approach | What you get | Who it suits |
|---|---|---|
| Publish nothing | No negative signal in Chrome’s audit, which marks the absence as not applicable. An agent visiting you has to clean up your HTML itself | Sites whose content agents do not consult |
| An llms.txt index alone, version 1 style | Little work, a clean entry point. The links point to heavy HTML pages | Any site that wants a minimum without a technical project |
| Full version 2 | Index, Markdown versions, link relations, useful subdirectories. Needs development or an extension that generates the Markdown on every publication | Sites rich in reference content, documentation, product pages, guides |
| One big file with all the content | Handy to produce, but against the direction of version 2, which does not mention this format | To avoid as the centrepiece |
Tooling
If you are looking for an extension or a development for your CMS, four criteria are enough: it generates the Markdown version from published content, with no manual entry; it sets the link relations in the HTTP header and in the page; it points to the most specific file for each address; and it keeps the index up to date on every publication, without anyone having to think about it.
Method comes before the tool: a generator that creates one file per directory without looking at where your addresses live produces exactly the meaningless files the subdirectory rule helps you avoid.
The bigger picture
Version 2 records a real change: agents read websites every day, and a site can make their job easier or harder. But it remains a proposal, and the Chrome tool that checks it treats it as optional. Both are true at once, which is why the subject is so poorly told: it is presented either as a gimmick or as an obligation, when it is a technical courtesy that has become well equipped.
Our reading, in our artificial intelligence expertise: do it, because it costs little and it genuinely serves the agents already reading you. Do not wait on it to be cited: citation happens elsewhere, in what others write about you.
Sources
The verifiable claims in this article link to their primary source, consulted on 24 September 2026. The measurements on dotsland.com were taken the same day. We do not cite a source we have not read.
- Jeremy Howard, “The /llms.txt file, v2”, llmstxt.org, published 3 September 2024, modified 10 August 2026. View
- llmstxt.org, “Changes”, details of the version 2 changes. View
- Chrome for Developers, Lighthouse, “llms.txt” audit in the agentic browsing audits. View
- llms.txt files of the developer documentation of OpenAI, Anthropic and Google. OpenAI, Anthropic, Google
- Google Search Central, “Google’s Guide to Optimizing for Generative AI Features on Google Search”, updated on 10 July 2026, opened on 25 October 2026. Read
FAQ
Do I also need an llms-full.txt file?
Version 2 does not mention it. It recommends the opposite: a short index, with the detailed content in Markdown versions of the pages that the agent follows as needed. One big file can exist as a complement, but it does not replace that structure.
Do the Markdown versions create duplicate content?
That is the risk to handle. We serve ours with a noindex header, which asks search engines not to index them, and each HTML page remains the reference version. Do not publish Markdown versions without that precaution.
Does it help my site get cited by AI?
No public evidence shows it, and Chrome’s audit treats the file as optional. What it improves is how an agent already visiting your site reads it. Citation depends mostly on what other sources say about you, as we explained in our citability audit.
Does Google read llms.txt?
No, not for its search. Its guide to generative AI features states that it uses neither llms.txt nor Markdown versions, and that such a file will neither harm nor help. To appear in AI Overviews and AI Mode, the lever is still search engine optimisation. The file is for other agents: on our site, OpenAI’s and Anthropic’s crawlers read our Markdown versions.
Can Dotsland help?
Yes. A diagnosis of what your site serves agents today, a choice between the index alone and the full version depending on your content, then implementation and checks. It is part of our artificial intelligence expertise. Let’s talk, or start by typing /llms.txt after your site’s address.