What belongs in an llms.txt
A practical account of what to put in an llms.txt, what to leave out, and what it does and does not accomplish. Including the one on this site.
The short answer
An llms.txt is a plain markdown file at the root of a site that tells a language model what the site is and where the authoritative pages are. It exists because a model arriving at your domain otherwise has to reconstruct your information architecture from navigation markup, and it will do that badly.
It is a convention rather than a standard with an enforcement body, so nothing breaks if you do not have one, and no crawler is obliged to read it. The reason to write one anyway is that it costs an hour and it is the cheapest way to state plainly what you would otherwise hope a crawler infers.
The structure that works
Start with the name of the thing and a blockquote summary. One paragraph, concrete, no marketing. If a model reads only this, it should be able to say correctly what you do.
Then a short list of what you actually offer or provide, in plain sentences rather than nested marketing hierarchy. Then the pages that matter, as markdown links, so a crawler that wants depth knows exactly where to go. Then contact.
Keep it flat. The value of the file is that it is unambiguous, and nesting reintroduces the ambiguity you wrote it to remove.
What to leave out
Anything you would not want quoted back at you verbatim, because that is what it is for. Aspirational claims, unreleased work, and numbers you cannot link to all belong somewhere else.
Also leave out duplication. If the file and your documentation disagree, you have created exactly the contradiction that produces hallucinations. The file should summarise and point, never restate detail that lives elsewhere.
What it does not do
It does not make your documentation agent-readable. A clean summary pointing at pages that contradict each other helps nobody. The structural work, the typed interfaces and the deduplication and the tested examples, is separate and larger.
It also does not control training. Crawler behaviour is governed by robots.txt and by each operator's own policy. If you want to state a position on AI crawlers, robots.txt is where that goes, and it is worth being explicit either way rather than relying on defaults that may change.
The one on this site
This site serves an llms.txt at the root. It carries the positioning in one blockquote, the eight things the studio builds as plain sentences, the terms including the work we turn down, one linked proof point, the page list, and contact. It is about forty lines.
Reading it is a faster way to understand the format than any description of it, including this one.
Common questions
- What is llms.txt?
- A plain markdown file at the root of a site that tells language models what the site is and where its authoritative pages are. It is a convention, not an enforced standard, and it exists so a model does not have to infer your information architecture from navigation markup.
- Where do I put llms.txt?
- At the root of the domain, so it resolves at yourdomain.com/llms.txt. On a static site that means the public directory. Keep it plain markdown and keep it flat.
- Does llms.txt control whether my site is used for training?
- No. Crawler access is governed by robots.txt and by each operator's policy. An llms.txt describes your site for models that do read it. If you want to state a position on AI crawlers, do it explicitly in robots.txt rather than relying on defaults.
- Is llms.txt enough to make my docs work for AI agents?
- No. It is one useful piece. The structural work is typed interfaces over the real API, one canonical statement per fact, and examples tested in CI. A summary pointing at contradictory documentation does not fix the contradiction.
Working on something like this?
This is the layer we work in. If you have a hard problem underneath a product, tell us the hard part.