It is a plain markdown file at the root of your site (yoursite.com/llms.txt) that gives large language models a curated map of your most citable content. Think of it as a table of contents written for an AI reader: a short description of the site, then grouped links with one-line summaries. It is not robots.txt, which controls crawler access. llms.txt does the opposite job, it invites and guides.

Does llms.txt actually affect whether I get cited?

It is a proposed standard, not a ranking factor Google has confirmed, so treat it as an eligibility and efficiency signal rather than a guaranteed boost. Providing a clean index lowers the compute cost for a model to find and quote your content, and it points the model at the pages you want quoted instead of leaving it to guess. On a large programmatic site, where the citable pages are a small fraction of the total, that guidance is the whole point.

Why does it matter more for programmatic sites?

A programmatic site can publish tens of thousands of templated pages. An answer engine has no cheap way to tell which of those pages carry the citable facts and which are near-identical filler. llms.txt is where you say it explicitly: here are the 40 pages worth quoting, grouped and described. Without it, the model either skips the site or cites a page you did not intend.

Is llms.txt the same as blocking or allowing AI crawlers?

No, and the two are frequently in contradiction. In the same 20-site benchmark, 25% of sites blocked at least one AI crawler in robots.txt. Shipping an llms.txt while your robots.txt disallows GPTBot, ClaudeBot, or PerplexityBot cancels the effort: the model that reads your index cannot fetch the pages it points to. Fix crawler access first, then publish the index.

What is the difference between llms.txt and llms-full.txt?

llms.txt is the concise index of links and summaries. llms-full.txt is an optional companion that inlines the full text of those pages into one file, so a model can ingest your key content without following every link. Ship llms.txt first; add llms-full.txt only if your citable content is stable enough that a flattened snapshot will not go stale quickly.

Guide · updated July 18, 2026

llms.txt for Programmatic Sites: The AEO Index 80% of Sites Skip

In a July 2026 pseolint crawl of 20 production programmatic-SEO sites, four in five shipped no llms.txt at all. It is the single most common answer-engine gap in the wild, and also the cheapest to close. Here is what the file is, what belongs in it, and how to verify yours.

What llms.txt is (and is not)

llms.txt is a proposed standard: a plain markdown file served at yoursite.com/llms.txt that describes your site and links to the content most worth reading, each link annotated with a one-line summary. It is written for a machine reader, so it is concise and structured rather than designed to look good in a browser.

It is often confused with robots.txt. They are opposites. robots.txt tells crawlers what they may not touch. llms.txt tells an AI reader what is worth reading and why. One gates access; the other guides attention. You need both pointing the same direction, which many sites fail to do.

Why programmatic sites need it most

A hand-built site with 30 pages does not really need a map; a model can read the whole thing. A programmatic site with 30,000 templated pages is a different problem. The facts worth quoting live on a small fraction of those pages, and an answer engine has no cheap way to find them. It will either give up on the site or quote a page you never meant to surface.

llms.txt is where you resolve that ambiguity by hand. You name the citable pages, group them, and describe them, so the model spends its limited attention on your best content instead of your template chrome. Every other AEO gap in our 20-site benchmark (weak citation coverage on 75% of sites, blocked crawlers on 25%) compounds when the model cannot even locate your good pages first.

What goes in a good llms.txt

The format is deliberately simple. Four parts, in order.

#1
An H1 title and a blockquote summary
The first line is # Your Site Name. Follow it with a > blockquote of one or two sentences saying what the site is and who it serves. This is the context a model reads before anything else.
#2
Grouped links with one-line descriptions
Use ## headings to group your citable pages (for example: ## Guides, ## Data, ## Reference). Under each, a markdown list of links, and after each link a short description of what a reader gets there. Descriptions are not optional filler; they are how the model decides what to quote.
#3
Only pages worth citing
This is the discipline programmatic sites miss. Do not dump your sitemap. Include the pages that carry a verifiable, quotable fact, and leave the templated long tail out. A short, high-signal index beats an exhaustive one.
#4
An optional ## Optional section
A convention for links a model can skip if it is short on context: secondary references, deep-dive pages. It tells the model what to drop first, which is useful when your citable set is large.

A minimal example:

# Acme Integrations

> Acme connects 400+ SaaS tools. This index lists our
> most citable integration guides and data pages.

## Guides
- [Slack + Acme setup](/guides/slack): step-by-step, with rate limits and scopes.
- [Webhook reference](/guides/webhooks): payload schemas and retry behavior.

## Data
- [Integration uptime report](/data/uptime): monthly, per-connector, sourced.

## Optional
- [Changelog](/changelog): release history for context.

The mistake that cancels the whole thing

The most common failure is not a malformed file. It is shipping a clean llms.txt while robots.txt blocks the very crawlers meant to read it. In the benchmark, a quarter of sites disallowed at least one answer-engine crawler. If your robots.txt carries a blanket Disallow: / for GPTBot, ClaudeBot, PerplexityBot, or Bytespider, the model that reads your index cannot fetch the pages it points to, and the index is decoration.

Fix the access first. Confirm the crawlers you want to cite you are allowed, then publish the index that tells them where to look. Order matters.

Verify it in one command

Both halves of this, the index and the access, are checked in a single audit.

npx pseolint https://your-site.com

aeo/llms-txt reports whether the file exists and is structured, and aeo/crawler-access flags any AI crawler you are blocking in robots.txt. Green on both means your citable pages are reachable and mapped, which is the whole job.

Frequently asked questions

What is llms.txt?: It is a plain markdown file at the root of your site (yoursite.com/llms.txt) that gives large language models a curated map of your most citable content. Think of it as a table of contents written for an AI reader: a short description of the site, then grouped links with one-line summaries. It is not robots.txt, which controls crawler access. llms.txt does the opposite job, it invites and guides.
Does llms.txt actually affect whether I get cited?: It is a proposed standard, not a ranking factor Google has confirmed, so treat it as an eligibility and efficiency signal rather than a guaranteed boost. Providing a clean index lowers the compute cost for a model to find and quote your content, and it points the model at the pages you want quoted instead of leaving it to guess. On a large programmatic site, where the citable pages are a small fraction of the total, that guidance is the whole point.
Why does it matter more for programmatic sites?: A programmatic site can publish tens of thousands of templated pages. An answer engine has no cheap way to tell which of those pages carry the citable facts and which are near-identical filler. llms.txt is where you say it explicitly: here are the 40 pages worth quoting, grouped and described. Without it, the model either skips the site or cites a page you did not intend.
Is llms.txt the same as blocking or allowing AI crawlers?: No, and the two are frequently in contradiction. In the same 20-site benchmark, 25% of sites blocked at least one AI crawler in robots.txt. Shipping an llms.txt while your robots.txt disallows GPTBot, ClaudeBot, or PerplexityBot cancels the effort: the model that reads your index cannot fetch the pages it points to. Fix crawler access first, then publish the index.
What is the difference between llms.txt and llms-full.txt?: llms.txt is the concise index of links and summaries. llms-full.txt is an optional companion that inlines the full text of those pages into one file, so a model can ingest your key content without following every link. Ship llms.txt first; add llms-full.txt only if your citable content is stable enough that a flattened snapshot will not go stale quickly.
How do I check my llms.txt is valid and complete?: Run npx pseolint https://your-site.com. The aeo/llms-txt rule checks for the file's presence and basic structure, and aeo/crawler-access flags any AI crawler you are blocking in robots.txt, so you catch the contradiction in the same pass. If both are clean, your citable pages are both reachable and mapped.

Cite this guide

Published under CC BY 4.0. Quote and reuse with attribution to the canonical URL.

Kam, P. (2026). llms.txt for programmatic sites: the AEO index 80% of sites skip. Ouranos Labs. https://pseolint.dev/research/llms-txt-programmatic-seo

Check whether your site is mapped for answer engines.

Run a free audit

Sources

llmstxt.org — the /llms.txt proposal — The llms.txt proposal defines the file format and intent this guide operationalizes for programmatic sites.
Google Search Central — AI features and your website — Google's AI-features guidance frames why reachable, well-structured content governs eligibility for citation in AI surfaces.
Google Search Central — Introduction to robots.txt — The robots.txt reference underpins the crawler-access fix that must precede publishing an llms.txt.