SEO··9 min read

What Is llms.txt? Format, Real Examples and Does It Matter

llms.txt is a plain Markdown file at the root of a website, at example.com/llms.txt, that gives large language models a short description of the site and a curated list of links to its most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024 as a way to help AI tools find the right content without wading through navigation menus, cookie banners and scripts.

The idea spread quickly through developer documentation sites, and then to SEO tools and marketing platforms. To see what it looks like in practice, we fetched /llms.txt from 24 well-known sites on 10 November 2026 and checked each file's structure. The results are below, followed by an honest look at whether it affects AI search.

The llms.txt format in one minute

# Site or project name

> One- or two-sentence summary of what the site is.

Optional details: facts a model should know, in paragraphs or lists.

## Docs

- [Quickstart](https://example.com/docs/quickstart/): what it covers
- [API reference](https://example.com/docs/api/): what it covers

## Optional

- [Changelog](https://example.com/changelog/)
  • An H1 with the site or project name. It's the only required element.
  • A blockquote summary right after it.
  • Optional details with no headings.
  • H2 sections, each a Markdown list of links with short notes.
  • An “Optional” section for links a tool can skip when its context window is short.

The proposal also suggests offering a clean Markdown copy of each page at the same URL plus .md. A companion convention, llms-full.txt, puts the full text of the documentation in one file. You can build and check a file with our llms.txt Generator & Validator.

Who publishes an llms.txt? We checked 24 sites

14 of the 24 sites had one. They were Anthropic's developer docs, Stripe Docs, Cloudflare Developers, Vercel, Next.js, Supabase, Zapier, Perplexity's docs, Svelte, GitHub Docs, WordPress.org, Yoast, Semrush and HubSpot.

Ten had none: Google, OpenAI's main site and its platform docs, X, The New York Times, Ahrefs, Moz, MDN, Hugging Face and Shopify's developer site. OpenAI's main domain refused our automated request outright.

The split is telling. Developer platforms and SEO or marketing tools publish one. The companies that run the biggest AI products and search engines mostly don't, and neither do the large publishers in our sample.

What the 14 real files look like

SiteSizeSectionsLinksSummaryOptional section
docs.stripe.com96.7 KB27478NoYes
Anthropic developer docs96.1 KB5890NoNo
docs.perplexity.ai42.1 KB27YesYes
hubspot.com34.7 KB29225YesNo
docs.github.com28.0 KB14117YesNo
developers.cloudflare.com16.4 KB9112YesNo
semrush.com14.6 KB942YesYes
nextjs.org14.2 KB554YesNo
zapier.com13.6 KB894YesNo
yoast.com7.5 KB862YesNo
wordpress.org5.6 KB936YesNo
vercel.com5.3 KB928YesYes
supabase.com3.2 KB442NoNo
svelte.dev1.6 KB37YesNo

What stands out:

  • Sizes vary 60-fold. The median file is 14.6 KB with 62 links, but Svelte's is 1.6 KB while Stripe's and Anthropic's are close to 100 KB. Large documentation sites treat llms.txt as a complete table of contents; product sites treat it as a short brief.
  • Every file had the required H1, and 11 of 14 included the blockquote summary. Stripe and Anthropic open with a paragraph instead.
  • Only 4 used an Optional section, even though it's the format's way of telling tools what to skip when space is tight.
  • 9 of 14 mix plain facts into their sections, such as licensing, market share or “check the npm registry for the latest version”, instead of only links. It's a departure from the strict format, but it shows what companies actually want models to know.
  • 7 of 14 link to .md versions of their pages, following the proposal's suggestion. That's how the larger documentation sites make their content readable without HTML.
  • Some files speak directly to AI agents. Stripe's tells models to check the current package version rather than rely on training data. Zapier calls its file a “root map for AI agents”. These instructions are aimed at coding assistants working on a user's behalf, not at search crawlers.

Does llms.txt help you appear in AI answers?

There's no evidence that it does, yet. Google has said it doesn't use llms.txt, and its Search Advocate John Mueller has compared it to the keywords meta tag. None of the major AI search products, including ChatGPT search, Perplexity, Gemini and Google's AI Overviews, has documented fetching llms.txt when it crawls or ranks sources. Notably, Google and OpenAI don't publish one on their main sites.

Where llms.txt is used today is on demand: a developer points a coding assistant at docs.example.com/llms.txt, a documentation MCP server loads it to know what's available, or an agent fetches it while completing a task. In those cases a good file saves the tool from crawling dozens of pages, and that is why developer platforms adopted it first.

You can check whether anything requests yours: search your server or CDN access logs for /llms.txt and look at the user agents. On most small sites, the answer is very little traffic.

Should you create one?

  • Yes, if you have documentation, an API or a developer product. It costs an hour, and the people most likely to point an AI tool at your site will benefit.
  • Probably, if you're a SaaS or tool company whose details models often get wrong: pricing basics, product names, what you don't do. The details section is a cheap place to state them.
  • Low priority for a blog or local business. Clear pages, accurate structured data and crawlable HTML do far more for how search engines and AI systems understand you. The Schema Markup Generator and Meta Tag Checker are better uses of the time.

What llms.txt is not

  • Not access control. It doesn't allow or block anything. To keep AI crawlers away from your content, add rules for their user agents, such as GPTBot, ClaudeBot or Google-Extended, to robots.txt and test them with the Robots.txt Tester.
  • Not a sitemap. Search engines rely on sitemap.xml to discover URLs. llms.txt is a curated reading list, not an inventory.
  • Not a ranking factor in any search engine that has said so publicly.

How to write a good llms.txt, based on the examples

  1. Start with the H1 and a one-sentence blockquote that says what you are and who it's for.
  2. Use the details section for facts that are often wrong or out of date in AI answers.
  3. Group links into a handful of H2 sections that match how people use your site: Docs, Guides, Pricing, Policies.
  4. Give every link a short note. A link title alone forces a tool to fetch the page to learn what it is.
  5. Put secondary material under ## Optional.
  6. Use absolute URLs, serve the file as plain text at the root, and regenerate it when your structure changes.

The llms.txt Generator can import your sitemap as a starting point and checks the result against the format.

How we checked

On 10 November 2026 we requested https://<domain>/llms.txt for each site, following redirects as a crawler would; Anthropic's docs, for example, redirect to platform.claude.com. Files returning 200 with text content were parsed with the same validator used in our generator. “Links” counts list items containing a Markdown link; “summary” means a blockquote before the first section. Files change often, so treat the figures as a snapshot.

The Tech Scouts

Writer at TheTechScout.