What Is LLMs.txt? A Complete Guide to AI Search, LLM Retrieval, and Best Practices

Published August 6, 2026.

What LLMs.txt Is

llms.txt is a plain text markdown file placed at a website’s root, such as yoursite.com/llms.txt. It gives AI systems and AI agents a short, curated map of a site’s most important content.

The format was proposed by Jeremy Howard, co founder of Answer.AI and fast.ai, on September 3, 2024, and published at llmstxt.org. It is not an official web standard.

The original proposal also included llms-full.txt, an optional companion file with the full content of key pages in one markdown document.

The Purpose in AI Search and LLM Retrieval

llms.txt is an orientation layer, not a retrieval system. It does not chunk content, generate embeddings, or rank results.

Its job is to help an agent find the right pages faster, before retrieval happens.

The article says mainstream AI search crawlers like GPTBot, ClaudeBot, and PerplexityBot mostly do not request llms.txt in meaningful volume today. Its clearest current use is in coding agents, IDE integrations, and Model Context Protocol tools.

Editorial Priority Matters

The file depends on human judgment. Every link, section order, and description is a deliberate editorial choice.

The article emphasizes that order matters because agents may stop reading before the end of the file.

The spec includes an optional final section called Optional for lower priority material.

robots.txt vs. sitemap.xml vs. llms.txt

File	Main question	Role
robots.txt	May you look?	Controls access with allow and disallow rules.
sitemap.xml	What exists?	Provides a comprehensive list of indexable URLs.
llms.txt	What actually matters?	Highlights the most important pages in a curated order.

How an llms.txt File Is Structured

One H1 heading naming the site, brand, or project.
A blockquote with one to a few sentences describing the site.
Optional extra context paragraphs.
H2 sections grouping links by topic or function.
Each link uses a title, absolute URL, and one line description.
An optional H2 section named Optional for lower priority content.

The file should be plain text, named exactly llms.txt, and placed at the site root. Links should use full absolute URLs.

How to Create an Effective llms.txt File

Choose the pages that define the site. The article suggests 20 to 50 pages for most sites.
Group them into 3 to 6 clear sections.
Write the blockquote first.
Write one specific, honest description per link.
Order pages by real priority.
Save and publish the file at the root.
Confirm that yoursite.com/llms.txt returns the raw file.
Update it as the site changes.

Example and Best Practices

The article includes a sample file for a fictional company called Nimbus Metrics. It shows sections for Product, Documentation, Pricing, and Optional.

Keep the file lean and human curated.
Use a specific blockquote.
Keep link formatting consistent.
Use Optional only for secondary material.
Review the file regularly.
Do not treat it as a substitute for strong site structure, semantic markup, or schema.

Frequently Asked Questions

What is llms.txt used for?

It gives AI systems and AI agents a short, curated map of a site’s most valuable pages.

Do ChatGPT, Gemini, Claude, and Perplexity actually read llms.txt?

The article says not in any meaningful volume based on current evidence.

Is llms.txt the same as robots.txt or sitemap.xml?

No. robots.txt controls access, sitemap.xml lists URLs, and llms.txt highlights the pages that matter most.

Where should the file be placed?

At the site root as llms.txt.

Does adding it improve AI search rankings or citations?

The article says there is no published evidence that it does.