llms.txt

llms.txt is a proposed convention for a Markdown file at the root of a domain that provides a clean, AI-readable index of a site’s most important content. It was first proposed by Jeremy Howard in September 2024 and has since been adopted by a growing list of documentation sites, SaaS platforms, and content publishers. Howard published a second version of the spec in August 2026, the first revision since launch, covered under what changed in v2 below.

What is Google’s position on llms.txt?

Google Search does not use llms.txt. Its May 2026 AI optimisation guide states the position directly, and names generative AI specifically: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.”¹ Because Search ignores the file, maintaining one carries no benefit and no risk. Google’s own wording is that publishing one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them”, a line added to the guide in June 2026.² Google may discover, crawl, and index llms.txt like any other URL, but that is not special treatment. If your site already earns search visibility through quality content, clear structure, and credible authorship, you have everything needed for AI visibility too.

The reason for this position lies in the nature of the format: unregulated and unverifiable. As Search Advocate John Mueller put it in May 2026, a self-reported file cannot help an AI system choose between sites: every llms.txt makes the same promotional claim, in effect “here are all the pages everyone must visit”, so it cannot work as a comparative signal for discovery.³ It is the meta keywords tag again, a self-declared field Google abandoned decades ago for the same reason. Mueller added a second flaw: because the file is self-reported, its contents need not match the actual HTML, leaving an AI system no way to verify what it claims.⁴ The most he allowed was that the file might help once a system has already arrived on a site, which is the agent-navigation case below, not a search one.

Two things appear to contradict this. Google itself publishes llms.txt and Markdown pages on developers.google.com: Mueller explained these exist for AI coding tools that parse reference material more accurately from clean Markdown, “a temporary crutch, perhaps to save some tokens”, not a search signal.⁵ And Chrome’s Lighthouse does check for llms.txt, via an audit in its Agentic Browsing category.⁶ Neither reverses Search’s position: they are different parts of Google answering a different question. Lighthouse asks whether a browser-based agent can navigate your site efficiently, not whether the file affects ranking. The audit is covered in full under adoption status below.

What is llms.txt for?

The problem llms.txt addresses is straightforward. Large language models reading a website at inference time face the same challenges a human would: navigation menus, footer noise, pop-ups, advertising, and HTML scaffolding all dilute the signal. Sites are built primarily for browser rendering, not for clean text extraction.

A useful frame: llms.txt addresses functionality, not discovery. Discovery is how pages get found via search. Functionality is what happens once a visitor or agent has already arrived. The two require different tools and have different success metrics. llms.txt belongs in the functionality column, which is why publishing it has no bearing on search visibility.

llms.txt provides a curated, Markdown-formatted entry point that says: here is the canonical structure of this site, here are the most important pages, here is what they contain. It is a hint to AI systems about what matters, in a format they can ingest cleanly.

How does llms.txt differ from robots.txt and sitemap.xml?

File	Purpose	Audience
robots.txt	What can be crawled	All crawlers
sitemap.xml	What URLs exist	Search engine crawlers
llms.txt	What matters and what it means	Browser-based AI agents navigating a site
ai-catalog.json	What an agent can do on the site	Agents and registries discovering callable capabilities

For AI agents that need to act on a site rather than navigate it (completing a checkout, submitting a form, making a booking), WebMCP is the relevant protocol. llms.txt addresses navigation; WebMCP addresses action. The ai-catalog.json file, defined by the draft Agentic Resource Discovery spec, is a third, separate thing again: it declares which callable capabilities a site exposes so agents can discover them. Do not conflate the three. llms.txt is a content index for any site; ai-catalog.json is a capability manifest relevant only to sites that expose agent-callable tools.

The three are complementary, not competing. robots.txt and sitemap.xml are baseline requirements for any site; llms.txt is optional, worth adding only where the agent-navigation or documentation use cases below apply.

llms.txt syntax

The format is intentionally simple Markdown. A minimal example:

# Site Name

> One-line description of what the site is about and who maintains it.

Optional longer paragraph providing context about the site, its purpose, and any guidance for how AI systems should use the content (attribution, citation preferences, etc.).

## Section heading

- [Page title](https://example.com/page/): Short description of what this page contains.
- [Another page](https://example.com/another/): Description.

## Another section

- [Page](https://example.com/page2/): Description.

Headings group related content. Bullet links describe individual pages. Descriptions are optional but recommended; they give the AI system a hint about what each linked page covers.

What changed in llms.txt v2?

Jeremy Howard published version 2 of the spec on 10 August 2026, the first revision since the original proposal.⁷ The file format above is unchanged, so an existing llms.txt stays valid. What v2 adds is a way for an agent to find the Markdown behind a page instead of guessing at it, which was the most common request across two years of adoption.

Link relations. A page can point at its own Markdown version with rel="alternate" type="text/markdown", and at the llms.txt file that describes it with rel="describedby". Both work as HTML <link> elements in the head, or as an HTTP Link: response header:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

The header form is the more flexible of the two. It covers resources that are not HTML, including the Markdown files themselves, and it can be set in server or CDN configuration without touching page templates.

Two URL patterns for Markdown pages. v1 allowed only appending .md to the full page URL, so /docs/tutorial.html became /docs/tutorial.html.md. Some publishing tools replace the extension instead, giving /docs/tutorial.md. v2 permits both, on the reasoning that practice had already diverged.

Subpath scoping is defined. v1 permitted a file at a subpath without saying what it covered. v2 states it: a file covers the pages under its path, and where more than one applies, the most specific wins, so /docs/llms.txt covers everything under /docs/. This is also what lets a site that controls only a path, such as a GitHub Pages project site, use the convention properly.

Two things were removed. The llms_txt2ctx context-expansion tool is no longer part of the proposal, and with it goes the mechanical meaning of the Optional section, which existed to tell that tool what to skip. Optional sections remain valid as a convention for secondary links, but no longer instruct anything.⁸

None of this changes Google’s position above, and the Lighthouse audit described under adoption status tests only that the file exists and parses, not the new link relations or Markdown-page discovery. v2 is still open for feedback on GitHub and Howard notes the syntax may change before it settles, so treat anything you implement against the link relations as provisional.

llms-full.txt

Some sites also publish llms-full.txt, a longer document that contains the full content of the site (or a curated subset of it) in plain Markdown. The intent is to give AI systems the option of consuming the entire site content in a single fetch rather than crawling each page individually.

This is most useful for documentation sites, where the goal is to enable an LLM to answer detailed questions about a product without partial-context errors caused by retrieving only one page at a time.

When is llms.txt useful?

Since llms.txt does not improve AI search visibility, consider publishing it for agent-navigation readiness or user experience reasons, not SEO/GEO reasons:

• Documentation sites. A clean Markdown index of API references, guides, and tutorials lets someone hand a whole documentation set to a coding assistant in one fetch. The beneficiary is the model being pointed at the file, not a human reading it. Some AI systems may voluntarily use it, but it is not required for citations.
• Knowledge bases and reference content. If an agent sent to your site would benefit from a curated index rather than crawling it page by page, llms.txt serves that purpose, but not search visibility.
• Personal and professional sites where you control presentation. Publishing a curated index reflects your editorial judgment about what matters, which may be valuable for your audience even if AI systems ignore it.
• Browser-based AI agents. The case here is Chrome’s stated rationale rather than its audit: a browser-based agent can understand site structure without crawling every page. Note what the audit does and does not do, because it is often cited as a reason to publish. Lighthouse 13.3 added an llms.txt check to its Agentic Browsing category, but a missing file is marked not applicable rather than failed,⁶ so having no file costs nothing in the audit. It is a reason to keep an existing file valid, not a reason to create one. If your site is likely to be accessed by task-automation agents or AI-assisted browsers, publish on the navigation argument, separate from search visibility.

The upside of publishing llms.txt is minimal, and the downside is negligible. If you have existing use cases where visitors would benefit from a clean Markdown index, publish it. If you are considering it primarily for AI search visibility, save the effort.

When it does not help

• E-commerce product catalogues. For search visibility and AI-generated shopping answers, structured data (product schema) and a clean sitemap are what matter. Shopify’s native llms.txt implementation exposes MCP/UCP checkout endpoints for AI shopping agents: a transactional use case, not a retrieval one. Publishing llms.txt does not improve a product’s chances of appearing in an AI-generated shopping answer.
• General SEO or GEO. Publishing llms.txt does not improve rankings, citations, or AI visibility. Quality content, clear structure, and authoritative authorship are what matter.

Adoption status

As of August 2026, llms.txt remains a community convention, not a formal standard. The v2 revision does not change that: it is still one person’s proposal maintained on GitHub, with no standards body behind it. Google has explicitly confirmed that its AI systems do not require or prefer llms.txt files for search visibility. No confirmed citation benefit exists.

Chrome’s Lighthouse 13.3.0 (7 May 2026) moved its Agentic Browsing category out of experimental and into the default config, and that category includes an llms.txt check.⁶ Chrome’s documentation for that audit is narrower than the check’s existence suggests. A missing file is not a failure: where the server returns a 404, “the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment”. What Lighthouse does flag is a server error while fetching the file, and for a file that does exist the guidance is that it “should follow the llms.txt specification and provide a concise Markdown summary of your site’s purpose and key links”. Chrome for Developers describes llms.txt as helping AI agents understand site structure when browsing on a user’s behalf. This is distinct from Google Search’s position: Lighthouse is auditing for agent navigation readiness, not search ranking or citation inclusion. The llms.txt check is one audit within a broader Agentic Browsing category that also covers the accessibility tree, Cumulative Layout Shift, and WebMCP integration; a passing llms.txt audit is not evidence that the file improves AI search visibility.

The most significant platform-level adoption to date is Shopify. In early May 2026, Shopify added llms.txt, agents.md, and sitemap_agentic_discovery.xml to every store on the platform, with no public announcement: the rollout was found rather than announced, and the write-up that surfaced it records all three as generated by Shopify by default.⁹ The implementation exposes MCP/UCP commerce endpoints designed for agent-driven checkout rather than search retrieval, and is customisable via a templates/llms.txt.liquid theme file.

Crawl research from WISLR, tracking bot activity on a Shopify store over 60 days, found that major AI crawlers (Bingbot, GPTBot, ClaudeBot) are not reading any of the new agent-discovery files. All three keep fetching /pages/llms-txt, the Shopify-canonical path that predates the new files, instead. The only active reader is Microsoft’s commerce platform, polling the UCP endpoint weekly as part of Copilot’s checkout integration: a transactional signal, not a search one.¹⁰

Broader log data points the same way. Ahrefs examined server logs from 137,000 domains in May 2026; around 28% published an llms.txt file, yielding roughly 38,000 valid files. 97% of those files received zero requests during the measurement period, only about 1,100 saw any traffic at all.¹¹ Among files that were requested, 96% of requests came from bots that were mostly not AI systems: SEO audit tools accounted for the largest share (21%), ahead of unidentified bots (14%) and general web crawlers (13%). The bots tied to live AI retrieval, the systems that would generate citations, barely touched the files. Most AI-adjacent requests came from coding agents and training crawlers, not retrieval systems. The dataset is the strongest evidence yet that publishing llms.txt does not feed AI search.

Some publishers continue to publish llms.txt for documentation purposes or UX reasons, unrelated to search visibility. The convention persists as a voluntary publishing standard, similar to how some sites publish human-readable sitemaps alongside sitemap.xml, even though search engines do not require them.

The practical consensus: llms.txt is not a search-related tool. Do not publish it expecting SEO or AI search benefits. Publish it for agent navigation readiness or to provide curated documentation for your audience, the cases where it does useful work.

Frequently asked questions

Does publishing llms.txt improve AI search visibility?

No. Google’s 2026 AI optimisation guide explicitly states that llms.txt files are unnecessary for AI citations or search visibility. AI systems can parse web pages directly. If your site earns traditional search visibility, you already have everything needed for AI visibility.

Where should llms.txt live?

If you choose to publish one: the root of the domain at https://example.com/llms.txt. By convention, like robots.txt. However, placement and publication are optional since AI systems do not require or prefer it.

Does publishing llms.txt help with traditional SEO?

No. It is not a search engine signal. Googlebot does not use it for indexing or ranking.

Can I use llms.txt to opt out of AI scraping?

No. llms.txt is an opt-in surface for guiding AI systems toward your content. It cannot be used to block scraping. Opt out of AI scraping via robots.txt directives targeting specific user agents (GPTBot, ClaudeBot, Google-Extended, etc.).

Should I publish llms.txt?

Publish it for agent navigation readiness or to give your audience a curated, navigable Markdown index, not for SEO or GEO benefits. The two practical cases are: your site is likely to be accessed by browser-based AI agents (Chrome’s Lighthouse audits for the file), or your visitors would benefit from a clean single-document reference. The cost is one Markdown file and the downside is negligible, so the bar for publishing is low. Just do not expect it to improve rankings, citations, or AI search visibility.

Is there a tool that generates llms.txt automatically?

Several open-source generators exist for common static site generators (Astro, Next.js, Jekyll, Hugo). For larger sites, generating llms.txt from your sitemap and content collection is straightforward. The harder work is curating which pages to include and writing useful descriptions.

Footnotes

1. Optimizing your website for generative AI features on Google Search — Google Search Central ↩

2. Google says llms.txt files won’t harm or help your search rankings — Search Engine Land ↩

3. Google’s Mueller says llms.txt can’t help LLMs differentiate sites — Search Engine Journal ↩

4. Google exposes llms.txt flaw — Search Engine Journal ↩

5. John Mueller on llms.txt and Markdown for agents — Bluesky, via Lily Ray ↩

6. llms.txt audit — Chrome for Developers / Lighthouse ↩ ↩² ↩³

7. The /llms.txt file, v2 ↩

8. Changes, llms-txt ↩

9. Shopify quietly rolled out agentic commerce and llms.txt on every store — Nikhil Vemu ↩

10. The bots crawling Shopify’s agentic stack — WISLR Research ↩

11. 97% of llms.txt files got no requests, Ahrefs data shows — Search Engine Journal ↩