llms.txt

llms.txt is a proposed convention for a Markdown file at the root of a domain that provides a clean, AI-readable index of a site’s most important content. It was first proposed by Jeremy Howard in September 2024 and has since been adopted by a growing list of documentation sites, SaaS platforms, and content publishers.

What is Google’s position on llms.txt?

Google Search does not use llms.txt. Its June 2026 AI optimisation guide states the position directly, and names generative AI specifically: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.”¹ Because Search ignores the file, maintaining one carries no benefit and no risk: it “won’t harm (nor help) your visibility or rankings”.² Google may discover, crawl, and index llms.txt like any other URL, but that is not special treatment. If your site already earns search visibility through quality content, clear structure, and credible authorship, you have everything needed for AI visibility too.

The reason for this position lies in the nature of the format: unregulated and unverifiable. As Search Advocate John Mueller put it in May 2026, a self-reported file cannot help an AI system choose between sites: every llms.txt makes the same promotional claim, in effect “here are all the pages everyone must visit”, so it cannot work as a comparative signal for discovery.³ It is the meta keywords tag again, a self-declared field Google abandoned decades ago for the same reason. Mueller added a second flaw: because the file is self-reported, its contents need not match the actual HTML, leaving an AI system no way to verify what it claims.⁴ The most he allowed was that the file might help once a system has already arrived on a site, which is the agent-navigation case below, not a search one.

Two things appear to contradict this. Google itself publishes llms.txt and Markdown pages on developers.google.com: Mueller explained these exist for AI coding tools that parse reference material more accurately from clean Markdown, “a temporary crutch, perhaps to save some tokens”, not a search signal.⁵ And Chrome’s Lighthouse does check for llms.txt, via an audit in its Agentic Browsing category.⁶ Neither reverses Search’s position: they are different parts of Google answering a different question. Lighthouse asks whether a browser-based agent can navigate your site efficiently, not whether the file affects ranking. The audit is covered in full under adoption status below.

What is llms.txt for?

The problem llms.txt addresses is straightforward. Large language models reading a website at inference time face the same challenges a human would: navigation menus, footer noise, pop-ups, advertising, and HTML scaffolding all dilute the signal. Sites are built primarily for browser rendering, not for clean text extraction.

A useful frame: llms.txt addresses functionality, not discovery. Discovery is how pages get found via search. Functionality is what happens once a visitor or agent has already arrived. The two require different tools and have different success metrics. llms.txt belongs in the functionality column, which is why publishing it has no bearing on search visibility.

llms.txt provides a curated, Markdown-formatted entry point that says: here is the canonical structure of this site, here are the most important pages, here is what they contain. It is a hint to AI systems about what matters, in a format they can ingest cleanly.

How does llms.txt differ from robots.txt and sitemap.xml?

File	Purpose	Audience
robots.txt	What can be crawled	All crawlers
sitemap.xml	What URLs exist	Search engine crawlers
llms.txt	What matters and what it means	Browser-based AI agents navigating a site
ai-catalog.json	What an agent can do on the site	Agents and registries discovering callable capabilities

For AI agents that need to act on a site rather than navigate it (completing a checkout, submitting a form, making a booking), WebMCP is the relevant protocol. llms.txt addresses navigation; WebMCP addresses action. The ai-catalog.json file, defined by the draft Agentic Resource Discovery spec, is a third, separate thing again: it declares which callable capabilities a site exposes so agents can discover them. Do not conflate the three. llms.txt is a content index for any site; ai-catalog.json is a capability manifest relevant only to sites that expose agent-callable tools.

The three are complementary, not competing. robots.txt and sitemap.xml are baseline requirements for any site; llms.txt is optional, worth adding only where the agent-navigation or documentation use cases below apply.

llms.txt syntax

The format is intentionally simple Markdown. A minimal example:

# Site Name

> One-line description of what the site is about and who maintains it.

Optional longer paragraph providing context about the site, its purpose, and any guidance for how AI systems should use the content (attribution, citation preferences, etc.).

## Section heading

- [Page title](https://example.com/page/): Short description of what this page contains.
- [Another page](https://example.com/another/): Description.

## Another section

- [Page](https://example.com/page2/): Description.

Headings group related content. Bullet links describe individual pages. Descriptions are optional but recommended; they give the AI system a hint about what each linked page covers.

llms-full.txt

Some sites also publish llms-full.txt, a longer document that contains the full content of the site (or a curated subset of it) in plain Markdown. The intent is to give AI systems the option of consuming the entire site content in a single fetch rather than crawling each page individually.

This is most useful for documentation sites, where the goal is to enable an LLM to answer detailed questions about a product without partial-context errors caused by retrieving only one page at a time.

When is llms.txt useful?

Since llms.txt does not improve AI search visibility, consider publishing it for agent-navigation readiness or user experience reasons, not SEO/GEO reasons:

• Documentation sites. A clean Markdown index of API references, guides, and tutorials can improve user experience for visitors who want a single-document reference. Some AI systems may voluntarily use it, but it is not required for citations.
• Knowledge bases and reference content. If visitors would benefit from a curated, navigable index of your content, llms.txt serves that purpose, but not for search visibility.
• Personal and professional sites where you control presentation. Publishing a curated index reflects your editorial judgment about what matters, which may be valuable for your audience even if AI systems ignore it.
• Browser-based AI agents. Chrome’s Lighthouse 13.3 introduced an Agentic Browsing audit that includes an llms.txt check (a missing file is marked not applicable rather than failed).⁶ The rationale is agent navigation: a browser-based AI agent can understand site structure without crawling every page. If your site is likely to be accessed by task-automation agents or AI-assisted browsers, this is a practical case for publishing the file, separate from search visibility.

The upside of publishing llms.txt is minimal, and the downside is negligible. If you have existing use cases where visitors would benefit from a clean Markdown index, publish it. If you are considering it primarily for AI search visibility, save the effort.

When it does not help

• E-commerce product catalogues. For search visibility and AI-generated shopping answers, structured data (product schema) and a clean sitemap are what matter. Shopify’s native llms.txt implementation exposes MCP/UCP checkout endpoints for AI shopping agents: a transactional use case, not a retrieval one. Publishing llms.txt does not improve a product’s chances of appearing in an AI-generated shopping answer.
• General SEO or GEO. Publishing llms.txt does not improve rankings, citations, or AI visibility. Quality content, clear structure, and authoritative authorship are what matter.

Adoption status

As of June 2026, llms.txt remains a community convention, not a formal standard. Google has explicitly confirmed that its AI systems do not require or prefer llms.txt files for search visibility. No confirmed citation benefit exists.

Chrome’s Lighthouse 13.3.0 (7 May 2026) moved its Agentic Browsing category out of experimental and into the default config, and that category includes an llms.txt check.⁶ The audit, titled “llms.txt follows recommendations”, marks a missing file as “Not Applicable” (a 404 is fine) but checks that a file that does exist is valid Markdown with at least one H1 header, and flags server errors when fetching it. Chrome for Developers describes llms.txt as helping AI agents understand site structure when browsing on a user’s behalf. This is distinct from Google Search’s position: Lighthouse is auditing for agent navigation readiness, not search ranking or citation inclusion. The llms.txt check is one audit within a broader Agentic Browsing category that also covers the accessibility tree, Cumulative Layout Shift, and WebMCP integration; a passing llms.txt audit is not evidence that the file improves AI search visibility.

The most significant platform-level adoption to date is Shopify. In early May 2026, Shopify quietly added llms.txt, agents.md, and sitemap_agentic_discovery.xml to every store on the platform, with no public announcement. The implementation exposes MCP/UCP commerce endpoints designed for agent-driven checkout rather than search retrieval, and is customisable via a templates/llms.txt.liquid theme file.

Crawl research from WISLR, tracking bot activity on a Shopify store over 60 days, found that major AI crawlers (Bingbot, GPTBot, ClaudeBot) are not reading any of the new agent-discovery files. All three hit legacy paths from third-party workarounds instead. The only active reader is Microsoft’s commerce platform, polling the UCP endpoint weekly as part of Copilot’s checkout integration: a transactional signal, not a search one.⁷

Broader log data points the same way. Ahrefs examined server logs from 137,000 domains in May 2026; around 28% published an llms.txt file, yielding roughly 38,000 valid files. 97% of those files received zero requests during the measurement period, only about 1,100 saw any traffic at all.⁸ Among files that were requested, 96% of requests came from bots that were mostly not AI systems: SEO audit tools accounted for the largest share (21%), ahead of unidentified bots (14%) and general web crawlers (13%). The bots tied to live AI retrieval, the systems that would generate citations, barely touched the files. Most AI-adjacent requests came from coding agents and training crawlers, not retrieval systems. The dataset is the strongest evidence yet that publishing llms.txt does not feed AI search.

Some publishers continue to publish llms.txt for documentation purposes or UX reasons, unrelated to search visibility. The convention persists as a voluntary publishing standard, similar to how some sites publish human-readable sitemaps alongside sitemap.xml, even though search engines do not require them.

The practical consensus: llms.txt is not a search-related tool. Do not publish it expecting SEO or AI search benefits. Publish it for agent navigation readiness or to provide curated documentation for your audience, the cases where it does useful work.

Frequently asked questions

Does publishing llms.txt improve AI search visibility?
No. Google’s 2026 AI optimisation guide explicitly states that llms.txt files are unnecessary for AI citations or search visibility. AI systems can parse web pages directly. If your site earns traditional search visibility, you already have everything needed for AI visibility.

Where should llms.txt live?
If you choose to publish one: the root of the domain at https://example.com/llms.txt. By convention, like robots.txt. However, placement and publication are optional since AI systems do not require or prefer it.

Does publishing llms.txt help with traditional SEO?
No. It is not a search engine signal. Googlebot does not use it for indexing or ranking.

Can I use llms.txt to opt out of AI scraping?
No. llms.txt is an opt-in surface for guiding AI systems toward your content. It cannot be used to block scraping. Opt out of AI scraping via robots.txt directives targeting specific user agents (GPTBot, ClaudeBot, Google-Extended, etc.).

Should I publish llms.txt?
Publish it for agent navigation readiness or to give your audience a curated, navigable Markdown index, not for SEO or GEO benefits. The two practical cases are: your site is likely to be accessed by browser-based AI agents (Chrome’s Lighthouse audits for the file), or your visitors would benefit from a clean single-document reference. The cost is one Markdown file and the downside is negligible, so the bar for publishing is low. Just do not expect it to improve rankings, citations, or AI search visibility.

Is there a tool that generates llms.txt automatically?
Several open-source generators exist for common static site generators (Astro, Next.js, Jekyll, Hugo). For larger sites, generating llms.txt from your sitemap and content collection is straightforward. The harder work is curating which pages to include and writing useful descriptions.

Footnotes

1. Google’s AI optimization guide ↩

2. Google says llms.txt files won’t harm or help your search rankings — Search Engine Land ↩

3. Google’s Mueller says llms.txt can’t help LLMs differentiate sites — Search Engine Journal ↩

4. Google exposes llms.txt flaw — Search Engine Journal ↩

5. John Mueller on llms.txt and Markdown for agents — Bluesky, via Lily Ray ↩

6. llms.txt audit — Chrome for Developers / Lighthouse ↩ ↩² ↩³

7. The bots crawling Shopify’s agentic stack — WISLR Research ↩

8. 97% of llms.txt files got no requests, Ahrefs data shows — Search Engine Journal ↩