Google's Mueller: llms.txt and Content Signals Have No Confirmed Crawler Effect

Google’s John Mueller has told practitioners that three of the metadata conventions being promoted for AI search, the Content-Signal robots.txt directive, llms.txt and llms-author.txt, currently have no confirmed effect on any crawler or large language model. His comments, reported this week by Search Engine Journal and Search Engine Roundtable, arrived within days of Cloudflare announcing sweeping changes to how it manages AI crawler access, giving site owners a sharp reminder that a directive only works if the systems reading it agree to honour it.

What did Google say about Content Signals and llms.txt?

Responding to a site owner who was using a llms-author.txt file and Content-Signal directives to try to fix an entity disambiguation problem, sharing a name with two better-known people, Mueller was blunt. On the Content-Signal directive he said it “was made up by a CDN” and, as far as he was aware, “has no effects whatsoever for any crawler or llm”, adding that it “just adds bloat and future maintenance to your robots.txt file”.

On the two Markdown conventions his answer was the same in substance. Google does not use llms.txt or llms-author.txt, and he was not aware of any other crawler or LLM confirming that it reads them, beyond some SEO tools. This is consistent with Google’s standing position: the company has said repeatedly that its systems do not consume llms.txt.

The framing matters. Mueller speaks for Google’s systems, not for the wider web. Other platforms run their own retrieval and crawling stacks, so his answer confirms what Google does, not what every AI system does. What it does close off is the idea that dropping one of these files onto a site is a reliable lever for how Google reads, indexes or cites the content.

What are Content Signals, llms.txt and llms-author.txt?

All three are proposed conventions rather than established standards, and none has a confirmed effect on Google.

• Content Signals is a draft directive that lives in robots.txt and lets a site declare how crawlers may use its content across categories such as search indexing, AI training and AI inference input. It began as an IETF draft and continues through the IETF aipref working group, but no major platform has publicly confirmed that it reads the directive.
• llms.txt is a proposed Markdown file that lists a site’s key pages for large language models. It is an emerging convention with no standards body behind it, and Google has said its systems do not use it. Adoption is real but its citation impact remains unproven.
• llms-author.txt is a newer, even less established variant aimed at declaring authorship and identity for AI systems.

The common thread is that each is self-declared metadata. It only carries weight if a crawler or platform chooses to read and trust it, and for Google that choice has not been made.

What is Cloudflare changing for AI crawlers?

The timing is what makes this more than a routine clarification. On 1 July 2026 Cloudflare announced a set of changes it framed as giving site owners granular control over AI crawlers. Rather than one blanket block, Cloudflare now separates crawlers by purpose, roughly search, AI training and AI agents, and lets owners allow or block each.

From 15 September 2026, Cloudflare says it will apply new defaults for sites it manages. For new domains onboarding to Cloudflare, new sites set up by existing customers, and existing free customers, training and agent crawlers will be blocked by default on pages that display ads, while search crawling stays allowed by default. Multi-purpose crawlers that combine search with training will be judged by all of their behaviours. Cloudflare is also testing an additional use field to extend Content Signals, and continues to develop pay-per-crawl mechanisms that would let publishers charge AI companies for access.

Cloudflare’s controls do work, because they are enforced at its own network edge: if Cloudflare blocks a crawler, that crawler is blocked regardless of any directive. That is a different mechanism from the Content-Signal directive itself, which relies on a crawler voluntarily reading and respecting a stated preference. Mueller’s point is about the latter. A preference written into robots.txt is not the same as an enforced block, and on Google’s side that preference currently changes nothing.

What this means for site owners

The practical takeaway is to separate enforcement from declaration. If the goal is to stop a specific AI crawler reaching a site, network-level blocking or a disallow rule aimed at a named user agent is the mechanism that bites. Adding Content-Signal preferences, llms.txt or llms-author.txt is low cost, a line or a file, but should be treated as an optional bet on future adoption, not a control that does anything on Google today.

For the entity disambiguation problem that prompted Mueller’s comments, the durable answer lies elsewhere. Being recognised as a distinct entity still rests on crawlable profiles, accurate structured data, consistent bylines, canonical links and third-party references that name and describe you, the same signals that have underpinned entity recognition since well before AI search. Self-declared files do not substitute for that evidence, and for now they do not add to it either.

Sources

• John Mueller on Content Signals, llms.txt and llms-author.txt — r/TechSEO (Reddit)
• Google Answers Question About LLMs-Author.txt For SEO — Search Engine Journal
• Google Says Cloudflare Content Signals Robots.txt Directive Has No Effects Whatsoever — Search Engine Roundtable
• Your site, your rules: new AI traffic options for all customers — Cloudflare
• Cloudflare’s new policy pushes AI companies to pay for publishers’ content — TechCrunch