Markdown Pages for AI Agents

Last updated 20 May 2026

In February 2026, Cloudflare launched a feature called Markdown for Agents, which automatically converts HTML pages to Markdown and serves them to AI crawlers. The pitch: leaner content costs fewer tokens to ingest, so AI systems will use it more readily. The evidence does not support the claim, and Google’s position is that serving Markdown to bots is unnecessary. The longer answer matters, though, because there are specific contexts where publishing Markdown has genuine value.

What is Cloudflare Markdown for Agents?

Cloudflare Markdown for Agents sits in front of a site and intercepts requests from AI crawlers that signal a preference for text/markdown via content negotiation headers. It converts the HTML response to Markdown in real time before serving it.¹

The efficiency case is real: Cloudflare’s own blog post consumes 16,180 tokens as HTML and 3,150 as Markdown, a reduction of roughly 80%. The argument is that leaner input is cheaper for LLMs to process and therefore more likely to be used.

The feature launched in February 2026 as a Beta at no additional cost for Cloudflare Pro, Business, and Enterprise plans. It is one of several routes to the same outcome: WordPress plugins, static site generators, and manual .md routes can all produce Markdown versions of pages without Cloudflare.

Does serving Markdown improve AI citations?

No evidence supports a citation benefit. Otterly.ai ran a controlled 14-day experiment comparing HTML and Markdown versions of the same pages, giving both equal placement and discoverability on the site. AI crawlers recorded 137 visits to the HTML pages and zero visits to the Markdown pages. No AI platform cited a .md URL across the full test period.²

SALT.agency, analysing behaviour across hundreds of thousands of domains, found no meaningful correlation between serving Markdown to AI crawlers and increased citations in AI-generated answers.³

Google’s Search Advocate John Mueller initially called the approach “a stupid idea”⁴, directing that comment at general sites rather than developer documentation. His later comments draw the distinction explicitly: Markdown is a functional tool for developer tooling, not a discovery signal for most sites.⁵⁶ He has also questioned why publishers would create pages “that no user sees,” and noted that if specialised formats meaningfully improved AI responses, AI companies “would be very vocal about that.”⁷ Serving different content to bots than to human visitors also raises a cloaking concern: Google’s spam policies prohibit serving materially different content depending on the user agent, regardless of intent.

No major AI platform, including Google, Bing, Perplexity, or OpenAI, has published guidance stating a preference for Markdown versions of live pages.

Why does the token efficiency argument appeal?

The argument has surface logic. LLMs work with tokens, and a page that costs 3,000 tokens to process rather than 16,000 is cheaper to handle. The question is whether that saving translates into a search or citation benefit.

For AI search crawlers (GPTBot, ClaudeBot, BingbotLLM), it does not. These systems index pages at crawl time, not at inference time. The LLM generating an answer does not re-fetch your page when a user submits a query: it draws on previously indexed content. Token savings at crawl time do not translate into citation priority.

The argument is more coherent for agentic use cases, where an LLM calls your page live during a session (for example, an agent researching a purchase or reading documentation to write code). Even there, no confirmed citation benefit has been established. Mueller’s framing: “OF COURSE they can read HTML just fine, so this is imo more of a temporary crutch, perhaps to save some tokens.”⁸

When does publishing Markdown make sense?

The distinction Mueller draws is between discovery and functionality. Discovery is how pages get found via search: a ranking, an AI citation, a link followed by a crawler. Functionality is what happens once someone, or an agent, has already reached the page and is trying to do something with it.⁸

Markdown is a functionality tool. It does not influence discovery. Publishing Markdown versions of pages will not improve your rankings, your AI Overviews presence, or your citation rate in Perplexity. It may improve the experience of users already on your site who are using an AI coding tool to work with your content.

The use cases where this holds:

Developer documentation and API references. AI coding tools such as Cursor, GitHub Copilot, and Claude Code work directly with reference material at inference time. Clean Markdown reduces noise and improves the accuracy of generated code. This is the use case Mueller cited when explaining why developers.google.com publishes .md.txt versions of its pages: the audience is AI coding assistants, not search crawlers.⁸
Knowledge bases where agents are the primary audience. If your site is built for direct agent consumption (tool documentation, integration guides, structured reference content), Markdown versions have functional value for that audience.

The use cases where it does not help:

E-commerce product pages. A Markdown version of a product specification does not improve AI shopping citations. Structured data and catalogue quality are what matter.
Editorial and news content. AI citation of editorial content is driven by authority, accuracy, and structure. Serving Markdown does not change any of those signals.
General SEO or GEO. Publishing Markdown does not improve rankings or AI visibility. Accuracy, clear structure, direct answers, and credible authorship are what earn citations.

What are the risks?

Cloaking. Serving materially different content to bots than to human visitors is a Google spam policy violation, regardless of intent. Auto-conversion tools strip navigation, CTAs, and structural elements. Whether this constitutes cloaking depends on implementation, but the risk is real and the benefit is unproven.

Maintenance overhead. Two versions of every page means double the editorial work to keep content in sync, double the edge cases to test, and more surface area for inconsistencies. There is no confirmed return on that investment.

Misallocation. Teams spending time on Markdown conversion are not spending it on the content quality signals that actually influence AI citations: factual accuracy, clear structure, direct answers, and credible authorship.

Frequently asked questions

Does Cloudflare Markdown for Agents improve AI search visibility?
No evidence supports this. In controlled testing, AI crawlers visited HTML pages and ignored the Markdown equivalents, and no AI platform cited a .md URL. Google, Bing, and other major platforms have not published guidance suggesting they prefer Markdown versions of live pages.

Is serving Markdown to bots considered cloaking?
It depends on implementation. Serving content that differs materially from what human visitors see, even for efficiency reasons rather than manipulation, is cloaking-adjacent. Google’s position is that its systems can read HTML without difficulty.

Should I enable Cloudflare Markdown for Agents?
Only if your site is developer documentation or a knowledge base where AI coding tools are a meaningful part of your audience. For most sites, the feature offers no citation benefit and introduces cloaking risk and maintenance overhead.

Why does Google publish Markdown files on developers.google.com?
Because AI coding tools work more accurately when reading clean reference material. Google publishes these for the benefit of developers using AI coding assistants, not for search visibility. It is a functional publishing decision specific to developer documentation.

How does this relate to llms.txt?
Both are functionality tools rather than discovery tools: they affect what happens once an agent reaches your site, not whether your site gets found. See llms.txt for the index file convention. Neither improves AI search citations.