---
title: "Markdown Pages for AI Agents"
description: "How to publish Markdown versions of pages for AI agents, including Cloudflare Markdown for Agents, and what the evidence shows about citation benefit."
tldr: "A Markdown version of a page is its content without the navigation, styling and scripts, served from the page URL on request or from a separate .md URL. Cloudflare's Markdown for Agents converts pages at the edge, cutting tokens by about 80% on Cloudflare's own example; static site generators and CMS plugins are the other routes. There is no evidence that it earns AI citations, and Google's AI optimisation guide lists Markdown among the things a site does not need to create. John Mueller, Google's Search Advocate, treats Markdown as a tool for developer documentation, not a search signal. Publish it where agents or readers will use it."
publishDate: 2026-05-20
updatedDate: 2026-09-28
author: "Liam Hayward"
---
A Markdown version of a page is the same content with the navigation, styling and scripts taken out, published so that AI agents, and readers pasting a page into an AI assistant, get the text without the page furniture. Sites serve it either from the page's own URL when a client asks for Markdown, or from a separate `.md` URL. In February 2026 Cloudflare turned the first route into a dashboard setting, Markdown for Agents, pitched on the argument that leaner content costs fewer tokens to ingest, so AI systems will use it more readily. The evidence does not support a citation benefit, and Google's position is that Markdown is not needed to appear in Google Search. There are specific contexts where publishing Markdown has genuine value, though, and they depend on what an agent or reader does with a page once they have it.

## How can you serve Markdown versions of your pages?

There are two ways to deliver the file. Content negotiation serves Markdown from the page's own URL when a client sends an `Accept: text/markdown` header, the standard HTTP mechanism in which a client says which formats it prefers and the server picks one. A separate URL publishes the Markdown at its own address, such as `/page.md`, which a client has to know about or be told about. The routes below produce one or both.

### What is Cloudflare Markdown for Agents?

Cloudflare Markdown for Agents sits in front of a site and responds to any client that asks for `text/markdown` through content negotiation, AI crawler or not. It fetches the HTML from the site's server, strips non-content elements such as headers, footers and navigation, and converts the rest to Markdown in real time before serving it.[^15]

The efficiency case is real: Cloudflare's own blog post consumes 16,180 tokens as HTML and 3,150 as Markdown, a reduction of roughly 80%.[^1] Cloudflare's argument is that leaner input is cheaper for LLMs to process and therefore more likely to be used.

The feature launched in February 2026 as a beta.[^1] Cloudflare's current documentation lists it at no cost on Pro, Business and Enterprise plans and for SSL for SaaS customers, and it can be switched on for a whole domain or for particular subdomains or paths, such as a `/docs/` section alone.[^15] It needs no change to the site itself, which makes it the least effort of the routes here. One default to know before enabling it: where the site's server sends no `Content-Signal` header, Cloudflare adds `Content-Signal: ai-train=yes, search=yes, ai-input=yes` to the Markdown response, declaring the content available for AI training. Setting the header on the server overrides it.[^15]

### How can you publish Markdown versions without Cloudflare?

A site whose pages are written in Markdown, as they are on most static site generators, already has the clean version before any HTML is rendered, and the build can publish that file alongside each page. The web server or CDN can then serve it at a separate URL, answer `Accept: text/markdown` on the page URL, or both. The SEO Handbook works this way: every article, guide, pillar page and news story has a twin at its `.md` URL, the same file answers content negotiation, each page declares its twin with `rel="alternate"`, and each twin sends a `Link: rel="canonical"` header pointing back to the HTML page ([how the site's Markdown versions work](/seo-news-updates/introducing-markdown-page-versions/)). Against edge conversion, this route costs engineering time and returns a file that is the author's source rather than a conversion of the rendered page.

On WordPress and other CMSs, plugins are the third route. What they deliver varies, and some serve only one of the two mechanisms while claiming both, as the risks below set out.

## Does serving Markdown improve AI citations?

No evidence supports a citation benefit. The published tests measure whether AI bots fetch Markdown rather than whether it gets cited, and neither found a gain.

The most rigorous is a randomised controlled experiment by Profound, which sells analytics for AI bot traffic and AI visibility. It randomly split 380 pages across six websites and, for 20 days in January and February 2026, served the Markdown half to detected AI bots in place of HTML, while people still received HTML. Median bot visits per page were 6 for HTML and 7 for Markdown, and every statistical test put that gap within the range of chance. The best estimate was a 13.9% lift, but its 95% confidence interval, the range the true effect plausibly falls in, ran from an 11% fall to a 47% rise. The sample was large enough to rule out a lift above about 41% and too small to detect a modest one, and Profound did not measure citations.[^2]

Dries Buytaert, founder of Drupal and co-founder of Acquia, analysed a month of Cloudflare logs for his own site after publishing Markdown three ways: a `.md` URL for every page, a `rel="alternate"` link to it from each page, and content negotiation. Crawlers did fetch the Markdown, reaching it only through the `rel="alternate"` link: it made up 35% of GPTBot's fetches, 23% of OAI-SearchBot's and 2% of ClaudeBot's. None of them used content negotiation. Bots fetched both versions rather than swapping one for the other, and his conclusion was that "there is no clear benefit today".[^3]

Google's Search Advocate John Mueller initially called the approach "a stupid idea"[^5], directing that comment at general sites rather than developer documentation. His later comments draw the distinction explicitly: Markdown is a functional tool for developer tooling, not a discovery signal for most sites.[^7][^8] He has also questioned why publishers would create pages "that no user sees," and noted that if specialised formats meaningfully improved AI responses, AI companies "would be very vocal about that."[^6]

No major AI platform, including Google, Bing, Perplexity, or OpenAI, has published guidance stating a preference for Markdown versions of live pages. Google's published position is the opposite of a preference: its AI optimisation guide lists Markdown alongside llms.txt and other "special" markup among the things you do not need to create, "as Google Search itself doesn't use them".[^13]

## Why does the token efficiency argument appeal?

The argument has surface logic. LLMs work with tokens, and a page that costs 3,000 tokens to process rather than 16,000 is cheaper to handle. The question is whether that saving translates into a search or citation benefit.

For the crawlers that build a search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot), it does not. Google says its AI features retrieve from its Search index rather than fetching the page at answer time,[^13] and no platform documents a token saving at crawl time feeding into which pages get cited. The live case is the user-triggered fetchers, ChatGPT-User, Claude-User and Perplexity-User, which do fetch a page at the moment of a query; even there, ChatGPT-User took 8 Markdown files out of 13,864 fetches in Buytaert's logs, and in Profound's test its visits leaned towards the Markdown pages by 8.5% to 20% without reaching statistical significance.[^3][^2]

The argument is more coherent for agentic use cases, where an LLM calls your page live during a session (for example, an agent researching a purchase or reading documentation to write code). Even there, no confirmed citation benefit has been established. Mueller's framing: "OF COURSE they can read HTML just fine, so this is imo more of a temporary crutch, perhaps to save some tokens."[^4]

Which agents actually request Markdown matters here. Checkly tested seven AI coding tools by checking the `Accept` headers each sends when fetching a URL. Three send `Accept: text/markdown`: Claude Code, Cursor, and OpenCode. Four do not: OpenAI Codex, Gemini CLI, GitHub Copilot, and Windsurf.[^9] These are coding tools, not AI search crawlers. The distinction is significant: search-index crawlers (OAI-SearchBot, Claude-SearchBot, Bingbot) build an index at crawl time; coding agents make live page requests at inference time. Content negotiation addresses the coding agent audience, not the search crawler one, which matches Buytaert's logs: no AI crawler sent the header.[^3]

## When does publishing Markdown make sense?

The distinction Mueller draws is between discovery and functionality. Discovery is how pages get found via search: a ranking, an AI citation, a link followed by a crawler. Functionality is what happens once someone, or an agent, has already reached the page and is trying to do something with it.[^4]

Markdown is a functionality tool. It does not influence discovery. Publishing Markdown versions of pages will not improve your rankings, your AI Overviews presence, or your citation rate in Perplexity. It may improve the experience of users already on your site who are using an AI tool to work with your content.

The use cases where this holds:

- **Developer documentation and API references.** AI coding tools such as Cursor, Claude Code and OpenCode, the three that request Markdown in the Checkly test above, work directly with reference material at inference time. Mueller's own explanation, with his hedge intact, is that "these coding systems can be (I think) efficient and accurate with the code they produce if they can easily read / parse reference material, such as developer documentation".[^7] This is the use case he cited when explaining why `developers.google.com` publishes `.md.txt` versions of its pages: the audience is AI coding assistants, not search crawlers.[^4]
- **Knowledge bases where agents are the primary audience.** If your site is built for direct agent consumption (tool documentation, integration guides, structured reference content), Markdown versions have functional value for that audience.
- **Readers who paste pages into AI assistants.** "View as Markdown" and "Copy as Markdown" controls give a reader a clean copy, headings and links intact, to paste into an assistant, a brief or a notes app. Cloudflare's documentation carries both controls.[^15] A Markdown version linked from the page is also not a page "that no user sees", the basis of Mueller's objection.[^6]

The use cases where it does not help:

- **E-commerce product pages.** A Markdown version of a product specification does not improve AI shopping citations. For products in Google's AI responses, Google points to Merchant Center feeds and Business Profile data.[^13]
- **Editorial and news content.** Serving Markdown changes the container, not the content, and nothing in Google's guide, OpenAI's publisher FAQ or the published tests connects the container to citation.[^13][^14][^2][^3]
- **General SEO or GEO.** Publishing Markdown does not improve rankings or AI visibility. Generative engine optimisation sets out which content changes the evidence does support.

## How do agents find your Markdown pages?

Content negotiation needs no discovery step: a client that sends `Accept: text/markdown` gets Markdown from the URL it already has. A Markdown version at a separate URL is different, because the client would have to guess at the address.

The August 2026 llms.txt v2 revision addresses that second case with two standard link relations.[^11] A page declares its Markdown version with `rel="alternate" type="text/markdown"`, and the llms.txt file covering it with `rel="describedby"`. Both work as HTML `<link>` elements or as an HTTP `Link:` response header, the latter settable in server or CDN configuration without touching templates. v2 also permits two URL patterns for the Markdown file itself: appending to the full URL (`/docs/tutorial.html.md`), which was the only v1 form, or replacing the extension (`/docs/tutorial.md`).

The audience is unchanged from the section above. These are conventions aimed at coding agents and documentation tooling, the same clients that send `Accept: text/markdown` today, and the same ones that made the discoverability request in the first place. Nothing about v2 constitutes evidence of a search or citation benefit, and neither Google's AI optimisation guide nor OpenAI's publisher FAQ mentions the link relations.[^13][^14]

Declaring an alternate publicly, in markup or a response header that any client can read, is a different act from serving different content depending on which user agent asks. The declaration is visible to everyone, including Google, which is a reason to prefer it over an undeclared parallel `.md` URL. Google has not commented on the link relations.

## What are the risks?

**Cloaking.** Google's spam policies define cloaking as presenting different content to users and to search engines with the intent of manipulating rankings and misleading users.[^12] A Markdown version carrying the same main content as the HTML page shows everyone the same content in a different format, on either delivery route. Content negotiation at the same URL is standard HTTP (RFC 9110; `text/markdown` has been a registered media type since RFC 7763 in 2016), and a separate `.md` URL declared with `rel="alternate"` and pointing its canonical at the HTML page is open to users and crawlers alike. Removing navigation and page furniture, as Cloudflare's conversion does,[^15] changes the container rather than the content. The risk is real where the Markdown says something the HTML does not, such as extra copy or keywords written for machines, or where the file is served only to selected user agents. Google has not published guidance on Markdown versions specifically, so this is a reading of the general policy rather than a Google statement.

**Plugins that do half the job.** Some WordPress plugins advertise "Markdown for AI" but only create separate `.md` URLs, without reading the `Accept` header on the canonical URL. When an agent sends `Accept: text/markdown` to `https://example.com/page/`, it receives HTML because the plugin never checks the header.[^10] A separate URL is a standard pattern in its own right under llms.txt v2;[^11] the gap is between what the plugin claims and what it serves. Check both: request the canonical URL with the header, and look for a `rel="alternate"` declaration pointing at the `.md` file.

**Extra crawling.** Bots fetch the Markdown as well as the HTML, not instead of it: Buytaert's crawler traffic rose about 7% after he added Markdown versions.[^3]

**Maintenance overhead.** Where the Markdown is written or edited by hand rather than generated from the same source as the HTML, two versions of every page means double the editorial work to keep content in sync, double the edge cases to test, and more surface area for inconsistencies. There is no confirmed return on that investment.

**Misallocation.** Teams spending time on Markdown conversion are not spending it on the content work the evidence does support: precise claims, named sources and direct answers.

## Frequently asked questions

### Does Cloudflare Markdown for Agents improve AI search visibility?

No evidence supports this. Cloudflare's feature works by content negotiation, and in a month of logs from a site that offered it, Dries Buytaert, founder of Drupal and co-founder of Acquia, found no AI crawler using content negotiation at all; the clients known to send the header are coding tools such as Claude Code and Cursor.[^3][^9] Google's guide lists Markdown among the things you do not need to create, and OpenAI's publisher FAQ does not mention it.[^13][^14]

### Is serving Markdown to bots considered cloaking?

Not in itself. Google defines cloaking as showing users and search engines different content to manipulate rankings.[^12] A Markdown version with the same main content, served by content negotiation or at a public `.md` URL declared with `rel="alternate"`, gives everyone the same content in a different format. The risk arises when the Markdown adds copy the HTML page lacks, or is served only to selected bots. Google has not published guidance on Markdown versions specifically.

### Should I enable Cloudflare Markdown for Agents?

If your site is on a Cloudflare Pro plan or above and has developer documentation or reference content that AI coding tools use, it is the lowest-effort way to serve Markdown, and it can be limited to those sections by path. Set a `Content-Signal` header on your server first if you do not want the Markdown responses to declare the content available for AI training.[^15] For other sites there is no evidence it earns citations, so it is an option for agents that ask, not a visibility tactic.

### Why does Google publish Markdown files on developers.google.com?

John Mueller, Google's Search Advocate, explains that coding tools "can be (I think) efficient and accurate with the code they produce if they can easily read / parse reference material".[^7] Google publishes these for the benefit of developers using AI coding assistants, not for search visibility. It is a functional publishing decision specific to developer documentation.

### How does this relate to llms.txt?

Both are functionality tools rather than discovery tools: they affect what happens once an agent reaches your site, not whether your site gets found. See [llms.txt](/ai-search/llms-txt/) for the index file convention. Neither has a measured effect on AI search citations.

[^1]: [Introducing Markdown for Agents — Cloudflare Blog](https://blog.cloudflare.com/markdown-for-agents/)
[^2]: [Does Markdown Increase LLM Bot Traffic? A Randomized Controlled Experiment — Profound](https://23651831.fs1.hubspotusercontent-na1.net/hubfs/23651831/markdown_llm_paper%20%281%29.pdf). Brandon Punturo, February 2026: 380 pages on six sites, 19 January to 8 February 2026, measured with Profound's own bot-analytics product; Profound sells AI visibility tracking.
[^3]: [Markdown, llms.txt and AI crawlers — Dries Buytaert](https://dri.es/markdown-llms-txt-and-ai-crawlers). One month of Cloudflare logs for his own site, published 5 March 2026.
[^4]: [John Mueller on llms.txt and Markdown for agents — Bluesky, via Lily Ray](https://bsky.app/profile/lilyray.nyc/post/3mmaio3nogk2d)
[^5]: [Google's Mueller calls Markdown for bots "a stupid idea" — Search Engine Journal](https://www.searchenginejournal.com/googles-mueller-calls-markdown-for-bots-idea-a-stupid-idea/566598/)
[^6]: [Google's Mueller questions need for LLM-only Markdown pages — Search Engine Journal](https://www.searchenginejournal.com/googles-mueller-questions-need-for-llm-only-markdown-pages/561830/)
[^7]: [Mueller explains why Google uses Markdown on dev docs — Search Engine Journal](https://www.searchenginejournal.com/mueller-explains-why-google-uses-markdown-on-dev-docs/575465/)
[^8]: [Google adds Markdown files to help docs — Search Roundtable](https://www.seroundtable.com/google-adds-markdown-files-to-help-docs-41342.html)
[^9]: [State of AI agent content negotiation — Checkly](https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/)
[^10]: [Some SEO plugins claim Markdown for AI but ignore the Accept header — Roots.io](https://roots.io/some-seo-plugins-claim-markdown-for-ai-but-ignore-the-accept-header/)
[^11]: [The /llms.txt file, v2](https://llmstxt.org/)
[^12]: [Cloaking — Google Search Central spam policies](https://developers.google.com/search/docs/essentials/spam-policies#cloaking)
[^13]: [Optimizing your website for generative AI features on Google Search — Google Search Central](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
[^14]: [Publishers and Developers FAQ — OpenAI Help Center](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq). As of 18 September 2026 it covers OAI-SearchBot access, `noindex` and referral parameters, and does not mention Markdown, llms.txt or link relations.
[^15]: [Markdown for Agents — Cloudflare Docs](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/). Documentation last updated 13 July 2026.