Agent-readiness

Agent-readiness is whether AI agents can find a site, read its content in formats suited to machine consumption, and, where the site offers them, carry out tasks on it. It is to agents roughly what crawlability is to search engines: content they cannot reach does them no good however well it is written. Generative Engine Optimisation and agentic search address the content layer, what agents retrieve and how well individual passages are selected for citation; agent-readiness is the infrastructure underneath.

What does agent-readiness mean?

Two distinct requirements sit under the umbrella, and they are often conflated.

Can agents find and read your content? This covers discovery signals (robots.txt, sitemaps, response headers), machine-readable content formats (structured data, content negotiation), and access declarations (what agents are permitted to do with content they retrieve).

Can agents take actions on your site? This covers protocol endpoints, callable tool definitions, and transaction interfaces: the infrastructure that lets an agent complete a task on behalf of a user rather than simply read what is on the page.

Most content and editorial sites need only address the first. E-commerce, booking, SaaS, and tool sites need both. The action layer is covered in depth in WebMCP. This article covers the full infrastructure picture across both requirements.

How are major platforms defining agent-readiness?

No single authority owns the definition. Four platforms are converging on the concept simultaneously, each from a different vantage point.

Cloudflare launched isitagentready.com, a public scanner grading a domain against the agent stack.¹ At launch it scored 12 checks across four categories, Discoverability, Content, Bot Access Control and Capabilities, with a separate group of commerce checks that Cloudflare states “do not currently count towards the score”.¹ The check list has grown since: as at 1 September 2026 the live scanner runs 20 checks across five categories, discoverability, content accessibility, bot access control, protocol discovery, and commerce.² Because the scanner is updated in place rather than versioned, treat any count as as-at rather than fixed. It is designed for the full range of Cloudflare’s platform, from content sites to API services to commerce, which is why the commerce checks sit outside the score.

Cloudflare’s scan of 200,000 top domains (filtered to exclude redirects, ad servers, and tunnelling services) shows how far the web is from agent-ready by default: 78% of sites have a robots.txt, but the vast majority were written for search engine crawlers, not AI agents. Only 4% have declared AI usage preferences via Content Signals. Only 3.9% support Markdown content negotiation. Fewer than 15 sites in the entire dataset have MCP Server Cards or API Catalogs.³

Shopify is making every store agent-ready by default through its Agentic Storefronts initiative.⁴ Its own public scanner at shopify.com/agentic-readiness runs 31 checks, of which the URL layer makes 45 requests across 41 distinct URLs, and it expects /llms.txt, /agents.md and /.well-known/ucp to return a 200. Shopify does not publish that check list: those figures come from WISLR running a research storefront through the scanner in May 2026.⁵ The emphasis is on machine-readable product data: price, title, dimensions, and availability must appear in structured fields, not embedded in Liquid templates or marketing copy. Shopify’s own growth figures for the channel have moved fast enough that any single one dates quickly: it reported AI-driven traffic up roughly eightfold and orders up around thirteenfold year on year in its Q1 2026 results, then both at threefold in Q2.⁶ They are the platform’s own numbers, published in support of the initiative, so read them as directional rather than independently established. The current figures are tracked in AI shopping agents and ecommerce visibility.

WordPress introduced an MCP Adapter as part of its AI Building Blocks initiative, announced in early 2026.⁷ The Adapter exposes every registered WordPress ability as a callable MCP tool, allowing AI agents to discover and execute WordPress functionality directly. The Abilities API provides a standardised way for plugins to register callable actions; once a plugin registers an ability, an AI agent can discover and invoke it. For WordPress-hosted sites, agent-readiness at the action layer is becoming a platform default rather than a custom implementation.

Google and Chrome moved WebMCP from an experimental Canary flag to a formal origin trial in Chrome 149, open since June 2026.⁸ Sites declare callable tools (JavaScript functions and HTML forms) that in-browser agents can invoke without simulating clicks. Early participants include Expedia, Booking.com, Shopify, Etsy, Instacart, and Target. Shopify has since made it a platform default, registering WebMCP tools on every Liquid storefront from August 2026 and on eligible checkouts from 28 September 2026, with no merchant setup.⁹ WebMCP is covered in full at WebMCP.

Chrome’s Lighthouse also formalised agent-readiness auditing in version 13.3 (May 2026), moving an Agentic Browsing category from experimental to its default configuration.¹⁰ It runs alongside Performance, Accessibility, Best Practices, and SEO on every Lighthouse run. The category checks for llms.txt presence, WebMCP tool declarations, accessibility tree quality, and visual stability. Unlike other Lighthouse categories it does not produce a single 0–100 score; the spec team’s stated rationale is that agentic web standards are still emerging and the priority is actionable signals rather than a definitive ranking.

Across these four converging definitions, agent-readiness is becoming platform-level infrastructure rather than a DIY checklist. The common infrastructure threads across all four approaches are what matter for most publishers.

The five infrastructure layers

Synthesised from the common threads across platform approaches, five layers describe what agent-readiness requires in practice.

1. Discovery

How agents learn a site exists and navigate its structure. The basic signals are the same as those for search: a valid robots.txt with named crawler rules, a current XML sitemap, and structured markup that identifies what content exists. HTTP Link headers (RFC 8288) add a machine-readable pointer to the sitemap and key resources directly in the HTTP response, without requiring agents to parse robots.txt first. llms.txt, placed at the domain root, provides a curated Markdown index of site structure that in-browser agents can read without crawling the full site; Chrome’s Lighthouse Agentic Browsing audit checks for its presence in the default configuration.¹⁰ For sites running MCP servers or agent skills, /.well-known/ endpoints advertise those capabilities at a known, discoverable location: the prefix is reserved by RFC 8615 for site-wide metadata.

2. Content format

Whether agents can receive content in efficient, machine-readable form. What the two operators that have said anything document their browser agents reading is the page itself: Google’s guide lists “analyzing visual renderings (like screenshots), inspecting the DOM structure, and interpreting the accessibility tree”,¹¹ and OpenAI said ChatGPT Atlas, the browser it shut down in August 2026, used ARIA tags.¹²¹³ Structured data (schema.org Article, Product, Organization) sits behind that: Microsoft says structured content, with schema-marked pages as its example, helps its AI answers interpret and summarise a page,¹⁴ and Google says its AI features do not require it.¹¹ Markdown content negotiation serves .md responses when a request includes Accept: text/markdown, reducing token overhead for agent retrieval without affecting how browsers receive HTML. Microsoft’s Web IQ, a grounding API built on Bing’s index, returns “passages and structured evidence objects” as well as documents, and Microsoft grades its results with GDSAT (grounding satisfaction), a metric covering “completeness, freshness, and authority”.¹⁵ Microsoft’s President of Search and AI has said Web IQ is used directly in Copilot and by ChatGPT “for some of its web answers”,¹⁶ so it serves part of ChatGPT’s web retrieval, not all of it. GDSAT is how Microsoft measures its own system; Microsoft publishes no rule for how individual pages or passages are scored. For commerce sites, machine-readable product data (structured fields rather than marketing copy) is the commerce-layer equivalent. Markdown pages for AI agents covers the implementation in full, including its limits.

A related convention is emerging for curated knowledge rather than page content. In June 2026 Google Cloud published the Open Knowledge Format (OKF), a draft spec that represents knowledge as a directory of Markdown files with YAML frontmatter for AI agents to consume. Google scopes it to internal organisational data, not web search, but it is the same Markdown-with-frontmatter shape as llms.txt and agents.md, and some SEOs are asking whether sites could publish OKF bundles externally. Google’s own announcement scopes it to internal data and says nothing about reading external bundles, so for now it sits in the watch column alongside the other draft conventions of the agentic web.

3. Access declarations

What agents are permitted to do with content they access. Per-crawler rules in robots.txt handle allow and disallow at the access layer; AI crawlers covers the major user-agents and compliance behaviour.

The layer above robots.txt is becoming the decisive one. Cloudflare sorts AI crawlers into Search, Agent and Training categories and enforces the policy at the network layer rather than through the file.¹⁷ Since 15 September 2026 it offers newly onboarded ad-supported domains a preset that blocks training-only crawlers across the domain and agents on pages with ads, with Search allowed.¹⁸ Two things follow for agent-readiness. Your access declarations may no longer be the operative control if your CDN is enforcing its own, and choosing Block on the Training category now also takes out Googlebot, Bingbot and Applebot, which crawl for both search and training. The “Disallow AI Training” setting added the same day opts out of training while keeping them crawling for search, though it will not reach Bing until Microsoft’s robots.txt support arrives, targeted for early 2027.¹⁸ Bing’s own route today is a noarchive robots meta tag, which Bing documents as keeping content out of training for Microsoft’s generative AI models but also as “Do not link in Chat and Copilot”; nocache limits both uses to the URL, title and snippet.¹⁹ Any site already blocking Training, through the legacy preset or the granular controls, was migrated to it automatically, and the new-domain preset does not touch an existing site’s configuration. Any agent-readiness audit that stops at robots.txt will still miss the layer that actually decides what reaches your server.

Content Signals, a draft IETF specification originally published as draft-romm-aipref-contentsignals-00 (October 2025, published at contentsignals.org), adds a separate layer of semantic intent. That specific draft expired in April 2026 under IETF’s standard 6-month lifecycle; the underlying work continues through the IETF aipref working group.²⁰ A Content-Signal: directive in robots.txt declares preferences for three specific uses:

• ai-train: permission to use content for training AI model weights
• search: permission to include content in search indexing and results
• ai-input: permission to use content as input to AI at inference time (retrieval, RAG)

The distinction between ai-train and ai-input is not obvious but matters: a site can permit retrieval citation (ai-input=yes) while declining to have its content absorbed into model training data (ai-train=no). The spec is a draft with no confirmed adoption by major platforms yet. The implementation cost is one line in robots.txt.

Microsoft is separately engaging with IETF on publisher controls for AI infrastructure through its Web IQ platform,¹⁵ part of a broader industry movement toward formalised standards in this space. A related strand is request-time verification rather than declared preferences: in June 2026 Cloudflare, Google, Microsoft, Mozilla and Shopify announced PACT, a draft protocol for cryptographically proving a human, or an agent authorised by one, is behind a request. It is in development with no deployment timeline, but signals where the access layer is heading: from self-reported user-agents towards proof of who a client is and on whose behalf it acts.

agents.md, introduced in the Shopify agentic commerce context, is a plain-language description of what agents can and cannot do on a site, including usage terms for AI interaction. It is specific to commerce at present but the convention may broaden.

4. Protocol and action layer

Whether agents can do things on the site beyond reading. MCP Server Cards (/.well-known/mcp/server-cards.json) advertise an MCP server’s endpoint, capabilities, and transport method. The spec is an open draft, SEP-2127, still unmerged into the core MCP specification and last updated on 24 August 2026.²¹ It is no longer named on the MCP roadmap, whose five priority areas as at 22 August 2026 are agentic messaging primitives, transport unification, agent identity, improved primitives and SDK developer experience, and there is no Server Card working group among the chartered groups.²² Real-world implementations exist, but the artefact is running ahead of the specification rather than behind an agreed one. Agent Skills indices (/.well-known/agent-skills/index.json) list callable actions with name, type, description, URL, and a SHA256 digest. Both are relevant only for sites that run MCP servers or expose agent-callable actions. WebMCP handles the browser-native variant. NLWeb takes a server-side approach: it converts existing schema.org structured data into queryable endpoints for MCP-compatible agents, making a site’s catalogue accessible without custom JavaScript. It is most relevant for commerce, media, and documentation sites with deep content catalogues.

Agentic Resource Discovery (ARD) sits above these per-capability artifacts as a unifying discovery layer. Announced in June 2026 by Google, Microsoft, and Hugging Face, ARD defines a manifest at /.well-known/ard.json that lists a site’s agentic resources (MCP servers, A2A agents, skills, APIs) in one place, plus a registry API that crawls and indexes those catalogs so agents can find capabilities by natural-language query. A catalog can also be advertised through an Agentmap directive in robots.txt or a <link rel="ard"> tag. Version 0.91 (August 2026) renamed both from their launch names, ai-catalog.json and rel="ai-catalog", and says publishers on the old path should move;²³ the ARD audit in Lighthouse 13.5.0 still checks only the old names, pending an open fix.²⁴ The spec is an early v0.91 proposal (Apache 2.0) with a broad backing coalition but no confirmed registry or agent adoption at scale yet; like Server Cards, it is relevant only to sites that expose agent-callable capabilities. Cloudflare’s scanner already scores an ARD manifest check inside its protocol discovery category, which is a quick way to see whether a site publishes one.² See the ARD announcement for detail.

A2A (Agent-to-Agent protocol, now under the Linux Foundation alongside MCP) defines how agents delegate tasks to other agents.²⁵ Publishers do not implement it: it lives in the platforms that run the agents, not on individual sites.

One item in this layer is documented, first-party and already on most sites’ accessibility list. OpenAI states that ChatGPT Atlas “uses ARIA tags, the same labels and roles that support screen readers, to interpret page structure and interactive elements”, and recommends WAI-ARIA best practices, “adding descriptive roles, labels, and states to interactive elements like buttons, menus, and forms”, so the agent can recognise what each element does.¹² It is the one first-party statement that a browser agent reads accessibility markup, and it is scoped to Atlas, which stopped working on 9 August 2026 when OpenAI moved its agent browsing into the ChatGPT desktop app.¹³ Neither the FAQ nor OpenAI’s help page on the desktop app’s browser says whether the ARIA guidance carries over. A site that meets WCAG for its own users has done this work already.

For most content and editorial sites the rest of this layer is a watch item, not a current implementation priority.

5. Commerce and transaction layer

Whether agents can complete transactions on behalf of users, rather than only read the page. This is the one layer that is not a publishing concern: it applies to commerce sites only, and informational and editorial sites skip it entirely.

The protocols are the Universal Commerce Protocol (UCP, which a participating site advertises at /.well-known/ucp), Google’s Agent Payments Protocol (AP2) as its recommended payments layer,²⁶ the Agentic Commerce Protocol (ACP) from Stripe and OpenAI, and x402 from Coinbase. Google states UCP is compatible with AP2 for payments and supports A2A and MCP, so at least those layers are designed to stack rather than compete.²⁷ Whether ACP and x402 compose with UCP as cleanly is asserted by the projects rather than demonstrated, so do not read the set as a settled stack a merchant simply plugs into.

What a merchant should actually implement, what the platforms have shipped, and the gap between the available infrastructure and real transaction volume are covered in agentic commerce. The short version, because it changes how seriously to take this layer: the plumbing is real and working, and the buying behaviour has not followed it yet.

Infrastructure requirements by site type

Commerce layer protocols (UCP, AP2, ACP, x402) apply to e-commerce sites only; the other site types skip that layer entirely.

“Monitor only” for the protocol layer means understanding the protocols without implementing them. As platforms mature, the action layer will often be handled at the hosting or CMS level without site-specific configuration.

What to implement now

For content and editorial sites, four things are actionable today. Actionable means the work is small and the downside is negligible, not that the benefit is confirmed: two of the four are bets on adoption that nothing currently reads, and each says so below.

Content Signals (minutes): add one line to robots.txt:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Set ai-train to no to decline training use while permitting retrieval. The original draft (-00) expired April 2026; the specification work continues in the IETF aipref working group. Adoption by major platforms is unconfirmed, and the cost is near zero.

If Cloudflare serves your site it may already have added the directive for you, so check what your robots.txt currently says before writing it. Note also that Google’s John Mueller said in July 2026 that, as far as he was aware, no crawler or LLM acts on the directive (see the FAQ below): this is a low-cost bet on future adoption, not a control that does anything today.

HTTP Link headers (minutes): add one response header pointing at a site-level resource, using a relation type IANA registers. In Cloudflare’s _headers file:

Link: </llms.txt>; rel="describedby"

RFC 8288 is an established standard, and Link headers are one of the discoverability checks in Cloudflare’s scanner, whose own example is rel="api-catalog" for a list of APIs.¹ Take the relation from the IANA registry: describedby, “a resource providing information about the link’s context”, fits an llms.txt, and api-catalog fits a site that publishes APIs.²⁸ There is no registered relation for a sitemap, so rel="sitemap", which some agent-readiness guides suggest, is one no agent is documented reading. The sitemap protocol’s own discovery route is a Sitemap: line in robots.txt.²⁹

Markdown content negotiation (hours to days): configure the server to return .md responses when requests include Accept: text/markdown. Implementation options, limits, and risks are in Markdown pages for AI agents.

llms.txt (minutes): a Markdown file at the domain root listing the site’s key pages with short descriptions. The case for it is agent navigation, so that an in-browser agent can read site structure without crawling every page. Chrome’s Lighthouse Agentic Browsing audit is not that case: it marks a missing file not applicable rather than failed, so having none costs nothing in the audit and it is a reason to keep an existing file valid rather than to create one. Neither Google nor OpenAI lists the file among what their browser agents read: Google names screenshots, the DOM and the accessibility tree, and OpenAI names ARIA.¹¹¹² No SEO or citation benefit, and low cost. See llms.txt for syntax and implementation.

Strong structured data (Article, Organization) underlies all four and stays on the list for a content site, with its documented scope: it is the form of structured content Microsoft says helps its AI answers interpret a page, and it earns rich results in Google Search, and what Google and OpenAI document their agents reading is the rendered page, the DOM and the accessibility tree.¹¹¹² Say who produced the page in the prose and the byline, which every agent reads, and mirror it in the markup.

What not to implement, and the reasons differ:

• Counterproductive: API catalog and OAuth discovery. Publishing /.well-known/openid-configuration or similar OAuth discovery files without actual protected APIs creates a worse agent experience than omitting them: it signals capabilities the site does not have.
• Not applicable: commerce layer protocols and WebMCP. An editorial site has no transactions to complete and no on-page actions for an agent to call.
• Conditional: MCP Server Card and Agent Skills indices each describe something that has to exist first, an MCP server or a set of callable actions, and publishing either without it repeats the OAuth mistake above. Most content sites run neither, so the answer is usually still no. It is a cost question rather than a category error, though: MCP is not only an action protocol, and of the three server primitives in its specification, tools perform actions while resources supply read-only context.³⁰ A catalogue deep enough to justify exposing that is the NLWeb case described above, and building it is a project with ongoing maintenance rather than a file to publish. Nothing in it improves search or AI visibility.

Current state and roadmap

Agent-readiness is moving from opt-in to default at the platform level. Shopify is making every store agent-ready without merchant action. WordPress is building the action layer into the core CMS. WebMCP is moving from an experimental flag to a formal browser origin trial with major commerce and travel brands already participating. Cloudflare’s scanner formalises checks that may appear in CDN-level defaults.

For most content publishers the near-term implication is limited: the action layer is handled by platforms, not individual sites. The decisions that remain are about access declarations (what Content Signals values to publish) and content formats (whether to serve Markdown). The infrastructure layer is developing quickly enough that checks which require monitoring in 2026 may require implementation in 2027.

As specific platforms (Shopify, WordPress) publish their own agent-readiness requirements, those will be covered in platform-specific articles. This article covers the common infrastructure layer that applies across site types.

Frequently asked questions

Is agent-readiness the same as technical SEO?

Related but distinct. Technical SEO ensures search engines can crawl, index, and understand a site. Agent-readiness ensures AI agents can access, read, and act on it. The two overlap (structured data, crawlability, response codes) but agent-readiness extends into areas technical SEO does not cover: access declarations for different AI uses, machine-readable content formats, and action protocols.

Does improving agent-readiness affect search rankings?

No confirmed mechanism. The infrastructure layer affects whether agents can access content, not how search algorithms rank it.

Should I implement WebMCP for a content site?

No. WebMCP is the action layer, relevant when the site’s primary value involves completing tasks on behalf of users. For informational sites the relevant investment is the discovery and content format layers.

What is Content Signals and does it work?

Content Signals is a draft IETF specification adding semantic intent to robots.txt: three fields declaring preferences for AI training, search indexing, and inference input. The original draft (draft-romm-aipref-contentsignals-00) expired in April 2026; the work continues through the IETF aipref working group. No major platform has publicly confirmed it reads these directives, and in July 2026 Google’s John Mueller said that, as far as he was aware, no crawler or LLM acts on the directive, calling it something “made up by a CDN” that “just adds bloat” to robots.txt.³¹ The implementation cost is one line, so treat it as a low-risk optional bet on future adoption rather than a change with confirmed impact today.

How do I audit my site’s agent-readiness?

isitagentready.com is a useful starting point. Use the Content Site preset rather than All Checks; the default preset penalises informational sites for missing API and commerce features they have no reason to implement. Of the 20 checks the scanner runs as at 1 September 2026, 12 sit in its protocol discovery and commerce categories, covering API, authentication, MCP and payment infrastructure that does not apply to content sites; the Content Site preset unchecks both categories along with Web Bot Auth. A low score on those checks reflects the scanner’s broad scope, not a site deficiency.³

Footnotes

1. Introducing the Agent Readiness score — Cloudflare ↩ ↩² ↩³

2. Is It Agent Ready? — Cloudflare ↩ ↩²

3. All you need to know about Cloudflare’s Agent Readiness Score — Search Engine Journal ↩ ↩²

4. Shopify Agentic Storefronts — Shopify ↩

5. What Shopify checks for agentic readiness — WISLR Research. Derived by running a research storefront through Shopify’s public scanner, 9 May 2026. ↩

6. Shopify (SHOP) Q2 2026 earnings call transcript — The Motley Fool ↩

7. From Abilities to AI Agents: introducing the WordPress MCP Adapter — WordPress Developer Blog ↩

8. Chrome at I/O 2026: powering the agentic web — Chrome for Developers ↩

9. WebMCP tools — Shopify Developers; checkout tools at Checkout WebMCP — Shopify Developers. ↩

10. Lighthouse agentic browsing scoring — Chrome for Developers ↩ ↩²

11. Optimizing your website for generative AI features on Google Search — Google Search Central. The agentic section and the “overfocusing on structured data” entry. ↩ ↩² ↩³ ↩⁴

12. Publishers and developers FAQ — OpenAI Help Center, Developer FAQs, “What can I do to improve my website performance with ChatGPT agent in Atlas?” ↩ ↩² ↩³ ↩⁴

13. Evolving Atlas into ChatGPT for browser-based agentic work — OpenAI Help Center ↩ ↩²

14. How AI Search Is Changing the Way Conversions are Measured — Bing Webmaster Blog, Fabrice Canel and Krishna Madhavan, 20 November 2025: “Structured content, like schema-marked product pages, FAQs, and comparison tables, helps AI systems interpret and summarize your content more effectively.” ↩

15. Announcing Microsoft Web IQ — Bing Search Blog ↩ ↩²

16. Microsoft Web IQ (Microsoft names “AI platforms like Copilot and OpenAI”), with Jordi Ribas, President of Search and AI, quoted in Microsoft releases Web IQ, powered by Bing but designed for how AI-agents search — Search Engine Land ↩

17. Content Independence Day: bringing AI options to every site — Cloudflare ↩

18. Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog ↩ ↩²

19. Robots meta tags and attributes that Bing supports — Bing Webmaster Tools ↩

20. draft-ietf-aipref-attach-04 — IETF datatracker ↩

21. MCP Server Cards — PR-2127 (open draft) ↩

22. Model Context Protocol roadmap — modelcontextprotocol.io ↩

23. Agentic Resource Discovery Specification v0.91, agenticresourcediscovery.org, section 5.1 on the well-known URI and its predecessor path. ↩

24. ard-schema not updated for ARD v0.91, Lighthouse issue #17251, GitHub ↩

25. A2A Protocol Surpasses 150 Organizations, Lands in Major Cloud Platforms — Linux Foundation ↩

26. Announcing Agent Payments Protocol (AP2) — Google Cloud Blog ↩

27. Under the Hood: Universal Commerce Protocol — Google Developers Blog ↩

28. Link Relations registry — IANA ↩

29. Sitemaps XML format: specifying the Sitemap location in your robots.txt file — sitemaps.org ↩

30. Architecture overview — Model Context Protocol ↩

31. John Mueller’s reply in r/TechSEO — Reddit, 6 July 2026; reported by Search Engine Roundtable ↩