Industry Snippet

Google Publishes a Guide to Letting Google-Agent Past Bot Defences

Markdown
View as Markdown Open this page as plain text
This page as Markdown, ready to paste into ChatGPT, Claude or a doc.
RSS
A cartoon of the house Google character, a blue ball in a four-colour propeller cap, standing outside a shop with a purple and white striped awning and an OPEN sign on its open door. A large white panel with an empty checkbox and the words I'M NOT A ROBOT stands across the doorway, blocking it. The character holds an empty shopping basket, scratches its cap and looks unsure, with a question mark above its head.
Google's guide says verified Google-Agent requests should skip bot challenges, CAPTCHAs included, or an agent sent to shop leaves with nothing. Illustration: AI-generated.

Google added a page to its crawling documentation on 17 November 2026 on how to let Google-Agent through a content delivery network (CDN), web application firewall (WAF) or bot-detection service. Google-Agent is the name, in Google’s words, “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”. Google’s changelog says the guide is “to help publishers that want to enable Google-Agent for their website”. Google does not say which of its products send it.

robots.txt is not the control here. Google lists Google-Agent among its user-triggered fetchers, which “generally ignore robots.txt rules”, so whether it gets through is decided by the CDN or firewall.

How does Google say to verify Google-Agent?

The guide gives three checks, then says to skip bot challenges for requests that pass:

  • The User-Agent header contains Google-Agent.
  • The IP address is in Google’s user-triggered-agents.json list, a separate list from the ranges Googlebot crawls from.
  • If the request is signed, the Signature and Signature-Input headers verify for Signature-Agent: g="https://agent.bot.goog". This is Web Bot Auth, a protocol for signing automated requests that Google says it is experimenting with for this agent.

Some CDNs and bot-detection services verify Google-Agent themselves, and Google says to follow your provider’s steps where they do.

Why can a Google-Agent allow rule still fail?

A security rules panel marked Evaluated in order, listing four numbered rules. Rule 1, Allow verified Google-Agent, is set to Skip and highlighted in yellow. Below it come rule 2, Cloud and data centre ASN block, set to Block; rule 3, Managed bot rules, set to Challenge; and rule 4, AI agent category block, set to Block.
The order Google's guide asks for: the verified Google-Agent rule first, above the three kinds of block it says can otherwise catch the request.

The guide’s troubleshooting section lists where a correct rule is overridden or too narrow:

  • Rule order. The allow rule has to evaluate before broad cloud or data centre ASN blocks, managed bot rules and “category-level AI agent block policies”, the kind of per-category control CDNs now offer for AI crawlers.
  • Challenges as well as blocks. It has to skip CAPTCHAs and JavaScript challenges, including in-page CAPTCHAs on search or form submission.
  • The whole journey. It has to cover on-site search, filtering, detail pages, add-to-cart, booking and the XHR, GraphQL or api.* endpoints behind them, not only the homepage.
  • Shared IPs. Requests from many users share one egress pool, so per-IP rate limits need to exempt the verified ranges.
  • Upkeep. IP rules need syncing with the published list, and load balancers and proxies must pass on the client IP, user agent and signature headers.

For a shop or booking site, where an agent acting for a customer would add to basket or book, the change is one verified allow rule placed above existing bot rules and checked against that list. A site agents only read can add the same rule; the checkout, booking and API items do not apply to it.

Sources

More news