How AI Search Works

AI search systems generate answers differently from traditional search engines. Traditional search returns a ranked list of links. Understanding the mechanics behind AI answer generation explains why some content can be cited and some cannot, and separates what the platforms have documented about selection from what is inferred about it.

What are the two layers of AI search?

AI search systems involve two distinct processes that are often confused with each other.

Training is the process by which a large language model (LLM) learns from a large corpus of text: web pages, books, code, and other documents. Training happens once (or periodically) and produces a model with general knowledge baked into its weights. A model trained on web data from a given period has knowledge up to that point, but no further.

Retrieval is what happens at query time. When a user asks a question, most AI search systems do not rely solely on what the model memorised during training. They fetch current content from the web, extract relevant passages, and use those passages to inform the answer the model generates. This fetch-and-use process is called grounding (also known as RAG).

The key implication: training data and retrieval are separate mechanisms. A site does not need to have been included in a model’s training corpus to appear as a cited source in its answers.

How does grounding work?

Grounding, also called Retrieval-Augmented Generation (RAG), is the process of anchoring an AI model’s output to specific, retrieved sources. Google defines the two as the same thing, describing RAG as “a technique (also known as grounding)” that relies on its core Search ranking systems to retrieve relevant, up-to-date pages from its Search index.¹ The model generates its answer by synthesising information from those sources rather than relying solely on training memory.

In practice, this is what happens when Google generates an AI Overview, Perplexity compiles a sourced answer, or ChatGPT Search cites pages in its response. The system:

1. Takes the user’s query
2. Retrieves a set of relevant web pages using a search index
3. Extracts passages from those pages that appear relevant to the query
4. Uses those passages as context for the model to generate a response
5. Cites the source pages in the output

The model’s training knowledge provides language ability and general reasoning. The retrieved passages provide the specific, current content of the answer.

This numbered sequence is a simplification. In practice, grounding is often iterative rather than linear: the system may run follow-up retrievals as it evaluates what it has, and a single query can fan out into multiple parallel sub-queries whose results are combined before the answer is composed (see Query Fan-Out).

Why does crawlability feed into AI citations?

Because AI search surfaces that use grounding retrieve content from the web at query time, they depend on having access to that content. This access depends on crawlability.

If a site blocks AI crawlers in its robots.txt, or if pages are not indexed, those pages cannot be retrieved and therefore cannot be cited. Google states the requirement for its own generative features directly: to be eligible for them, a page must be indexed and eligible to be shown in Google Search with a snippet.¹ AI crawlers (including Googlebot for AI Overviews, PerplexityBot, and OAI-SearchBot for ChatGPT Search) must be able to access and index content for it to enter the retrieval pool.

This means the same technical SEO fundamentals that affect traditional search visibility also affect AI citation potential: pages must be crawlable, indexable, and accessible to the relevant bots. Blocking crawlers eliminates the possibility of being cited entirely, regardless of content quality. JavaScript rendering is a separate barrier for third-party AI crawlers such as OAI-SearchBot and PerplexityBot: as of a December 2024 Vercel analysis, none of the major third-party AI crawlers executed JavaScript, so a client-side-rendered page may return no usable content even when successfully fetched.² Googlebot, which powers AI Overviews, does render JavaScript as part of its standard indexing pipeline, so client-side-rendered content remains accessible to Google’s AI surfaces: Google’s own guidance is that it is able to process content within JavaScript as long as that content is not blocked.¹

Why can new sites appear in AI answers?

A common misconception is that AI citation requires being in a model’s training data. This implies only established sites with historical web presence can appear. This is not accurate for most AI search surfaces.

Because grounding retrieves content at query time, a site published recently can appear in AI-generated answers as soon as its pages are indexed. Nothing in the eligibility rules Google or OpenAI publish requires age: Google’s condition is that a page is indexed and snippet-eligible,¹ and OpenAI’s is that “any public website can appear” if OAI-SearchBot is not blocked.³ Whatever selection each platform applies beyond that, it starts from a pool a new site enters on indexing.

What do retrieval systems look for?

Less than the advice around them implies, and each platform documents a different amount.

Google says its AI features apply “the same foundational SEO best practices” as Search, with “no additional requirements” and no special optimisation; its list of fundamentals includes keeping important content “available in textual form” and structured data that matches the visible page.⁴ On the unit of selection, its AI guide says its systems “are able to understand the nuance of multiple topics on a page and show the relevant piece to users”, and that there is no need to chunk content for them.¹

Microsoft documents the most. Web IQ, the grounding API on Bing’s index, returns passage-level evidence rather than whole pages, evaluated against grounding satisfaction, a metric it defines across completeness, freshness and authority.⁵ Fabrice Canel has confirmed that schema markup helps Microsoft’s LLMs understand content and that “Gen AIs value fresh content in particular”.⁶

OpenAI publishes the least. Its ChatGPT Search guidance says “any public website can appear in ChatGPT search” provided OAI-SearchBot is not blocked, and names no selection criteria.³ The one measured mechanism is the stored 200-character snippet, covered on that page.

What none of the three has said is that headings, question-shaped or otherwise, cited sources, named authors or any formatting pattern affect which passage is chosen. Those are the recurring claims in AI-search advice, and they describe pages that are better for readers; writing to them costs nothing. Treating them as documented selection criteria is inference.

What the three agree on is the negative: there is no separate technique for AI retrieval. The page has to be indexed, readable as text, and good.

The practical difference from traditional SEO

Traditional SEO optimises for a ranked position. The goal is to appear in a results list and earn a click. The metric is rankings, impressions, and click-through rate. Grounding optimises for a different outcome: it asks not which pages a user should visit, but what information an AI system can responsibly use.⁷

AI search optimises for citation. The goal is to be retrieved as a source and have a passage from your content included in a generated answer. The user may see your brand name and a link, but may not click. Citation rate and brand visibility in answers are the relevant metrics, not click-through rate.

This does not require a different body of work. It requires a different measurement framework and a clearer understanding of what “visibility” means when the answer surface, not the results list, is where most users stop.

Frequently asked questions

Do I need to be in GPT’s training data to appear in ChatGPT answers?

Not for ChatGPT Search. ChatGPT’s web search feature retrieves and cites current web content at query time, independently of training data. Content published after the model’s training cutoff can still be cited if it is indexed and accessible to the retrieval system.

Does blocking AI crawlers affect traditional search rankings?

No. Blocking a specific AI crawler (such as GPTBot) in robots.txt does not affect Google’s crawling or indexing of your content. Crawl directives are bot-specific. However, blocking Googlebot would affect both traditional rankings and Google AI Overviews, as both rely on Google’s index.

Is RAG the same thing as GEO?

No. RAG (Retrieval-Augmented Generation) is the technical architecture used by AI systems to retrieve and ground their answers. GEO (Generative Engine Optimisation) is the SEO practice of making content more likely to be retrieved and cited by those systems. RAG describes how AI search works; GEO describes what publishers do in response to it.

Footnotes

1. Optimizing your website for generative AI features on Google Search — Google Search Central ↩ ↩² ↩³ ↩⁴ ↩⁵

2. The rise of the AI crawler — Vercel (December 2024) ↩

3. Publishers and developers FAQ — OpenAI Help Center ↩ ↩²

4. AI features and your website — Google Search Central ↩

5. Microsoft Web IQ ↩

6. Microsoft Bing/Copilot use schema for its LLMs — Search Engine Land, 20 March 2025, quoting Fabrice Canel at SMX Munich. ↩

7. Evolving role of the index: From ranking pages to supporting answers — Bing ↩