---
title: "How AI Search Works"
description: "How AI-powered search engines retrieve and cite content: the difference between training data and real-time retrieval, grounding, and why crawlability matters for AI visibility."
tldr: "AI search engines operate in two distinct layers: training (what the model learned during development) and retrieval (what it fetches at query time to answer a specific question). Most AI search surfaces (Google AI Overviews, Perplexity, ChatGPT Search) use real-time retrieval to ground their answers in current web content. This means a site does not need to be in a model's training data to be cited in its answers. It needs to be crawlable, high quality, and relevant to the query being asked."
publishDate: 2026-05-05
updatedDate: 2026-09-28
author: "Liam Hayward"
---
AI search answers a question in its own words instead of returning a list of links, drawing on two sources: what its model learned in training, and pages it retrieves from a search index when the question is asked. For a site hoping to be cited, retrieval is the more direct route: the surfaces covered here (Google's AI features, ChatGPT Search, Perplexity and Copilot) cite pages they fetch at query time,[^3] so a page does not need to be in a model's training data to appear in its answers. The sections below separate what the platforms document about how they choose those pages from what is inferred.

## What are the two layers of AI search?

AI search systems involve two distinct processes that are often confused with each other.

**Training** is the process by which a large language model (LLM) learns from a large corpus of text: web pages, books, code, and other documents. Training happens once (or periodically) and produces a model with general knowledge baked into its weights. A model trained on web data from a given period has knowledge up to that point, but no further. How much it knows about any one subject tracks how often that subject appeared: a 2023 study of pre-training data found a model's ability to answer a fact-based question "relates to how many documents associated with that question were seen during pre-training", a relationship they showed to be causal and to hold across model sizes, and one that retrieval "can reduce the dependence on".[^9] Being in the training data means being mentioned frequently, across many independent documents, not once.

**Retrieval** is what happens at query time. When a user asks a question, the AI search surfaces covered on this page (Google's AI features, ChatGPT Search, Perplexity and Copilot) do not rely solely on what the model memorised during training. They retrieve current pages from a search index, pick out the information relevant to the question, and use it to inform the answer the model generates. This is called grounding, or retrieval-augmented generation (RAG).[^3]

The key implication: training data and retrieval are separate mechanisms. A site does not need to have been included in a model's training corpus to appear as a cited source in its answers.

## What does a model learn in training?

Not facts, in the sense of a record it can look up. Training adjusts a model so that, given some text, the next word it predicts matches what its corpus contained: it is, in one widely read explanation, always "trying to produce a 'reasonable continuation'" of the text so far, where reasonable means "what one might expect someone to write after seeing what people have written on billions of webpages".[^10] What the weights end up holding are associations: how strongly one word, name or idea pulls towards another. Anthropic's interpretability work found these directly, as "features corresponding to concepts", one of which "fires on a range of model inputs, from English mentions of the name of the bridge to discussions in Japanese, Chinese, Greek, Vietnamese, Russian", and measured "a kind of 'distance' between features" so that related concepts sit close together.[^11]

For a brand, that has two consequences. The association has to be built from many documents, which is the frequency finding above. And it is built from what those documents say: a brand described the same way each time, the same name, the same category, the same two or three attributes, strengthens one association, while a brand described differently in each place spreads it across several weaker ones. That second consequence follows from how the objective works rather than from a measurement of brands.

## How does grounding work?

Grounding, also called Retrieval-Augmented Generation (RAG), is the process of anchoring an AI model's output to specific, retrieved sources. Google defines the two as the same thing, describing RAG as "a technique (also known as grounding)" that relies on its core Search ranking systems to retrieve relevant, up-to-date pages from its Search index.[^3] The model generates its answer by synthesising information from those sources rather than relying solely on training memory.

In practice, this is what happens when Google generates an AI Overview, Perplexity compiles a sourced answer, or ChatGPT Search cites pages in its response. The system:

1. Takes the user's query
2. Retrieves a set of relevant web pages using a search index
3. Extracts passages from those pages that appear relevant to the query
4. Uses those passages as context for the model to generate a response
5. Cites the source pages in the output

The model's training knowledge provides language ability and general reasoning. The retrieved passages provide the specific, current content of the answer.

This numbered sequence is a simplification. In practice, grounding is often iterative rather than linear: the system may run follow-up retrievals as it evaluates what it has, and a single query can fan out into multiple parallel sub-queries whose results are combined before the answer is composed (see [Query Fan-Out](/ai-search/query-fan-out/)).

## Why does crawlability feed into AI citations?

Because AI search surfaces that use grounding retrieve content from the web at query time, they depend on having access to that content. This access depends on crawlability.

If pages are not indexed, or a platform's search crawler is blocked in robots.txt, that platform has no copy of them to retrieve. Google states the requirement for its own generative features directly: to be eligible for them, a page must be indexed and eligible to be shown in Google Search with a snippet.[^3] Each platform names the crawler that has to reach the content: Googlebot for Google's AI features,[^4] OAI-SearchBot for ChatGPT Search,[^6] and PerplexityBot for Perplexity's search results.[^12]

This means the same technical SEO fundamentals that affect traditional search visibility also affect AI citation potential: pages must be crawlable, indexable, and accessible to the relevant bots. Blocking a platform's search crawler takes a page's content out of that platform's retrieval pool, with two documented exceptions. OpenAI's publishers FAQ says that if it finds a disallowed URL through a third-party search provider or other pages, it may still show the link and page title, but names only ChatGPT Atlas, a browser OpenAI shut down on 9 August 2026, and does not say whether the same applies elsewhere in ChatGPT.[^6][^15] Perplexity says Perplexity-User, which fetches pages when a user asks a question, "generally ignores robots.txt rules".[^12] JavaScript rendering is a separate barrier for third-party AI crawlers such as OAI-SearchBot and PerplexityBot: as of a December 2024 Vercel analysis, none of the major third-party AI crawlers executed JavaScript, so a client-side-rendered page may return no usable content even when successfully fetched.[^2] Googlebot, which powers AI Overviews, does render JavaScript as part of its standard indexing pipeline, so client-side-rendered content remains accessible to Google's AI surfaces: Google's own guidance is that it is able to process content within JavaScript as long as that content is not blocked.[^3]

## Why can new sites appear in AI answers?

A common misconception is that AI citation requires being in a model's training data. This implies only established sites with historical web presence can appear. This is not accurate for most AI search surfaces.

Because grounding retrieves content at query time, a site published recently can appear in AI-generated answers as soon as its pages are indexed. Nothing in the eligibility rules Google or OpenAI publish requires age: Google's condition is that a page is indexed and snippet-eligible,[^3] and OpenAI's is that "any public website can appear" if OAI-SearchBot is not blocked.[^6] Whatever selection each platform applies beyond that, it starts from a pool a new site enters on indexing.

## What do retrieval systems look for?

Less than the advice around them implies, and each platform documents a different amount.

**Google** says its AI features apply "the same foundational SEO best practices" as Search, with "no additional requirements" and no special optimisation; its list of fundamentals includes keeping important content "available in textual form" and structured data that matches the visible page.[^4] On the unit of selection, its AI guide says its systems "are able to understand the nuance of multiple topics on a page and show the relevant piece to users", and that there is no need to chunk content for them.[^3]

**Microsoft** documents the most. Web IQ, the grounding API on Bing's index, uses "passage-level retrieval",[^5] and Microsoft grades its results with grounding satisfaction, a metric it defines across "completeness, freshness, and authority".[^16] Microsoft says structured content, with schema-marked pages as its example, helps its AI answers interpret and summarise a page,[^8] and Microsoft's Fabrice Canel has said that "Gen AIs value fresh content in particular".[^7] It is also the only one of the four to give formatting advice for its AI answers. In an October 2025 post on the Microsoft Advertising blog, Krishna Madhavan, a principal product manager at Microsoft Bing, wrote that "Assistants like Copilot break content down, a process called parsing, into smaller, structured pieces", that headings "act like chapter titles that define clear content slices", and recommended question-and-answer pairs, lists and tables.[^13]

**Perplexity** documents its index rather than any advice for site owners. A September 2025 research post says pages are split into "self-contained spans, each of which can be individually retrieved and ranked at query time", with "clearly non-responsive or stale content" filtered out before ranking; it gives no signal weights and says nothing about headings.[^17]

**OpenAI** publishes the least. Its [ChatGPT Search](/ai-search/chatgpt-search-optimisation/) guidance says "any public website can appear in ChatGPT search" provided OAI-SearchBot is not blocked, and names no selection criteria.[^6] The one measured mechanism is the stored 200-character snippet, covered on that page.

Google, OpenAI and Perplexity have not said that headings, question-shaped or otherwise, cited sources, named authors or any formatting pattern affect which passage is chosen. Microsoft's post is advice with no data behind its formatting recommendations. The patterns describe pages that are better for readers; writing to them costs nothing. Treating them as documented selection criteria for Google, ChatGPT or Perplexity is inference.

Google's position is that there is no separate technique: optimising for its AI features "is optimizing for the search experience, and thus still SEO".[^3] Microsoft calls SEO fundamentals "just the starting point" and adds its formatting advice on top.[^13] OpenAI asks only that its crawler is not blocked.[^6] Perplexity asks site owners to allow PerplexityBot and documents how its index works, not how to write for it. What all four share is the floor: a page their crawler can reach and index.

## The practical difference from traditional SEO

Traditional SEO optimises for a ranked position. The goal is to appear in a results list and earn a click. The metric is rankings, impressions, and click-through rate. Microsoft's Bing team puts the difference as two questions: traditional search asks "which pages should a user visit?", while grounding asks "what information can an AI system responsibly use to construct a response?"[^1]

AI search optimises for citation. The goal is to be retrieved as a source and have a passage from your content included in a generated answer. The user may see your brand name and a link, but may not click. Citation rate and brand visibility in answers are the relevant metrics, not click-through rate.

This does not require a different body of work. It requires a different measurement framework and a clearer understanding of what "visibility" means when the answer surface, not the results list, is where most users stop.

## Frequently asked questions

### Do I need to be in GPT's training data to appear in ChatGPT answers?

Not for ChatGPT Search. ChatGPT's web search feature retrieves and cites current web content at query time, independently of training data. Content published after the model's training cutoff can still be cited if it is indexed and accessible to the retrieval system: OpenAI's condition is that "any public website can appear in ChatGPT search" if OAI-SearchBot is not blocked.[^6]

### Does blocking AI crawlers affect traditional search rankings?

No. Blocking a specific AI crawler (such as GPTBot) in robots.txt does not affect Google's crawling or indexing of your content. Crawl directives are bot-specific. However, blocking Googlebot would affect both traditional rankings and Google AI Overviews, as both rely on Google's index.[^3] To stay in Search but out of AI Overviews and AI Mode, Google's control is a Search Console setting rather than a robots.txt rule: excluding a site removes its links and content from Search's generative AI features, is not "used as a ranking or inclusion signal affecting other parts of Search", and has been available to all websites since 31 August 2026.[^14]

### Is RAG the same thing as GEO?

No. RAG (Retrieval-Augmented Generation) is the technical architecture used by AI systems to retrieve and ground their answers. GEO (Generative Engine Optimisation) is the SEO practice of making content more likely to be retrieved and cited by those systems. RAG describes how AI search works; GEO describes what publishers do in response to it.

[^1]: [Evolving role of the index: From ranking pages to supporting answers — Bing](https://blogs.bing.com/search/May-2026/Evolving-role-of-the-index-From-ranking-pages-to-supporting-answers)
[^2]: [The rise of the AI crawler — Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) (December 2024)
[^3]: [Optimizing your website for generative AI features on Google Search — Google Search Central](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
[^4]: [AI features and your website — Google Search Central](https://developers.google.com/search/docs/appearance/ai-features)
[^5]: [Microsoft Web IQ](https://webiq.microsoft.ai)
[^6]: [Publishers and developers FAQ — OpenAI Help Center](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq)
[^7]: [Microsoft Bing/Copilot use schema for its LLMs — Search Engine Land](https://searchengineland.com/microsoft-bing-copilot-use-schema-for-its-llms-453455), 20 March 2025, relaying David Mihm's LinkedIn account of Fabrice Canel's SMX Munich talk; the freshness sentence is the only wording attributed to Canel.
[^8]: [How AI Search Is Changing the Way Conversions are Measured — Bing Webmaster Blog](https://blogs.bing.com/webmaster/November-2025/How-AI-Search-Is-Changing%E2%80%AFthe%E2%80%AFWay%E2%80%AFConversions%E2%80%AFare-Measured), Fabrice Canel and Krishna Madhavan, 20 November 2025: "Structured content, like schema-marked product pages, FAQs, and comparison tables, helps AI systems interpret and summarize your content more effectively."
[^9]: [Large Language Models Struggle to Learn Long-Tail Knowledge — Kandpal, Deng, Roberts, Wallace and Raffel, arXiv 2211.08411](https://arxiv.org/abs/2211.08411). ICML 2023; UNC Chapel Hill and UC Berkeley.
[^10]: [What Is ChatGPT Doing … and Why Does It Work? — Stephen Wolfram](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/), February 2023.
[^11]: [Mapping the Mind of a Large Language Model — Anthropic](https://www.anthropic.com/research/mapping-mind-language-model), May 2024.
[^12]: [Perplexity Crawlers — Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
[^13]: [Optimizing Your Content for Inclusion in AI Search Answers — Microsoft Advertising](https://about.ads.microsoft.com/en/blog/post/october-2025/optimizing-your-content-for-inclusion-in-ai-search-answers), Krishna Madhavan, Principal Product Manager, Microsoft Bing, 8 October 2025.
[^14]: [Manage how your site appears in Search generative AI features — Google Search Help](https://support.google.com/webmasters/answer/16908024)
[^15]: [Evolving Atlas into ChatGPT for browser-based agentic work, archived 25 July 2026 — OpenAI Help Center, Wayback Machine](https://web.archive.org/web/20260725115420/https://help.openai.com/en/articles/20001371)
[^16]: [Announcing Microsoft Web IQ — Bing Search Blog](https://blogs.bing.com/search/June-2026/Announcing-Microsoft-Web-IQ)
[^17]: [Architecting and evaluating an AI-first search API — Perplexity Research](https://research.perplexity.ai/articles/architecting-and-evaluating-an-ai-first-search-api), 25 September 2025