How AI Search Works
Last updated
AI search systems generate answers differently from traditional search engines. Traditional search returns a ranked list of links. Understanding the mechanics behind AI answer generation explains why some content gets cited, why some does not, and what the signals that drive visibility actually are.
What are the two layers of AI search?
AI search systems involve two distinct processes that are often confused with each other.
Training is the process by which a large language model (LLM) learns from a large corpus of text: web pages, books, code, and other documents. Training happens once (or periodically) and produces a model with general knowledge baked into its weights. A model trained on web data from a given period has knowledge up to that point, but no further.
Retrieval is what happens at query time. When a user asks a question, most AI search systems do not rely solely on what the model memorised during training. They fetch current content from the web, extract relevant passages, and use those passages to inform the answer the model generates. This fetch-and-use process is called grounding (also known as RAG).
The key implication: training data and retrieval are separate mechanisms. A site does not need to have been included in a model’s training corpus to appear as a cited source in its answers.
How does grounding work?
Grounding, also called Retrieval-Augmented Generation (RAG), is the process of anchoring an AI model’s output to specific, retrieved sources. The model generates its answer by synthesising information from those sources rather than relying solely on training memory.
In practice, this is what happens when Google generates an AI Overview, Perplexity compiles a sourced answer, or ChatGPT Search cites pages in its response. The system:
- Takes the user’s query
- Retrieves a set of relevant web pages using a search index
- Extracts passages from those pages that appear relevant to the query
- Uses those passages as context for the model to generate a response
- Cites the source pages in the output
The model’s training knowledge provides language ability and general reasoning. The retrieved passages provide the specific, current content of the answer.
Why does crawlability feed into AI citations?
Because AI search surfaces that use grounding retrieve content from the web at query time, they depend on having access to that content. This access depends on crawlability.
If a site blocks AI crawlers in its robots.txt, or if pages are not indexed, those pages cannot be retrieved and therefore cannot be cited. AI crawlers (including Googlebot for AI Overviews, PerplexityBot, and OAI-SearchBot for ChatGPT Search) must be able to access and index content for it to enter the retrieval pool.
This means the same technical SEO fundamentals that affect traditional search visibility also affect AI citation potential: pages must be crawlable, indexable, and accessible to the relevant bots. Blocking crawlers eliminates the possibility of being cited entirely, regardless of content quality. JavaScript rendering is a separate barrier for third-party AI crawlers such as OAI-SearchBot and PerplexityBot: these do not execute JavaScript, so a client-side-rendered page may return no usable content even when successfully fetched. Googlebot, which powers AI Overviews, does render JavaScript as part of its standard indexing pipeline, so client-side-rendered content remains accessible to Google’s AI surfaces.
Why can new sites appear in AI answers?
A common misconception is that AI citation requires being in a model’s training data. This implies only established sites with historical web presence can appear. This is not accurate for most AI search surfaces.
Because grounding retrieves content at query time, a site published recently can appear in AI-generated answers as soon as its pages are indexed. What matters is whether the content is indexed, whether it is relevant to the query, and whether it meets the quality signals the retrieval system uses to select sources.
A new site with well-structured, accurate, clearly authored content on a specific topic can be cited in AI Overviews or Perplexity answers shortly after its pages are crawled. A long-established site with low-quality or poorly structured content on that same topic may not be.
What do retrieval systems look for?
Retrieval systems select passages from the available indexed content based on relevance and quality signals. The patterns that correlate with selection across major AI search surfaces are consistent:
- Passage-level clarity. Each section of content should answer a specific question and make sense extracted from its surrounding context. The retrieval unit is a passage, not a page.
- Factual accuracy with cited sources. Content that itself links to primary sources (research, official documentation, original data) signals reliability to retrieval systems.
- Clear authorship. Content from identifiable, credible authors with verifiable credentials is preferred over anonymous content.
- Structured formatting. Direct definitions, question-shaped headings, tables, and step-by-step instructions map well to the formats AI systems use to render answers.
These are the same signals that influence traditional search quality assessment. There is no separate set of optimisation techniques for AI retrieval; the shared foundation is high-quality, well-structured content from credible sources.
The practical difference from traditional SEO
Traditional SEO optimises for a ranked position. The goal is to appear in a results list and earn a click. The metric is rankings, impressions, and click-through rate. Grounding optimises for a different outcome: it asks not which pages a user should visit, but what information an AI system can responsibly use.1
AI search optimises for citation. The goal is to be retrieved as a source and have a passage from your content included in a generated answer. The user may see your brand name and a link, but may not click. Citation rate and brand visibility in answers are the relevant metrics, not click-through rate.
This does not require a different body of work. It requires a different measurement framework and a clearer understanding of what “visibility” means when the answer surface, not the results list, is where most users stop.
Frequently asked questions
Do I need to be in GPT’s training data to appear in ChatGPT answers?
Not for ChatGPT Search. ChatGPT’s web search feature retrieves and cites current web content at query time, independently of training data. Content published after the model’s training cutoff can still be cited if it is indexed and accessible to the retrieval system.
Does blocking AI crawlers affect traditional search rankings?
No. Blocking a specific AI crawler (such as GPTBot) in robots.txt does not affect Google’s crawling or indexing of your content. Crawl directives are bot-specific. However, blocking Googlebot would affect both traditional rankings and Google AI Overviews, as both rely on Google’s index.
Is RAG the same thing as GEO?
No. RAG (Retrieval-Augmented Generation) is the technical architecture used by AI systems to retrieve and ground their answers. GEO (Generative Engine Optimisation) is the SEO practice of making content more likely to be retrieved and cited by those systems. RAG describes how AI search works; GEO describes what publishers do in response to it.