How Search Engines Work
Last updated
Search engines do three things: they discover web pages, they store and process those pages, and they decide which pages to show when someone searches. Google documents its own version of the sequence as crawling, indexing and serving search results.1 This is the first of the SEO fundamentals to learn, because knowing how each stage works makes it much easier to diagnose why a page might not be ranking.
Stage one: crawling
Crawling is how search engines discover pages. A search engine sends out automated programs called crawlers (Google’s is called Googlebot) that visit web pages, read their content, and follow any links they find. Those links lead to more pages, which contain more links, and so on.
This is why internal linking matters. If a page has no links pointing to it from anywhere on the web or within your own site, crawlers are unlikely to find it. A page that is never crawled cannot be indexed or ranked.
Crawlers do not visit every page on the internet every day. How frequently they return to a site depends on how often it publishes new content, how authoritative it is, and technical signals like the sitemap and crawl budget.
Stage two: indexing
Once a crawler visits a page, the content is sent back to the search engine’s servers to be processed and stored in the index. The index is a vast database of pages and their content, organised so that relevant pages can be retrieved quickly when a query arrives.
During indexing, the search engine analyses the text, images, links, and structured data on a page. It tries to understand what the page is about, who wrote it, how it relates to other pages, and how trustworthy the source appears.
Not every page that gets crawled makes it into the index. Pages can be excluded by directives in the site’s code (a noindex tag, for example), or because the search engine judges the content to be too thin, duplicate, or low quality to be worth storing.
Stage three: ranking and serving
When someone types a query, the search engine retrieves pages from the index and decides what to show. Most SEO writing calls this stage ranking; Google’s own documentation calls it serving, and the broader word has become the more useful one. A ranked list of links is one way of serving a query, and a generated answer citing a few sources is another. Ranking is what happens within the list of links. This is where most SEO work has its effect.
The ranking algorithm weighs hundreds of signals. Google groups its own into five, meaning, relevance, quality, usability and context, and adds that “the weight applied to each factor varies depending on the nature of your query”.2 The four that matter most for a publisher to think about:
- Relevance: does the page content match what the searcher is looking for?
- Authority: do reputable sites link to this page, and does the broader site have a track record of producing useful content?
- Experience: does the page load quickly, work on mobile, and deliver what it promises?
- Context: where the searcher is, what language they use, what device they are on. This is the one you do not control and cannot optimise, and it is why two people entering the same words see different results. Search personalisation covers what it changes and what it means for rank tracking.
The algorithm is not static. Google updates it thousands of times per year, with periodic larger updates that can shift rankings significantly: it reports 4,781 launches in 2023, from over 700,000 quality tests and nearly 17,000 live traffic experiments.3 Chasing algorithm changes is rarely productive. Building pages that genuinely serve a query tends to be stable over time.
How do AI answers fit into crawling, indexing and ranking?
The three stages did not become four. Google still describes Search as crawling, indexing and serving, and its AI features run on that same pipeline rather than beside it: they are “rooted in our core Search ranking and quality systems”, and the technique behind them, retrieval-augmented generation, works “by relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index”.4
A gating rule follows from that. To appear as a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet”.5 The crawling and indexing problems that keep a page out of the blue links keep it out of the generated answer as well, and the controls are shared: nosnippet, data-nosnippet and max-snippet limit what Google shows from a page in AI features too.5 What changes is the shape of the third stage. Query fan-out issues several related searches at once rather than one, and the output is a composed answer citing a few sources rather than a list to hold a position in. The outcome that matters there is whether the page is cited at all.
Other AI systems do not share Google’s index. OpenAI runs its own crawler, and “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links”.6 Perplexity fetches at the moment of the question instead: “when users ask Perplexity a question, it might visit a web page to help provide an accurate answer”.7 Being indexed by Google earns eligibility in Google’s AI answers and nothing else. How AI Search Works sets out those differences.
What this means for SEO
Each stage is a potential point of failure:
- A page that cannot be crawled will never appear in results.
- A page that is crawled but not indexed cannot rank.
- A page that is indexed but does not match the signals the algorithm rewards will rank poorly.
Technical SEO addresses the crawling and indexing stages. On-page SEO addresses the relevance signals the ranking algorithm reads. Off-page SEO addresses the authority signals. The disciplines are distinct, but they address different parts of the same process.
Most of the questions this raises have short answers rather than long ones. The common SEO questions reference collects them in that form.
