How Google Crawls, Renders, and Indexes

Before a page can rank, Google must complete four sequential steps: discover the URL, crawl the page, render it, and index it. These stages happen in order: failure at any one means the next does not occur. Understanding the pipeline helps diagnose exactly where a technical SEO problem is occurring.

Stage 1: Discovery

Google learns about URLs through:

• Internal links from already-crawled pages: the primary discovery mechanism
• External backlinks from other sites
• XML sitemaps submitted via Search Console
• URL inspection: manual submission for individual urgent pages

Pages with no internal links pointing to them are rarely discovered or crawled consistently, even if they appear in a sitemap. Both aid discovery: a sitemap lists the URLs you want crawled, and internal links let Google find pages as it crawls the site.

Stage 2: Crawling

Once a URL is discovered, Googlebot fetches it, requesting the page’s HTML from the server, subject to:

• robots.txt: if the URL is disallowed, Googlebot will not fetch it (but may still index the URL if it has inbound links)
• Crawl budget: the number of URLs Google will crawl on your site in a given period; a constraint Google scopes to sites of 1 million or more pages, or 10,000 or more that change daily¹
• Server response: 5xx errors, timeouts, and slow response times reduce crawl frequency¹

Crawling only retrieves HTML. It does not execute JavaScript. At this stage, Google sees the raw HTML source: what you’d see with curl or “View Source,” not what renders in a browser.

Google’s own estimates put discovery of a new URL at about 20 hours and a recrawl of a known one at about 30 days, with the slowest cases running to weeks, or never.²

See Crawlability and robots.txt for the full detail on how crawl access works.

Stage 3: Rendering

After crawling, Google adds the page to a rendering queue. Googlebot renders pages using a headless version of Chrome: executing JavaScript, applying CSS, and building the full DOM the way a browser would.

Key points:

• Rendering is deferred. It does not happen immediately after crawling. Google’s estimate is seconds to render a page and hours waiting in the queue, with days to weeks in the slowest cases.²
• JavaScript-dependent content is invisible at crawl time. If your page renders content via client-side JavaScript that is not in the initial HTML, that content will not appear in the crawled version. It will only be available after rendering.
• Server-side rendering (SSR) and static generation avoid this delay. If content is in the HTML at request time, Google sees it immediately on crawl. No rendering queue is required.
• Blocked resources affect rendering quality. If Googlebot cannot load your CSS or JavaScript files (blocked via robots.txt or server rules), the rendered page will differ from what users see. This affects mobile usability and content evaluation.

If your content or navigation depends on JavaScript, Google will eventually see it. This may take hours or days after crawling, and the process may be inconsistent. For content that matters to rankings, server-side rendering is safer.

See JavaScript SEO for the full detail on rendering and its implications.

Stage 4: Indexing

After rendering, Google evaluates whether to add the page to its index. Indexing is not guaranteed even for crawled, rendered pages. What decides it includes:

• noindex directive. A <meta name="robots" content="noindex"> tag or X-Robots-Tag: noindex HTTP header prevents indexing. The page must be crawlable for Google to see this directive. A page blocked in robots.txt cannot be noindexed this way.
• Canonical tags. If the page has a canonical pointing to a different URL, Google will consolidate signals to the canonical and may not index this version.
• Content quality. Thin, duplicate, or low-quality content may be crawled and rendered but excluded from the index at Google’s discretion.
• HTTP status codes. 404 and 410 pages are not indexed, and Google removes the URL from the index if it was previously indexed. A 503 buys time rather than immunity: Google decreases the crawl rate, and its “indexing pipeline removes from the index URLs that persistently return a server error”.³

Once those processes run, Google’s estimate for indexing a page end to end is about 1.5 hours. Where Google judges the page low quality, the slowest case is months, or never.²

See Indexing for the full detail on how indexing decisions are made.

Does this pipeline apply to AI crawlers?

Google’s own AI features do run on it. To appear as a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet”, so all four stages gate the AI answer exactly as they gate the blue links.⁴ Gemini uses Googlebot’s infrastructure and renders JavaScript accordingly.⁵

Other companies’ crawlers are a different matter, and rendering is where this model stops describing them. Vercel and MERJ found in December 2024 that the crawlers run by OpenAI, Anthropic, Meta, ByteDance and Perplexity did not render JavaScript, while Applebot did, through a browser-based crawler.⁵ For most of them, stage three’s reassurance that Google will eventually see client-side content does not transfer. AI Crawler User-Agents covers what each one can actually read.

Diagnosing problems by stage

Symptom	Likely stage	How to confirm
Page not discovered	Discovery	No URL in Search Console; no internal links
Page blocked from crawling	Crawling	robots.txt disallow; URL Inspection shows “blocked by robots.txt”
JS content missing	Rendering	View Source vs rendered DOM differ; URL Inspection (Live Test) rendered HTML lacks the content
Page crawled, not indexed	Indexing	URL Inspection shows “crawled - currently not indexed”
Page indexed but not ranking	Post-indexing	Ranking/quality signals, not a pipeline problem

Common misconceptions

Crawling and indexing are not the same thing. A page can be crawled every day and never indexed. Crawling is access. Indexing is a separate editorial decision.

Indexed does not mean ranked. The index contains billions of pages. Being indexed means you are eligible to rank. It does not guarantee any specific position.

robots.txt does not prevent indexing. A disallowed URL can appear in search results if it has inbound links. Google will show the URL without a description. To prevent indexing, use noindex. But the page must be crawlable for Google to read the tag.

Sitemaps do not override crawl signals. Including a URL in your sitemap does not force indexing. It signals that you consider the URL important; Google makes the indexing decision independently.

Footnotes

1. Optimize your crawl budget — Google Search Central ↩ ↩²

2. Google Search Central Live Deep Dive, Barcelona: Day 3 recap — ROAST. An attendee’s recap of Gary Illyes’s slides, which Google has not published; Estela Franco’s photos of the slides show the same figures. ↩ ↩² ↩³

3. How HTTP status codes affect Google’s crawlers — Google Crawling Infrastructure ↩

4. AI features and your website — Google Search Central ↩

5. The rise of the AI crawler — Vercel ↩ ↩²