--- title: "How Google Crawls, Renders, and Indexes" description: "How Google processes a web page from discovery to indexing: the four stages of the crawl-render-index pipeline and where SEO problems occur at each stage." tldr: "Google processes pages in four sequential stages: discovery, crawling, rendering, and indexing. Failure at any stage stops the process. Crawling fetches the raw HTML; rendering processes JavaScript to build the full DOM; indexing adds the rendered content to Google's search index. A page can be crawled but not rendered (JS blocked), rendered but not indexed (noindex, thin content), or indexed but not ranking. Most technical SEO problems map to one of these stages. The pipeline is Google's own, and it does not describe most other companies' AI crawlers, which Vercel and MERJ found in December 2024 did not render JavaScript." publishDate: 2026-04-27 updatedDate: 2026-10-05 author: "Liam Hayward" --- Before a page can rank, Google must complete four sequential steps: discover the URL, crawl the page, render it, and index it. These stages happen in order: failure at any one means the next does not occur. Understanding the pipeline helps diagnose exactly where a technical SEO problem is occurring. ## Stage 1: Discovery Google learns about URLs through: - **Internal links** from already-crawled pages: the primary discovery mechanism - **External backlinks** from other sites - **XML sitemaps** submitted via Search Console - **URL inspection**: manual submission for individual urgent pages Pages with no internal links pointing to them are rarely discovered or crawled consistently, even if they appear in a sitemap. Both aid discovery: a sitemap lists the URLs you want crawled, and internal links let Google find pages as it crawls the site. ## Stage 2: Crawling Once a URL is discovered, Googlebot fetches it, requesting the page's HTML from the server, subject to: - **robots.txt**: if the URL is disallowed, Googlebot will not fetch it (but may still index the URL if it has inbound links) - **Crawl budget**: the number of URLs Google will crawl on your site in a given period; a constraint Google scopes to sites of 1 million or more pages, or 10,000 or more that change daily[^5] - **Server response**: 5xx errors, timeouts, and slow response times reduce crawl frequency[^5] Crawling only retrieves HTML. It does not execute JavaScript. At this stage, Google sees the raw HTML source: what you'd see with `curl` or "View Source," not what renders in a browser. Google's own estimates put discovery of a new URL at about 20 hours and a recrawl of a known one at about 30 days, with the slowest cases running to weeks, or never.[^4]
Timeline of Google's crawl times on an axis from seconds to months and past one year. Discovery of a new URL is typically about 20 hours, at worst weeks to never; refresh of a known URL about 30 days, weeks to never; sitemap processing about 24 hours, up to 14 days or never; robots.txt update about 24 hours, 25 hours at most; crawl capacity update 4 hours to 1 to 2 weeks, dropping within seconds in back-off; crawl demand update about 20 hours, weeks to months.
Google's typical and slowest times for crawling. Redrawn from a slide Gary Illyes, an analyst on the Google Search team, showed at Search Central Live Deep Dive Europe on 2 October 2026. Google describes the times as estimations based on internal analysis; positions within each time band are approximate. Key figures as text, with what to do about each: Has the page had long enough?
See [Crawlability and robots.txt](/technical-seo/crawlability-and-robots-txt/) for the full detail on how crawl access works. ## Stage 3: Rendering After crawling, Google adds the page to a rendering queue. Googlebot renders pages using a headless version of Chrome: executing JavaScript, applying CSS, and building the full DOM the way a browser would. Key points: - **Rendering is deferred.** It does not happen immediately after crawling. Google's estimate is seconds to render a page and hours waiting in the queue, with days to weeks in the slowest cases.[^4] - **JavaScript-dependent content is invisible at crawl time.** If your page renders content via client-side JavaScript that is not in the initial HTML, that content will not appear in the crawled version. It will only be available after rendering. - **Server-side rendering (SSR) and static generation avoid this delay.** If content is in the HTML at request time, Google sees it immediately on crawl. No rendering queue is required. - **Blocked resources affect rendering quality.** If Googlebot cannot load your CSS or JavaScript files (blocked via robots.txt or server rules), the rendered page will differ from what users see. This affects mobile usability and content evaluation. If your content or navigation depends on JavaScript, Google will eventually see it. This may take hours or days after crawling, and the process may be inconsistent. For content that matters to rankings, server-side rendering is safer. See [JavaScript SEO](/technical-seo/javascript-seo/) for the full detail on rendering and its implications. ## Stage 4: Indexing After rendering, Google evaluates whether to add the page to its index. Indexing is not guaranteed even for crawled, rendered pages. What decides it includes: - **noindex directive.** A `` tag or `X-Robots-Tag: noindex` HTTP header prevents indexing. The page must be crawlable for Google to see this directive. A page blocked in robots.txt cannot be noindexed this way. - **Canonical tags.** If the page has a canonical pointing to a different URL, Google will consolidate signals to the canonical and may not index this version. - **Content quality.** Thin, duplicate, or low-quality content may be crawled and rendered but excluded from the index at Google's discretion. - **HTTP status codes.** 404 and 410 pages are not indexed, and Google removes the URL from the index if it was previously indexed. A 503 buys time rather than immunity: Google decreases the crawl rate, and its "indexing pipeline removes from the index URLs that persistently return a server error".[^3] Once those processes run, Google's estimate for indexing a page end to end is about 1.5 hours. Where Google judges the page low quality, the slowest case is months, or never.[^4] See [Indexing](/technical-seo/indexing/) for the full detail on how indexing decisions are made. ## Does this pipeline apply to AI crawlers? Google's own AI features do run on it. To appear as a supporting link in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet", so all four stages gate the AI answer exactly as they gate the blue links.[^1] Gemini uses Googlebot's infrastructure and renders JavaScript accordingly.[^2] Other companies' crawlers are a different matter, and rendering is where this model stops describing them. Vercel and MERJ found in December 2024 that the crawlers run by OpenAI, Anthropic, Meta, ByteDance and Perplexity did not render JavaScript, while Applebot did, through a browser-based crawler.[^2] For most of them, stage three's reassurance that Google will eventually see client-side content does not transfer. AI Crawler User-Agents covers what each one can actually read. ## Diagnosing problems by stage | Symptom | Likely stage | How to confirm | |---------|-------------|----------------| | Page not discovered | Discovery | No URL in Search Console; no internal links | | Page blocked from crawling | Crawling | robots.txt disallow; URL Inspection shows "blocked by robots.txt" | | JS content missing | Rendering | View Source vs rendered DOM differ; URL Inspection (Live Test) rendered HTML lacks the content | | Page crawled, not indexed | Indexing | URL Inspection shows "crawled - currently not indexed" | | Page indexed but not ranking | Post-indexing | Ranking/quality signals, not a pipeline problem | ## Common misconceptions **Crawling and indexing are not the same thing.** A page can be crawled every day and never indexed. Crawling is access. Indexing is a separate editorial decision. **Indexed does not mean ranked.** The index contains billions of pages. Being indexed means you are eligible to rank. It does not guarantee any specific position. **robots.txt does not prevent indexing.** A disallowed URL can appear in search results if it has inbound links. Google will show the URL without a description. To prevent indexing, use noindex. But the page must be crawlable for Google to read the tag. **Sitemaps do not override crawl signals.** Including a URL in your sitemap does not force indexing. It signals that you consider the URL important; Google makes the indexing decision independently. [^1]: [AI features and your website — Google Search Central](https://developers.google.com/search/docs/appearance/ai-features) [^2]: [The rise of the AI crawler — Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler) [^3]: [How HTTP status codes affect Google's crawlers — Google Crawling Infrastructure](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes) [^4]: [Google Search Central Live Deep Dive, Barcelona: Day 3 recap — ROAST](https://weareroast.com/news/google-search-central-live-deep-dive-barcelona-day-3-recap/#:~:text=How%20long%20does%20it%20take%20to%E2%80%A6%3F). An attendee's recap of Gary Illyes's slides, which Google has not published; Estela Franco's [photos of the slides](https://www.linkedin.com/posts/estelafranco_how-long-does-it-take-for-google-to-crawl-ugcPost-7512112385164300288-EuFA/) show the same figures. [^5]: [Optimize your crawl budget — Google Search Central](https://developers.google.com/crawling/docs/crawl-budget)