Cape Wired SEO Guides · Article 7 of 17

How Does Google Crawl and Index an Ecommerce Website?

A practical guide to discovery, crawling, rendering, canonicalisation and indexation for online stores.

When an ecommerce page does not appear in Google, the problem is often described simply as an indexing issue. That can hide the real cause. Google may never have discovered the URL, crawling may be blocked, the server may return the wrong status code, essential content may only appear after problematic rendering, or Google may have grouped the page with a different canonical URL.

Understanding the stages separately makes technical SEO much easier to diagnose. It also prevents a common mistake: trying to fix every search visibility problem by rewriting the page when the actual issue is technical discovery or processing.

What is the difference between crawling and indexing?

These stages are related but not interchangeable. A sitemap can help Google discover URLs, but it does not guarantee crawling or indexing. Likewise, a page can return HTTP 200 and still be excluded from the index.

The ecommerce crawl and index process, step by step

How should an ecommerce site help Google discover products?

The strongest starting point is a crawlable architecture rather than relying on Google to guess the catalogue. Important products should be reachable through normal links from the broader site structure.

• Link the homepage to important departments or categories.
• Link category pages to relevant subcategories where the catalogue requires them.
• Ensure category and pagination URLs provide crawlable paths to products.
• Use standard <a href> links for navigation wherever possible.
• Link strategically important products from relevant collections, guides or featured areas.
• Include canonical, indexable priority URLs in XML sitemaps.
• Where relevant, use product feeds such as Google Merchant Center alongside the website’s crawlable structure.

What role does the XML sitemap play?

An XML sitemap is a discovery aid. It gives search engines a list of URLs that you consider important, but it does not replace internal linking and it does not force Google to index every URL submitted.

For an ecommerce sitemap, the cleanest approach is to prioritise URLs that are:

• live and returning HTTP 200
• intended for indexing
• canonical or clearly preferred versions
• current rather than redirected or deleted
• valuable enough to appear in organic search

A sitemap full of redirected, noindex, duplicate or error URLs sends mixed signals and makes audit work harder. The sitemap should support the site’s preferred URL structure rather than preserve every URL the platform has ever generated.

How robots.txt and noindex affect ecommerce crawling

robots.txt

robots.txt controls crawler access. It can be useful for reducing crawling of certain low-value URL patterns, but blocking a page does not automatically remove it from Google’s index.

noindex

A noindex robots directive is used when a page can be crawled but should not appear in Google Search. Because Google needs access to read that directive, blocking the same URL in robots.txt can prevent Google from seeing the noindex instruction.

Why this becomes complicated with filters

Faceted navigation can create large combinations of filtered and sorted URLs. Some combinations may have genuine search value; many others are near-duplicates or purely functional. The correct strategy depends on the catalogue and search demand, so broad rules such as “block every filter” or “index every filter” are rarely appropriate.

What happens when ecommerce pages rely on JavaScript?

Google can render JavaScript, but that does not mean JavaScript implementation is irrelevant to SEO. A page may be crawlable while its useful content, links or canonical signals depend on a rendering step.

Technical checks should therefore compare:

• the initial HTML response
• the rendered HTML
• the page shown in Search Console URL Inspection
• the content and links visible to users

For ecommerce stores, pay particular attention to product grids, lazy-loaded products, variant selectors, mobile navigation, app-generated content, reviews and related-product components.

Google’s JavaScript guidance also warns that a noindex directive in the original HTML can cause Google to skip rendering, so relying on JavaScript to remove that noindex later is unsafe.

How pagination and “Load more” can affect crawling

A user may be able to scroll through hundreds of products while a crawler sees only the initial group. This happens when additional products depend on user interaction and are not exposed through distinct crawlable URLs.

For catalogue discovery, pagination should provide crawlable links to additional result pages. Infinite scroll and “Load more” experiences can still be used for users, but the underlying architecture should allow crawlers to reach the same products without needing to click or scroll.

Avoid assuming that page two and later are duplicates of page one. If they expose different products, they form part of the discovery route through the catalogue.

Why duplicate ecommerce URLs complicate indexing

Ecommerce platforms often make the same or very similar content accessible through multiple URLs. Examples include tracking parameters, sort orders, filters, variants, alternative collection paths and legacy URLs.

Google’s canonicalisation systems group duplicate or substantially similar pages and choose a representative canonical. Signals can include redirects, sitemap inclusion and rel="canonical" annotations, but Google can select a different canonical when the overall signals disagree.

The goal is not to eliminate every technically duplicate URL from existence. It is to make the preferred search version clear and keep internal links, sitemaps, canonicals and redirects aligned.

Common reasons ecommerce pages are crawled but not indexed

How to diagnose a product or collection page that is missing from Google

Use a stage-by-stage process rather than jumping straight to content changes.

1. Confirm the exact URL. Make sure you are checking the preferred URL rather than a redirect, parameterised version or outdated path.
2. Check the HTTP status. Confirm the page returns the intended status code and is not producing server errors, soft 404 behaviour or an unexpected redirect.
3. Check robots.txt. Confirm Googlebot is allowed to crawl the URL and required resources.
4. Check robots directives. Look for noindex in HTML and X-Robots-Tag headers.
5. Check the canonical. Compare the user-declared canonical with Google’s selected canonical in URL Inspection.
6. Check discovery. Confirm the URL is linked internally and, where appropriate, included in the sitemap.
7. Inspect the rendered page. Confirm the important text, product information, links and metadata appear after rendering.
8. Review Search Console. Use URL Inspection and the Page Indexing report to understand Google’s latest known state.
9. Compare the page with competing URLs. If technically healthy, determine whether the page is too similar, too weak or simply not competitive for its target searches.

What should you check in Google Search Console?

• URL Inspection for representative products, collections and problem URLs
• Page Indexing report for excluded and indexed URL patterns
• Sitemaps report to confirm submitted files and URL discovery
• Crawl Stats for broader Googlebot activity and response patterns
• Core Web Vitals for real-user performance groups
• Merchant Listings and Product Snippets reports where applicable
• Manual Actions and Security Issues

Search Console reports should be interpreted alongside a crawler and manual checks. The tool shows Google’s view of the site, but it does not replace a complete inventory of URLs or explain every business-specific decision.

A practical ecommerce crawl and index checklist

• Important categories and products are reachable through crawlable internal links.
• Priority URLs return HTTP 200.
• robots.txt does not accidentally block important pages or resources.
• Indexable pages do not contain unintended noindex directives.
• Canonical tags point to the correct preferred URL.
• Internal links point directly to canonical URLs rather than redirects or duplicates.
• XML sitemaps contain current canonical URLs rather than errors, redirects or noindex pages.
• Pagination exposes crawlable links to deeper product sets.
• JavaScript-generated content is visible in rendered HTML.
• Product grids do not depend entirely on user interaction for discovery.
• Discontinued and out-of-stock product handling is intentional.
• Filter, sort and parameter URLs have a defined indexing strategy.
• Search Console exclusions are reviewed by URL pattern, not treated as one generic error.
• Representative templates are tested through URL Inspection.

Does every ecommerce URL need to be indexed?

No. A healthy ecommerce site usually contains many URLs that do not need to appear in organic search. Tracking parameters, internal search results, low-value filter combinations, account areas and other functional URLs may be useful to users without being useful search landing pages.

The objective is not the largest possible index. It is a clean, intentional set of indexable URLs covering the products, categories, content and other pages that can satisfy real search demand.

How crawling and indexing connect to ecommerce SEO strategy

Crawlability and indexation are foundations, not the whole strategy. Once Google can reliably find and process the right pages, those pages still need to match search intent, communicate useful product or category information, earn appropriate internal prominence and compete with the alternatives already appearing in search.

This is why technical SEO should be connected to keyword mapping, collection-page optimisation, product-page optimisation and measurement rather than handled as an isolated engineering task.