Ecommerce
An ecommerce catalogue generates more URLs than it intends to. Facets, variants, sorting and filters multiply a few thousand products into a URL space large enough to consume any crawl budget, and deciding which of them should exist is the central problem.
What makes engineering for ecommerce different
These are the conditions that change the shape of the problem. They are why a generic approach underperforms in this sector.
The URL space is generated, not authored
A catalogue of a few thousand products can produce millions of reachable URLs through facet combinations alone. Nobody chose those URLs; they are a consequence of a filtering system built for users. Deciding which combinations should be crawlable is a product and engineering decision, and it is the one that determines whether the catalogue ranks.
Availability changes are frequent and consequential
Products go out of stock, are discontinued, return, and are replaced by successors. Each state change is an indexation decision — keep, redirect, or remove — and catalogues usually handle them with a default rather than a policy, which is how seasonal stock quietly removes pages that carried traffic all year.
Revenue is concentrated in a small share of the catalogue
Most catalogues follow a long tail where a minority of products carry most revenue and a much larger set carries occasional long-tail search demand. Those two groups need different treatment, and applying one strategy to both is the common error.
Demand arrives in dated peaks
Seasonal and promotional traffic is concentrated around known dates, which makes capacity planning a calendar problem. The failure mode is not gradual degradation but a site that falls over on the day it matters most.
Every template change is a catalogue-wide change
Product, category and listing templates render the entire estate. A metadata or rendering fault in one of them affects every page it generates at once, and is usually detected through a traffic report rather than through monitoring.
What usually goes wrong
Crawl attention consumed by facet combinations
Filter and sort permutations absorbing the crawl an entire catalogue needs, leaving priority products discovered late and recrawled rarely.
Out-of-stock and discontinued handling that erases accumulated value
Availability changes accumulating as removed or redirecting URLs with no policy behind the choice, removing pages that ranked for durable queries.
Variant URLs competing with each other
Size, colour and configuration variants indexed separately, splitting signals across what is one purchasable product.
Failing at the seasonal peak
Availability and response-time collapse during demand concentrations the infrastructure was not provisioned for.
How we work in this sector
The work is grouped by objective rather than by service, because in this sector the objectives interact.
Decide the crawlable URL space explicitly
- Facet combinations classified as indexable, crawlable but excluded, or blocked
- Canonical resolution for sort and tracking parameters
- Facet values with genuine search demand promoted to real category pages
Make availability a policy, not a default
- Per-state handling for out-of-stock, discontinued and superseded products
- Retention of pages with durable demand rather than blanket removal
- Successor redirect mapping for replaced products
Resolve variants to the purchasable unit
- Canonical consolidation where variants share intent
- Separate pages only where demand is genuinely distinct
- Structured data reflecting actual availability and pricing
Engineer for dated peaks
- Capacity modelled against the promotional calendar
- Caching and asset delivery reviewed before the peak, not during it
- Pre-peak verification rather than a post-incident review
Frequently asked
What work is most relevant to ecommerce?
Our closest work spans large catalogues, crawl architecture, performance and high-traffic platforms. We can map those patterns to your catalogue, facets and promotional traffic during discovery.
Should we block all faceted URLs?
Rarely, and blanket blocking causes its own problems. Facets with real search demand — a brand filter, a material or size with meaningful volume — are frequently better promoted into genuine category pages than hidden. The work is classifying combinations into those worth indexing, those worth allowing but not encouraging, and those worth blocking, and that classification should be revisited as demand changes.
What should we do with out-of-stock products?
It depends on whether the product is coming back, and the default should reflect that rather than treating every absence as an exit. Temporarily unavailable products generally keep their page and say so; discontinued ones with no successor should return a clear response rather than redirecting to a category; replaced products should redirect to the successor. What causes the damage is a single default applied to all three.
How far ahead do we need to prepare for peak?
Long enough that the work is a planned change rather than an emergency one — realistically several weeks, because capacity changes, caching strategy and asset delivery all need verifying under load rather than at the moment demand arrives. Peak failures are usually traceable to something that was known and deferred, not to something unpredictable.
Related capabilities and work
Tell us what your ecommerce platform is doing
The symptoms are usually more informative than the diagnosis. We will tell you what we think is happening.