Search Engineering

Search engineering for a marketplace

A marketplace's inventory is created and removed by other people, changes daily, and generates URL space in combinations nobody chose. That is a different problem from a site whose content is authored.

The problem

The inventory is not yours and it does not hold still

On a publisher's site the content is authored and stable. On a marketplace it is listed by users, changes price and availability continuously, and disappears when it sells. Every listing has a URL, every filter combination produces more, and every seller and location is a page. The estate is large, volatile and generated — and the indexation model has to account for all three rather than assuming the content behaves like a catalogue.

Filtered and faceted views are indexed in combinations nobody intended
Sold or expired listings remain indexed and send visitors to dead pages
The same item is listed several times and the duplicates compete
Seller and location pages are thin and indexed in their thousands
Listing URLs change when an item is edited or relisted
Crawl is spent on filter combinations rather than on listings
New listings take days to appear in search
Nobody can say how many URLs the marketplace generates

Who this is for

The people who usually bring us this problem

A marketplace with an indexation problem

More pages are indexed than anyone intended and a large share of them should not be.

A marketplace whose crawl is being consumed

Filter and pagination combinations are being fetched at the expense of the inventory.

A marketplace planning for growth

The estate is manageable now and will not be at the inventory volume being planned.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

Faceted navigation generates URL space in combinations, not in pages

Six filters with five options each is not thirty pages, it is a combinatorial space. Some of those combinations are genuinely useful to a searcher and most are not, and the decision about which are crawlable is a routing and linking decision rather than a robots one. This is the largest single difference between a marketplace and a catalogue.

Inventory churn means the estate is different every week

Listings appear and are removed continuously, so a sitemap is stale immediately and a redirect map has to handle URLs that were valid last month. The indexation model has to be expressed as rules about classes of URL rather than as a list of URLs, because a list is out of date before it is finished.

Sold and expired listings are the marketplace's version of a 404

A listing that has sold is not a page to delete, it is a page that has to say something useful — the item is gone, here is what is similar — or be consolidated toward a relevant category. Sending the accumulated authority of thousands of listings to a generic not-found page discards what the marketplace built.

Duplication is structural because users create the content

The same item listed by two sellers, the same seller listing under several categories, an item relisted after expiry. On a marketplace this is unavoidable and the response is consolidation toward a canonical version rather than prevention — which means deciding which of a duplicate set should be the one that ranks.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

URL estate inventory

Every class the marketplace generates — listings, categories, filters, seller pages, location pages, search results, pagination across all of them — enumerated and counted. On a marketplace this is the step that establishes the scale, and it is consistently larger than anyone expects.

Indexation model per class

Which classes are findable and which are not, expressed as rules about URL patterns rather than as a list. The model is what survives inventory churn, and it is what a new filter or page type is checked against when it is added.

Faceted navigation control

Which filter combinations are crawlable, how they are linked, and which are canonicalised or excluded. Decided against what a searcher actually looks for, because a marketplace's filters are frequently its most valuable search surface as well as its largest liability.

Listing lifecycle handling

What happens to a listing's URL when it sells, expires, is withdrawn or is relisted — with the outcome chosen per case: retained with a status, consolidated toward a category, or removed. The lifecycle is where a marketplace accumulates dead URLs and where its authority is either kept or discarded.

Duplicate and near-duplicate resolution

Deciding which of a duplicate set is canonical and consolidating the others toward it, including the same item across categories and the same seller across locations. Structural on a marketplace, so it is handled by rule rather than case by case.

Seller and location page treatment

Whether these are indexable in their own right, and what makes one worth indexing — inventory depth, unique content, genuine demand. A marketplace generates thousands of them and most are thin, which is a decision rather than a default.

Crawl allocation

Directing crawl toward inventory and away from combinations, measured from server logs rather than from a summary. On a marketplace with churn, crawl is a budget that is spent on whatever the site makes easiest to fetch.

Structured data for listings

Describing listings as what they are, including availability and price where the page genuinely carries them, and the state of a listing that has sold. Eligibility-sensitive types are implemented against their actual requirements rather than optimistically.

Sitemaps for a volatile estate

Generated from the inventory source, split by class, and prioritised so crawl reaches listings rather than combinations. A sitemap on a marketplace is a snapshot of something that changed while it was being written.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Count the estate before deciding anything about it
Every URL class and its volume, including the combinations. The proportions decide the priority, and on a marketplace the filter and pagination space usually dwarfs the inventory it exists to serve.
Express the indexation model as rules
Patterns, not lists. Inventory churn makes a list stale immediately, and a rule is what a new filter or page type can be checked against — which is the property that keeps the model correct as the marketplace changes.
Decide the facet question against demand
Some filter combinations are what people search for and are worth being findable; most are not. The decision is made against search demand rather than against tidiness, because a marketplace's facets are frequently its best search surface and excluding all of them discards real visibility.
Design the listing lifecycle, not only the listing
Sold, expired, withdrawn and relisted, each with a defined outcome for its URL. This is where a marketplace accumulates dead pages, and it is where the authority built by thousands of listings is either consolidated or thrown away.
Consolidate duplicates by rule
Deciding which of a duplicate set is canonical, and applying that consistently. Users create the duplication and it cannot be prevented, so the model has to absorb it rather than the content team chasing it.
Measure crawl from the logs
Distribution by class, per period. On a volatile estate a summary is out of date and the log is the only record of what was actually fetched — which is what establishes whether crawl is reaching inventory or combinations.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Audit

Every URL class with its volume, including filter and pagination combinations
The indexation model as rules per class
Where crawl is being spent, measured from server logs
Listing lifecycle states and what currently happens to each URL
Duplicate patterns, and how the canonical choice is currently made
Which classes are thin, and which are worth indexing in their own right

Engineering

Indexation model implemented as routing and linking rules
Faceted navigation controlled at the layer that generates it
Listing lifecycle outcomes implemented per state
Duplicate consolidation applied by rule
Seller and location page treatment decided and implemented
Sitemaps generated from the inventory source, split by class
Structured data for listings, including the sold state

Verification

Crawl distribution by class, before and after
Indexation checked against the model, per class
Listing lifecycle states exercised, including sold and expired
Canonical agreement across duplicates
Coverage, impressions and clicks by class, against a baseline

Under the hood

Architecture and technology

What a marketplace generates

Listing pages, one per item, changing and disappearing continuously
Category and subcategory pages, frequently with their own pagination
Filter and facet combinations — a combinatorial space, not a page count
Search result URLs from the site's own search
Seller and location pages, in the thousands and frequently thin
Pagination across every one of the above
Duplicate listings, created by users and unavoidable

Why a marketplace is not a catalogue

The inventory is authored by users and changes without notice
URL space is generated in combinations rather than in pages
Listings expire, so the estate is different every week
Duplication is structural rather than an editorial error
A sold item is a page with an ending rather than a deletion
Crawl is a budget spent on whatever is easiest to fetch

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Marketplaces

Platform recovery

A large marketplace platform in the United States was experiencing critical downtime from compounding traffic and application faults. Soludome analysed requests through the web application firewall, identified and blocked the DDoS patterns, and repaired a third-party payment flow that allowed links to be generated repeatedly without authentication. Server load fell and platform performance improved after the work.

Around 2020initial recovery; occasional support since

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

Crawl is the specific constraint

What is crawlable and reachable, and what limits each.

Crawl and indexation

The facets need structured markup

Describing listings as what they are, including availability.

Structured data implementation

The sitemap cannot keep up

Sitemap structure and generation for a volatile estate.

XML sitemap optimisation

The sector context

Marketplace platforms and their engineering characteristics.

Marketplaces

Questions

Frequently asked

Should we noindex our filtered pages?

Some of them, and the decision is against search demand rather than tidiness. A marketplace's facets are frequently its most valuable search surface — people search for the combination, not the category — so excluding all of them discards real visibility. What matters is that the combinations are controlled: the ones with genuine demand crawlable and linked, the rest canonicalised or excluded by rule. Left alone, a combinatorial space means crawl is spread across thousands of pages nobody searches for while the inventory waits.

What should happen to sold listings?

Something deliberate, because it is not a deletion. A listing that has sold has accumulated whatever authority it earned, and sending it to a generic not-found page discards that. The options are to keep the URL with a clear sold state and links to similar items, to consolidate it toward a relevant category, or to remove it — chosen per case. On a marketplace this happens thousands of times a week, so it is a rule rather than a case-by-case decision.

How do you deal with duplicate listings?

By consolidating rather than preventing, because users create the duplication and it cannot be stopped. The same item listed twice, the same seller across categories, an item relisted after expiry. The model decides which of a duplicate set is canonical and points the others at it, applied by rule so it holds as the inventory changes. Leaving them competing means neither accumulates enough evidence to rank, which is the common state of a marketplace that has not addressed it.

Do you have marketplace experience?

One published engagement involved restoring stability to a large marketplace platform, and it is described on the case study page rather than restated here — the client is not named, the figures are not published, and the narrative was confirmed by the account owner. What that engagement establishes is that we have worked on a platform at that scale; what this page describes is the search engineering a marketplace specifically needs, which follows from the structure rather than from that client.

How long until new listings appear in search?

It depends on crawl reaching them, which is why crawl allocation is part of the work rather than a consequence of it. On a marketplace with churn, crawl is spent on whatever the site makes easiest to fetch — and a large filter space makes a great deal of it easy. Directing crawl toward inventory is a routing and linking decision, and the measurement is discovery timing per listing class rather than a general impression of how quickly things appear.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work