Search Engineering

Search engineering: building discoverability into the system

Discoverability is an architectural property. This is the work of designing it into a platform — how content is structured, discovered, refreshed and prioritised — rather than treating it as an audit finding to be patched afterwards.

The problem

Discoverability is decided by architecture, not by keywords

On a site with hundreds of thousands of URLs, what search engines can reach is determined by decisions made long before anyone considers SEO: how routes are generated, how content is versioned, how pagination works, what the sitemap contains, and whether freshness is expressed anywhere in the system. Those decisions have owners, and the search consequences are usually nobody's job.

Programmatically generated pages that never get indexed
A large estate where a fraction is reachable by any discovery path
No way to express that some content is time-critical and some is permanent
Structured data maintained by hand and therefore drifting from the content
Internal linking that produces no meaningful hierarchy
Discovery behaviour that regresses whenever the platform is refactored

Who this is for

The people who usually bring us this problem

Head of SEO on a platform team

You need discovery treated as a system with owners and tests, not as a set of recommendations that decay after each release.

VP Engineering

Search requirements keep arriving as late-stage requests that destabilise sprints. You want them expressed as architectural requirements the platform can satisfy by construction.

Founder of a content or listings business

Your product's core asset is content and its discoverability, and you need that reflected in how the system is built.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

Content volume stops producing proportional returns

Once a discovery ceiling is reached, publishing more content produces diminishing traffic. The marginal page is competing for crawl attention with every page that came before it.

Refactors silently remove discoverability

When discovery behaviour is emergent rather than specified, a platform refactor can change it without anyone noticing until traffic moves. Without tests, the regression is attributed to the wrong cause.

Engineering effort is spent on re-fixing the same class of defect

A discovery property that is not enforced by the system is one that will regress. Each regression costs a diagnosis, a fix and a recovery period.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Discovery architecture design

Define what classes of content exist, how each is discovered, how each is refreshed, and how each is prioritised — then express those definitions in sitemaps, linking structure and route design.

Freshness and lifecycle modelling

Time-critical content should be discoverable in hours and then age gracefully. That requires the platform to know which content is which, which is a data-model question before it is a sitemap question.

Crawl-budget engineering

Reduce the crawl work the site demands of itself: eliminate wasteful URL patterns, control faceted and parameterised navigation, and ensure crawl attention goes to pages that matter.

Structured data generated from the content model

Schema emitted from the same source of truth as the page content, so it cannot drift and cannot describe something the page does not say.

Internal linking as a system

Linking rules that produce a coherent hierarchy at scale — not hand-curated links, which do not survive a hundred thousand pages.

Regression tests for search behaviour

Automated checks on canonicals, status codes, sitemap coverage and structured-data validity, so a refactor that breaks discoverability fails the build rather than the quarter.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Model the content estate
Enumerate the page types, their cardinality, their lifecycle and their commercial value. Most discovery problems are visible as soon as the estate is described this way.
Specify the discovery path for each class
Each content class gets an explicit answer to: how is it found, how often is it refreshed, and what tells a crawler it exists. Unexpressed requirements are the ones that get lost.
Reduce before adding
Cutting crawl waste is usually higher-leverage than adding discovery signals. A site that asks crawlers to do less with the same budget gets more indexed.
Make it verifiable
Every discovery property we introduce is paired with a check. This is what separates engineering from advice.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Architecture

Content-class and lifecycle model
Discovery and refresh specification per class
Sitemap architecture
Internal linking rules

Implementation

Crawl-waste reduction
Content-model-driven structured data
Freshness signalling
Regression test suite for search properties

Under the hood

Architecture and technology

Discovery signals

Segmented sitemaps by content class and lifecycle
Last-modified and change-frequency semantics that are accurate
Structured data emitted from the content model
Hierarchical internal linking generated from taxonomy

Crawl-waste controls

Faceted and parameterised URL policy
Pagination handling
Canonical logic at template level
Status-code correctness across the estate

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

The problem is a specific fault, not the architecture

See Technical SEO for crawl and indexation diagnosis on an existing site.

The site cannot serve the crawl volume

Crawl-budget problems are sometimes capacity problems. See Infrastructure Engineering.

Questions

Frequently asked

Does this replace technical SEO?

It contains it, and it is the harder half. Technical SEO typically begins with a site that exists and finds what is wrong. Search engineering also asks whether the site's architecture makes discoverability achievable at all, and changes the architecture when it does not.

Can this be done without changing our platform?

Partly. Sitemap architecture, crawl-waste reduction and structured data can often be delivered at the edge or in the application layer without a platform change. The architectural work — lifecycle modelling, linking systems, regression tests — is more durable but does require changing the platform.

How do you handle very large sites?

By sampling and modelling rather than by crawling everything. A site with a million URLs has a handful of page types; the work is in the types, and the per-page detail follows from getting the types right.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work