Search Engineering

Crawl and indexation engineering

Pages that are crawled but not indexed are the most commonly misdiagnosed problem in technical SEO. It is not one problem. It is at least four, they present identically, and they have opposite remedies.

The problem

"Not indexed" is a symptom, not a diagnosis

A site can be under-indexed because search engines are choosing not to include pages, because they are unable to crawl them, because the pages are not worth including, or because crawl attention is being spent somewhere it should not be. Each of those has a different fix, and two of them are made worse by the usual response of submitting more URLs. Separating them requires server-side evidence rather than Search Console reports alone.

Crawl rate fell and never recovered, with no content change to explain it
A whole page type is excluded at once — which means a template, not the pages
Crawled-but-not-indexed counts that stay high however much content is improved
Recurring 5xx responses, or unplanned downtime at predictable times
Crawl attention spent on URLs that can never rank — filters, parameters, duplicates
Time-sensitive content discovered days late, long after its commercial life
Impressions fell without any corresponding change in content or competition

Who this is for

The people who usually bring us this problem

Head of SEO / SEO Director

You can see the problem in Search Console and cannot get past it. You need evidence that identifies which of the four causes it actually is, and something a development team will act on.

CTO / VP Engineering

You are being asked to change crawl behaviour and want to know what is genuinely broken in the serving path before anyone touches it.

Platform / Infrastructure Lead

You suspect the application is serving different responses to crawlers than to users, and need it established rather than assumed.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

Submitting more URLs makes it worse

If crawl attention is being wasted on unusable URLs, growing the sitemap dilutes it further. The instinct that follows a low index count is frequently the thing suppressing it.

Server instability is a crawl-budget decision

Every 5xx response tells a search engine the site is unreliable, and it responds by crawling less. A reliability complaint and a visibility decline are often the same incident.

Fresh content loses its window

For news, listings and anything time-sensitive, discovery speed is the entire value. Content found a week late has usually already lost the traffic it existed to capture.

A template defect looks like a content problem

When one page type collapses across the whole estate, the symptom reads as declining content quality. Teams respond by writing more content, which cannot address it.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Server log analysis

What is actually being crawled, by which agent, at what rate, and what response it receives. Search Console reports intent; logs report behaviour. The two disagree more often than expected.

Crawl-rate diagnosis and recovery

Establishing why a crawl rate fell — error response rates, latency, server capacity, robots directives, internal link structure — and verifying the recovery rather than assuming it.

Index triage

Separating the four causes of an index gap so each gets the remedy it needs: exclusion by choice, exclusion by error, exclusion by quality, or simple crawl starvation.

Sitemap architecture and segmentation

Splitting sitemaps by content class and lifetime so crawl attention can be directed. Time-critical content and permanent pages have different discovery requirements and should not compete for the same budget.

Crawl waste reduction

Identifying and closing the URL space that consumes crawl attention without any prospect of ranking — parameter permutations, near-duplicate filter states, tracking variants.

Indexation defect diagnosis

Template-level faults that suppress an entire page type at once: canonical errors, render-blocking conditions, incorrect robots directives generated in the application.

Verification after change

Confirming that a fix produced the intended crawl or indexation effect, with a before-and-after measurement rather than a closing ticket.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Logs first, reports second
Search Console and equivalent reports tell you what a search engine has concluded; logs tell you what it did. Because the remedies for a crawl problem and an index problem are opposite, the diagnosis is established from server behaviour before anything is changed.
Segment the URL space before measuring it
A sitewide crawl-to-index ratio averages over page types that have nothing in common. The estate is split into classes first, because the useful signal is always within one of them and never in the total.
Classify the index gap, do not just count it
Excluded-by-directive, error-blocked, quality-excluded and crawl-starved are separated explicitly. Counting crawled-but-not-indexed URLs produces a number with no remedy attached.
Find the template, not the pages
When exclusion correlates with a page type rather than with content, the defect is in the template or the application that renders it. Fixing it in one place restores the whole class, which is why this is worth the effort of finding.
Direct crawl attention rather than petition for more
Crawl rate is largely earned through reliability and navigation structure. The work is removing waste and making the important content cheap to reach — not requesting a higher rate and waiting.
Measure the recovery
Every fix is paired with a stated expected effect and a period over which it is checked. Without that, a change that did nothing is indistinguishable from one that worked slowly.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Diagnosis

Server log analysis by crawler and URL class
Crawl-to-index ratio segmented by page type
Index-gap classification with cause per class
Error-response and latency analysis

Remediation

Sitemap segmentation by content class and lifetime
Crawl-waste elimination
Template-level indexation defect fixes
Server response and reliability remediation, with the development team

Verification

Before-and-after crawl and indexation measurement
Discovery-latency measurement for time-critical content
Regression monitoring so the defect does not return silently

Under the hood

Architecture and technology

Crawl path

robots directives generated by the application
XML sitemaps segmented by class and lifetime
Internal linking depth to priority content
Response code correctness across the estate
Canonical resolution at template level

Measurement

Request logs retained long enough to compare periods
Per-class crawl and index tracking
Discovery latency for news and listings
Alerting on crawl-rate and error-rate change

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

Pages render but search engines appear to see nothing in them

See JavaScript SEO. A rendering failure presents as an indexation failure and is frequently misdiagnosed as one.

You are moving a site and need indexation preserved through it

See Website Migrations. Migration loss is usually a crawl and indexation failure with a specific cause.

Questions

Frequently asked

Is this different from technical SEO?

It is a subset of it, worked at depth. Technical SEO covers the whole serving layer; this is specifically the movement of crawlers through a site and what happens to the URLs they find. It is separated out because the diagnosis is log-driven and the remedies are largely architectural, which is a different conversation from a general technical audit.

Do you need server access?

For a proper diagnosis, yes — server logs and the ability to inspect responses as a crawler receives them. Without log access the work becomes inference from public reports, which is materially weaker and frequently wrong about cause. If logs are unavailable we will say what that limits rather than presenting a confident diagnosis without them.

How long before anything changes?

Some fixes show effect within days — a robots directive, a sitemap correction, a template defect. Crawl-rate recovery is slower, because it depends on observed reliability over time and is not something that can be requested. We set an expected window per fix before making it, so a change that is not working is visible rather than assumed to be working slowly.

Our index count dropped. Is that bad?

Not necessarily, and often the opposite. Sites frequently want a smaller index: fewer near-duplicate URLs, fewer parameter permutations, more of the crawl spent on pages that can actually rank. The question is which URLs left and whether the ones that remain are the ones you would choose. A falling count with rising clicks is a good outcome.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work