Search Engineering

When server errors are costing you search, not just uptime

A search engine that repeatedly meets errors during a visit does not complain. It reduces how often it comes back. That decision is slow, invisible and frequently the real cause of an impression decline that is being investigated as a content problem.

The problem

Crawl-rate loss looks nothing like a server problem

Errors that a user never sees — a 5xx on one request in fifty, latency during the hours a crawler happens to visit, a timeout on a page type that is expensive to render — are exactly the conditions that reduce crawl frequency. Nothing alerts on them, because the site is available. By the time the consequence is visible as an impressions decline, the correlation with a server change made months earlier is not obvious to anyone.

Impressions fell and no content or ranking change explains it
Crawl rate dropped and never recovered
Search Console reports server errors, timeouts or soft 404s
Errors are intermittent and correlate with traffic rather than with a deployment
Pages render correctly for users but slowly, and slowly is the whole issue
A page type is expensive to generate and is crawled less than its importance warrants
The server was changed or reconfigured at some point and nobody noted when
Crawl activity is concentrated in hours when the application is under most load

Who this is for

The people who usually bring us this problem

Head of SEO / Technical SEO Lead

You can see the crawl-rate and impression decline in the data and the cause appears to be server-side. You need it established with evidence rather than asserted.

CTO / VP Engineering handed a crawl-rate problem

An SEO team has reported that crawl rate fell and you need it translated into something measurable at the infrastructure layer.

Head of Platform with a reliability backlog

You have known intermittent errors that have never been prioritised, and you need to know whether the search consequence justifies moving them up.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

The consequence arrives weeks after the cause

Crawl frequency is a long-lived relationship rather than a per-request decision. The visit rate falls gradually after the errors, and the impressions decline follows the crawl decline, which means the cause is months behind the symptom and hard to correlate by inspection.

Recovery is slower than the decline

Once crawl frequency drops, restoring it requires an extended period of clean, fast responses before the rate returns. The fix is not the end of the incident; the observation period after it is.

Intermittent faults are the most expensive to find

An error that occurs under load, at a particular hour, or on one expensive page type resists reproduction. Each failed attempt to reproduce it looks like evidence that it is not happening.

The traffic loss is attributed to content

Impression decline with no ranking change reads as a content problem, so teams respond by writing more. That is expensive, it does not address the cause, and it delays the diagnosis by whatever the investigation cycle costs.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Error analysis from the crawler's side

Establishing what a crawler actually received rather than what monitoring recorded: status code distribution over time, which URLs and templates produce errors, and whether the pattern correlates with load, time of day or a specific application path.

Crawl-rate correlation

Lining the crawl data up against the error and latency data to establish the relationship with evidence. The claim that errors suppressed crawl rate is either supported by that correlation or it is a hypothesis, and the difference matters before work is commissioned around it.

Latency as a first-class cause

A response that is slow rather than failing produces the same crawl-rate consequence with none of the visibility. Response-time distributions by page type are measured, not just averages, because the tail is what a crawler experiences.

Timeout and hang diagnosis

Requests that never complete: long database calls, external dependencies that block the render, lock contention, connection pool exhaustion. These are frequently the most damaging and the least reported, because the request leaves no error to count.

Soft 404 and thin-response remediation

Pages that return a success status while presenting nothing useful — expired listings, empty search results, out-of-range pagination. These consume crawl budget and signal that a class of URL is not worth visiting.

Crawl-window alignment

Where crawl activity concentrates in the hours the application is busiest, that overlap is the problem to address. Whether the fix is serving-path work or capacity is an outcome of the diagnosis rather than an assumption.

Verification against crawl behaviour

Confirming the fix in the data that mattered: error rate, response time and crawl frequency, over a period long enough for the crawl relationship to respond. A fix confirmed only by a monitoring graph has not been confirmed against the thing it was meant to improve.

Monitoring the search consequence

Alerting on the signals that precede a crawl-rate loss rather than on the impressions decline, which arrives far too late to act on.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Establish whether this is a search problem or an outage
If the site is currently failing for users, that is an incident and it is handled first as one. This engagement starts when the site is broadly available and the question is why search is behaving as though it is not — because the two situations need different first actions.
Read the errors as a crawler would
Monitoring reports what the business defined as important. A crawler experiences status codes, response times and completion — including on page types nobody watches. The analysis starts from the request log rather than from the dashboard.
Correlate before concluding
Crawl rate against error rate against latency against load, over the same window. This is the step that separates a supported diagnosis from a plausible one, and it is also what establishes which of several concurrent problems is actually binding.
Separate failing from slow
Both suppress crawl and they have different fixes. A 500 is a defect; a four-second response is a cost decision. Treating a latency problem as an error problem leads to chasing a bug that does not exist.
Fix at the level the fault exists at
An application defect, a database query, a capacity limit and an unavailable dependency produce similar signatures and completely different work. Each fix is applied where the cause is, which frequently means it is an infrastructure engagement in substance.
Watch the crawl relationship recover
Verification runs until crawl frequency responds, which takes longer than the fix takes to deploy. Reporting a fix as complete on the day it shipped would be reporting the wrong event.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Diagnosis

Status code distribution over time, per template and for the crawler specifically
Response-time distribution including the tail, not the average
Crawl-rate correlation with the error and latency data
Page types producing errors or timeouts disproportionately
Soft 404 and thin-response inventory
Which layer is binding: application, database, capacity, dependency or configuration

Remediation

Application or configuration fixes for the identified error source
Latency work where the tail is the cause
Soft 404 and thin-content handling through status codes rather than content removal
Crawl-window load analysis where the overlap is the problem
Coordination with the infrastructure engagement where the fix is server-side

Verification

Post-fix error and latency measurement against the original window
Crawl-frequency tracking until it responds
Impression and indexation recovery tracked against a pre-decline baseline
Monitoring on the precursors, so the next occurrence is caught in the useful window

Under the hood

Architecture and technology

How server behaviour reaches search

5xx responses during a crawl reduce the frequency of subsequent visits
Latency, not just failure, prompts the same reduction in visit rate
Timeouts leave no error to count and are frequently the largest contributor
Soft 404s spend crawl budget on URLs with nothing to return
Errors concentrated on one template deprioritise an entire page class
Recovery requires sustained clean responses, not a one-day fix

What the diagnosis needs from your side

Request logs covering the period, with status code and response time retained
Crawl data for the same window
A record of deployments, configuration changes and infrastructure work
Access to monitoring at a resolution finer than the incident duration
The infrastructure owner, because the fix is often theirs rather than the SEO team's

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

The site is currently failing

An outage is an incident before it is a search problem. If users are affected, start there.

Server outage recovery

You want availability designed rather than a cause found

Reliability as an engineering property, including monitoring and capacity.

Platform reliability

Articles are crawled but not indexed

Discovery is working and something downstream is not.

Crawl and indexation diagnosis

The errors are already affecting a publishing pipeline

Discovery speed as a commercial property, where errors land hardest.

News publisher SEO

Questions

Frequently asked

How do we know errors are the cause rather than our content?

By correlating crawl and error data over the same window, which is the first deliverable rather than an assumption. A content-driven decline changes rankings and click-through while crawl activity stays steady; a server-driven one shows crawl frequency declining first and impressions following it. If the data does not support the server explanation we will say so, because the next step would otherwise be commissioning work against the wrong cause.

Our uptime monitoring says 99.9%. How can there be a problem?

Uptime monitoring tests a small number of URLs, usually cheap ones, from a small number of locations, at an interval you chose. A crawler experiences thousands of URLs including the most expensive, from its own infrastructure, at times you did not choose. The two measurements can both be accurate and describe completely different experiences, and the second one is what determines crawl behaviour.

Is this an SEO project or an infrastructure project?

The diagnosis is a search engagement and the fix frequently is not. Establishing that errors are suppressing crawl is search work; correcting the application defect, the query or the capacity limit that produces them is engineering work, and often infrastructure work. We do both, which is the reason this page can exist as a search page without pretending the remedy is an SEO one.

How long until crawl rate recovers?

Longer than the fix takes, and that is the part to plan around. Crawl frequency is a relationship built over time rather than a per-request decision, so it responds to a sustained period of clean responses rather than to a deployment. We track it until it moves and report the trajectory rather than declaring the work finished when the code ships.

Can you fix this without access to the server?

The diagnosis, yes — it works from logs and crawl data. The remedy usually needs somebody who can change the application or the configuration, and if that is your team we will produce the change specification and the evidence to prioritise it. What we will not do is diagnose a server fault and present the diagnosis as the deliverable when the value is in the fix.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work