Platform Engineering

Server and website outage diagnosis and recovery

A failing system is a diagnosis problem before it is anything else. Restarting the server is a decision about the last ten minutes. Finding out why it failed at 03:00 four nights running is a decision about the next six months.

The problem

The recovery is usually already known. The cause is not.

Most teams can get a system back up. What they cannot do is explain the failure, which means every incident is met with the same improvised response and the same likelihood of recurrence. Recurring 5xx errors, availability that drops at a predictable hour, degradation that arrives with load — these are patterns with causes, and the pattern is information only if somebody records it.

The site or application fails repeatedly and the fix each time is a restart
5xx errors appear at predictable times of day or week
Downtime correlates with traffic but the capacity looks adequate
Errors are visible to users before they appear in any monitor
The same incident recurs and the previous fix is not remembered
Nobody can say what the system was doing in the ten minutes before it failed
Recovery depends on one person's memory of what to do
The failure clears on its own and nobody finds out why

Who this is for

The people who usually bring us this problem

A business whose site keeps falling over

It comes back, and it comes back again next week. You need the pattern explained rather than the symptom restarted.

CTO / VP Engineering with an unexplained incident

The logs are large, the failure is intermittent, and the team has been through them without finding the thread.

Operations lead with no incident process

Recovery exists as folklore rather than as a procedure, and you need it written down before it is needed again.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

Restarting is not recovering

A restart returns the system to a state it will leave again, because nothing about the conditions that produced the failure has changed. The recurrence is not bad luck; it is the same cause meeting the same trigger.

Unrecorded incidents cannot be diagnosed

The evidence that identifies a cause — metrics, logs, timings, what changed — decays within hours and disappears within days. An incident nobody characterised is an incident that will be investigated from scratch when it returns.

Availability loss costs more than the outage itself

User trust, search crawl frequency and any transaction in progress all suffer. The crawl consequence is the one most often missed, because it takes weeks to surface and is attributed to something else entirely.

Improvising under pressure produces the next fault

A change made at 03:00 to restore service is a change nobody has reviewed. Some proportion of the next incident is caused by the previous recovery, and that is avoidable.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Triage and stabilisation

Stopping the immediate symptom with the least change possible, and recording what was done and when. Minimising intervention during triage matters because every change made under pressure is a variable that complicates the diagnosis afterwards.

Evidence capture during the incident

Metrics, logs, process state, resource exhaustion, connection behaviour and what changed recently, captured while the system is still in the failure state. This window closes quickly and is the difference between a diagnosis and a guess.

Dependency mapping

Establishing what the application actually depends on and which of those dependencies can take it down — database connections, cache availability, DNS, third-party APIs, file handles, disk. Outages are usually reachable through something nobody counted as critical.

Log and metric analysis

Reading the logs against the failure window rather than in isolation, and correlating what the application reported with what the infrastructure was doing at the time.

Root-cause analysis

Establishing the sequence: what changed, what condition it created, what triggered the failure, and why it presented the way it did. The output is a narrative with evidence at each step, not a single suspected line of code.

Capacity and load analysis

Where failures correlate with load, establishing whether the constraint is genuinely capacity or whether the system has a scaling behaviour change at a particular point. These have completely different fixes.

Remediation and the preventive fix

Correcting the cause, and then addressing why the failure was possible at all — a missing limit, an unhandled condition, a monitoring gap, a recovery procedure that only worked because one person remembered it.

Monitoring and alerting on the right signals

Alerting on the conditions that preceded the failure rather than on the failure itself, so the next occurrence is caught in the window where intervention still helps.

Runbook and handover

A written procedure for this class of incident, including what to capture before doing anything. Recovery knowledge that lives in one person's head is a single point of failure that nobody lists.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Capture before you change
The instinct is to intervene immediately, and acting on it destroys the evidence that would explain the failure. Where a service can be stabilised without clearing state, it is; where it cannot, what can be observed first is observed and recorded.
Establish the window precisely
When the failure started, when it was detected, when it was resolved, and what happened in between. Intermittent faults are diagnosed by correlating a precise window against everything else that varies — which is only possible if the window is exact.
Follow the dependency chain rather than the symptom
The component reporting an error is frequently not the component that failed. Symptoms appear where the failure surfaces, and the cause is often one hop away in something not regarded as critical.
Distinguish the cause from the trigger
A traffic spike is a trigger, not a cause — the cause is whatever made the system unable to absorb it. Fixing triggers is an endless exercise; fixing causes is finite, and the distinction is what stops the same incident being reported in different clothes.
Verify the fix under the conditions that produced the fault
A fix confirmed at normal load has not been confirmed. Where the failure was load-related it is re-tested under that load, because a change that works when nothing is happening is not evidence about the moment it matters.
Write it down while it is still fresh
A root-cause narrative and a runbook, produced during the engagement rather than reconstructed later. The next occurrence will be met by whoever is on duty, and what they have is whatever was written afterwards.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Incident record

Precise failure window with detection and resolution timestamps
Evidence captured while the system was in the failure state
The dependency chain as it was actually experienced
Cause and trigger, stated separately
What was changed during recovery, when, and by whom
An honest statement of what remains unexplained

Remediation

The corrective fix, applied and verified under representative conditions
The preventive change: limit, guard, capacity or configuration
Monitoring on the precursors rather than only the failure
Alerting routed somewhere a human reads it
Any recovery change reviewed before it stays in place

Handover

Written runbook for this class of incident
What to capture before intervening, and why the order matters
Escalation path and the boundary of what is covered
Preventive work carried into the operations plan where one exists

Under the hood

Architecture and technology

Where failures actually come from

An unhandled condition reached under a specific input or timing
A dependency that stops responding and was never treated as critical
Resource exhaustion that is invisible until it is total
A change deployed without the load conditions it would meet
A configuration difference between environments
A capacity ceiling crossed that nobody had measured
Two faults combining, either of which alone would be survivable

What has to exist before the diagnosis is possible

Retained logs covering the window, not just the moment of failure
Metrics at a resolution finer than the incident duration
A record of what was deployed and when
A way to observe the system without altering it
Agreement on who is authorised to intervene

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

Marketplaces

Platform recovery

A large marketplace platform in the United States was experiencing critical downtime from compounding traffic and application faults. Soludome analysed requests through the web application firewall, identified and blocked the DDoS patterns, and repaired a third-party payment flow that allowed links to be generated repeatedly without authentication. Server load fell and platform performance improved after the work.

Around 2020initial recovery; occasional support since

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

You want availability designed rather than an incident fixed

Monitoring, redundancy, capacity planning and reliability as an ongoing property.

Platform reliability

Errors are suppressing search rather than users

The same faults measured through their crawl and indexation consequences.

SEO server-error remediation

The server needs an owner rather than a diagnosis

Ongoing operations, patching, backup verification and escalation.

Server management

The failures look like attack traffic rather than faults

Volume aimed at taking a site down and volume aimed at exploiting it arrive the same way.

Website security

Questions

Frequently asked

Do you provide an emergency response time?

No, and we will not advertise one. A practice of this size cannot honestly commit to round-the-clock response, and a promise that fails at the moment you rely on it is worse than an honest statement of when we can start. What we will do is tell you plainly what we can and cannot take on, and if you need guaranteed out-of-hours cover that is a provider whose business is built for it.

The site is up again. Do we still need the analysis?

That is exactly when it is most valuable, because the alternative is waiting for the next occurrence and investigating under worse conditions. Recovery restores the state the system was in before the failure, which means the conditions that produced it are still there. We have seen environments where the same incident was restarted away for months before anyone established that a single configuration value was responsible.

Can you work from logs we already have?

Sometimes, and it depends entirely on what was retained and at what resolution. What usually decides it is whether anything was captured while the system was still in the failure state, and whether the failure window is known precisely enough to correlate against. Where that is missing we will say so rather than producing a confident narrative from insufficient evidence.

Will fixing this stop it happening again?

The cause we identify, yes. What nobody can promise is that a different fault will not produce a similar symptom, which is why the preventive work matters more than the corrective fix — monitoring on the precursors, limits where there were none, and a runbook so the next occurrence is met with a procedure rather than from memory.

Is this the same as your platform reliability work?

Related, and separated deliberately. That work is about availability as a designed property: monitoring, redundancy, capacity, failure modes reviewed before they occur. This is what happens when they occur anyway. If you are not currently in an incident and want the system to be less likely to produce one, the reliability engagement is the better starting point.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work