Platform Engineering

Reliability engineering: what happens when it breaks

Capacity planning keeps a platform up. It does not tell you what happens at 03:00 when something fails anyway. Reliability is the engineering of detection, diagnosis and recovery — and of making an intermittent fault reproducible.

The problem

Most platforms find out about outages from their users

The defining characteristic of an unreliable platform is not that it fails. It is that the failure is discovered externally, understood only in retrospect, and recurs because nothing was learned that could prevent it. That is a monitoring and process problem more often than it is a hardware one, and both are engineering work with known solutions.

Downtime discovered by a customer rather than by an alert
Intermittent errors that cannot be reproduced, so cannot be fixed
Recovery that depends on a specific person being available
Incidents diagnosed afterwards from logs that were nearly not kept
No agreed threshold for what counts as degraded versus down
The same class of failure recurring months apart under different symptoms
Deploys that cannot be rolled back quickly, so a bad release stays live

Who this is for

The people who usually bring us this problem

CTO / VP Engineering

You are carrying availability risk and know the platform's failure modes are not understood well enough to be planned for.

Head of Platform / Infrastructure

You have monitoring but it reports symptoms rather than causes, and incidents still take longer to diagnose than they should.

Founder

Reliability has started to matter commercially — a customer noticed, or a contract now depends on it — and there is no process around incidents yet.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

An unreproducible fault is effectively permanent

If an intermittent failure cannot be reproduced on demand, no fix can be verified and the incident will recur. Capturing enough context at the moment of failure is the only reliable route out.

Recovery time matters more than failure rate

Users and customers measure how long an outage lasted, not how rare outages are. A platform that fails occasionally and recovers in seconds is experienced as more reliable than one that fails rarely and stays down for hours.

Reliability is a search-visibility property too

Crawlers respond to error rates and instability by crawling less. On a content-led site an availability problem is also a discovery problem, and the two are usually investigated separately.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Failure-mode analysis

Establishing what can fail, what happens when it does, and what the user experiences — before it happens rather than during. Includes the dependencies whose failure takes the platform with them.

Observability and alerting

Instrumentation over the behaviours that indicate real problems, with alerts that fire on user-affecting conditions rather than on proxy metrics. An alert nobody trusts is worse than no alert.

Intermittent fault capture

Correlated request, error and infrastructure signals retained long enough to reconstruct a fault after the fact, including timestamps that make a pattern visible across occurrences.

Recovery engineering

Making the recovery path fast and rehearsed — rollback, failover, degraded operation — so restoring service does not depend on improvised decisions under pressure.

Incident process

Who is notified, who decides, what is recorded and what happens afterwards. Lightweight enough to be used rather than documented and ignored.

Deployment safety

Release practices that make a bad deploy survivable: progressive rollout, fast rollback, and the ability to determine whether a change caused an incident.

Reliability targets

An agreed definition of what counts as degraded and what counts as down, so availability is a measured quantity rather than an impression.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Define the failure modes before the monitoring
Deciding what would count as broken determines what is worth measuring. Monitoring built first tends to measure what is easy to collect, which is not the same thing.
Instrument for diagnosis, not just detection
An alert that says the site is down has limited value. The target is enough captured context to answer why, without needing the incident to happen again in order to find out.
Make the recovery path the first deliverable
Fast rollback and rehearsed failover are addressed before deeper fixes, because they shorten every incident that occurs in the meantime — including the ones caused by whatever is fixed next.
Treat intermittency as a data-capture problem
Intermittent faults are usually solved by recording more around the moment of failure, not by reasoning harder. Correlation across occurrences is what turns an anecdote into a reproducible defect.
Keep the process usable
A three-page incident process will not be followed at 03:00. What is agreed is deliberately short enough to be used by someone tired, and any part that is not used is removed rather than retained for completeness.
Review, and actually change something
Each incident produces at most a small number of specific changes, each with an owner. Reviews that produce observations rather than changes do not reduce the next incident's cost.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Assessment

Failure-mode inventory with user-visible impact
Dependency and single-point-of-failure map
Current detection coverage, and what would not be noticed
Recovery time and rollback capability as measured

Implementation

Instrumentation and correlated log retention
Alerting on user-affecting conditions
Rollback and failover paths, tested
Backup and restore verification — restore tested, not assumed

Operation

Incident process and notification routing
Availability and recovery-time measurement
Post-incident changes with owners
Periodic rehearsal of the recovery path

Under the hood

Architecture and technology

Reliability concerns

Health signals at the layers users actually traverse
Redundancy where a single failure is unacceptable
Graceful degradation instead of total failure
Deployment and rollback mechanics
Data durability and tested restore

Operational concerns

Alert routing and escalation
Log and trace retention across the window incidents are investigated in
Runbooks for the failures that have happened before
Availability reporting against an agreed definition

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

Online communities

Arrse.co.uk

An established community platform had accumulated three separate problems that compounded each other: page loads that had degraded with growth, search visibility that had not kept pace with the community's authority, and a security posture that had not been revisited as the platform's profile grew. The engagement addressed all three concurrently.

4%increase in organic clicks

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

You need the platform to hold up as traffic grows

See High-Traffic Websites, which is capacity and behaviour under load rather than failure handling.

The instability is specifically affecting search crawling

See Crawl and Indexation. Error responses reduce crawl rate, so a reliability problem can present as a visibility problem.

Security is the concern rather than availability

See Website Security, covering hardening, attack traffic and remediation.

Questions

Frequently asked

Is this different from what a hosting provider gives us?

Infrastructure monitoring tells you a machine is up. It does not tell you that a template is throwing errors for one class of request, that a dependency is degrading under a specific workload, or that a deploy broke one route while the health check passed. Reliability engineering works at the layer where users actually experience failure.

Do you take on-call?

No. On-call is best held by the team responsible for the platform day to day, because recovery depends on knowledge that is expensive to transfer. What we do is make the platform's behaviour visible, make the recovery path fast and rehearsed, and set up the process — so that the people holding on-call are not diagnosing from scratch while users are affected.

How do you fix something intermittent?

Generally by capturing more around the moment it happens: correlated request, application and infrastructure signals, kept long enough and with enough precision to line occurrences up against each other. The pattern is what makes it reproducible. Trying to reason a way to the cause from a single vague report is usually unsuccessful.

We have not had a serious outage. Is this worth doing?

Often not yet, and we will say so. If the platform is small, the failure modes are few and recovery is genuinely quick, the return on formal reliability work is low. The point at which it starts paying is when a failure costs materially more than the work to prepare for it — a commercial threshold rather than an engineering one.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work