Engineering

Engineering for websites that cannot afford to fall over

Traffic is only a problem when the architecture was sized for average load. We engineer and remediate platforms for sustained concurrency and unpredictable peaks — the conditions under which most failures actually occur.

The problem

Platforms are usually designed for the traffic they had

The failure mode is remarkably consistent. A platform is built and tuned against its current load, grows, and then fails at a peak. Every tuning decision was correct at the time. The problem is that average-load tuning produces a system that is comfortable most of the time and unavailable at the moments that matter most — which for a community, a marketplace or a publisher is precisely the wrong trade.

  • Intermittent 5xx errors under load, fine the rest of the time
  • Time to First Byte that degrades sharply at peak
  • Server resources pinned while measured utilisation looks reasonable
  • Database contention that appears only under concurrency
  • Traffic spikes that cause failure rather than slowdown
  • Infrastructure cost rising faster than traffic
  • An inability to say which resource will saturate first
Who this is for

The people who usually bring us this problem

CTO / VP Engineering

You have had incidents, or you can see one coming, and you need the architecture assessed on evidence rather than intuition.

Founder of a platform business

Downtime is directly commercially damaging and you cannot afford the reputational cost of a failure during your busiest period.

Head of Product

Peak-time degradation is damaging the user experience at exactly the moment your most engaged users are present.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

Downtime costs more than the lost sessions

On a marketplace or transactional platform an outage interrupts activity already in progress. On a community it breaks a daily habit that is expensive to rebuild. The direct traffic loss is the smallest part of the cost.

Crawl efficiency falls when the platform is unreliable

Recurring server errors cause search infrastructure to reduce how much it crawls, which degrades search visibility as a second-order effect of a reliability problem.

Sizing for peak by adding hardware is the expensive answer

Provisioning for the worst case without addressing what causes the worst case produces infrastructure that is idle most of the time and still fails at the top of the curve.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Load-behaviour diagnosis

Establishing which resource saturates first and at what concurrency, from evidence across the whole serving path. The visible symptom is frequently two tiers below the actual constraint.

Connection and concurrency handling

The connection layer is the first thing to break under concurrency and the most commonly under-configured. Tuning for concurrent connections rather than average throughput is a different exercise from tuning for load.

Caching architecture

Establishing what can be cached, for how long, and at which layer. Caching is the highest-leverage answer to high traffic, and it is a correctness question before it is a performance one — the failure mode of a wrong cache policy is serving something incorrect.

CDN and edge architecture

Moving what can be answered at the edge away from the application tier. This is frequently the single change with the largest effect on both capacity and latency.

Load balancing and distribution

Distributing demand across origins so that peak traffic is a throughput problem rather than a hard ceiling.

Database and application bottlenecks

Contention, query cost and connection pooling under concurrency — the tier where scaling problems are most often hiding and hardest to see from outside.

Incident diagnosis

Establishing the cause of failures that only occur under production conditions. These are the hardest bugs to reproduce and the ones with the largest consequences.

Capacity and observability

Instrumentation that answers 'what will break first' before it breaks, and capacity modelling grounded in observed load rather than guesswork.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

  1. Reproduce the load before changing anything

    A fix applied without reproducing the failure is a guess. We establish the load profile that causes the problem, whether that is in production instrumentation or a controlled environment.

  2. Work the whole serving path

    From connection layer to database, in order, because the constraint is rarely where the symptom appears. Solving the top of the path when the constraint is at the bottom produces no change and a lot of confusion.

  3. Stabilise, then optimise

    On a platform that is currently failing, restoring stability takes precedence even at the cost of temporary inefficiency. Optimisation on an unstable system is not measurable.

  4. Reduce before adding

    Moving work off the critical path, caching what can be cached and eliminating what need not be done is nearly always cheaper and more durable than adding capacity to do it faster.

  5. Remediate without an outage

    On a live platform there is no maintenance window. Every change is introduced into production, which constrains technique and pace and requires a disciplined approach to rollout and rollback.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Diagnosis

  • Full serving-path load analysis
  • Concurrency-limit identification by tier
  • Production failure investigation with timeline
  • Capacity model based on observed load

Remediation

  • Connection-layer configuration for concurrency
  • Caching architecture and policy
  • CDN and edge delivery
  • Load balancing and origin distribution
  • Database and application bottleneck resolution

Operation

  • Observability instrumentation
  • Capacity indicators and alerting
  • Runbooks for peak periods
  • Ongoing managed operation where required
Under the hood

Architecture and technology

Serving path

  • Connection layer and concurrent-connection limits
  • Reverse proxy and request routing
  • Load balancing across origins
  • Application tier sizing and concurrency
  • Database connection pooling and contention

Caching layers

  • Edge and CDN caching policy
  • Application-level caching
  • Object and query caching
  • Cache invalidation correctness
  • Static and semi-static asset delivery
Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Online communities

Spain's largest forum

Since 2019, Soludome has managed infrastructure and platform operations for one of Spain's largest online communities. The work covers server and performance optimisation, CDN architecture, load balancing and site security under sustained traffic.

Since 2019continuous engagement
Online communities

Arrse.co.uk

An established community platform had accumulated three separate problems that compounded each other: page loads that had degraded with growth, search visibility that had not kept pace with the community's authority, and a security posture that had not been revisited as the platform's profile grew. The engagement addressed all three concurrently.

4%increase in organic clicks
Marketplaces

Platform recovery

A large marketplace platform in the United States was experiencing critical downtime from compounding traffic and application faults. Soludome analysed requests through the web application firewall, identified and blocked the DDoS patterns, and repaired a third-party payment flow that allowed links to be generated repeatedly without authentication. Server load fell and platform performance improved after the work.

Around 2020initial recovery; occasional support since
Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

Pages are slow but the platform is stable

That is a different problem with a different remedy. See Website Performance Engineering.

You are not sure whether the issue is capacity or a defect

Distinguishing those two is the first thing we do. See Infrastructure Engineering.

Questions

Frequently asked

What counts as high traffic?

The number that matters is concurrent users, not monthly pageviews. A platform with a large but casual audience can be entirely comfortable where one with fewer users in long sessions is not. We size the question to your load profile rather than to a threshold, because the threshold would be arbitrary.

Do you work on live platforms?

Almost exclusively, and it is a genuine constraint on the work. There is no maintenance window on a platform with users. Every change is introduced into production and validated there, which is slower and requires more discipline than a rebuild — and it is the only option that does not involve taking the service offline.

Is this not just a hosting problem?

Hosting is where the capacity is, but most high-traffic failures are architectural rather than provisioned. A platform that does avoidable work on every request will fail at some load regardless of how much infrastructure is beneath it. We address both, but the architecture is usually the larger win and the cheaper one.

How do you find a bug that only happens at peak?

By instrumenting the moment it happens and reproducing the load that produces it. Production-only failures are the hardest class of problem in this work, and they are usually resolved by narrowing from a symptom to a specific resource that saturates at a specific concurrency — which requires evidence collected at the time, not afterwards.

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.