Forum Migration

Migrating a large community without taking it offline for a week

At a certain size a forum migration stops being a conversion problem and becomes a logistics one. The questions are how long the data takes to move, how much of it changes while it is moving, and how to prove nothing was lost when the counting itself takes hours.

The problem

Volume changes the method, not just the duration

A small forum is converted in a run and verified by inspection. A large one has to be moved in stages, reconciled by counting rather than by looking, and cut over inside a window whose length is set by how fast the data can be transferred and how much of it changes while that happens. Those are different problems, and the techniques that work at one size do not work at the other.

  • The database or attachment store is large enough that a straightforward import takes longer than a maintenance window
  • The bulk of the size is attachments or media rather than text
  • The community is active daily, so any window is visible to members
  • Previous migration attempts ran out of time or disk on a staging environment
  • Reconciliation would take hours even if the conversion were instant
  • Validation approaches that work by inspection are impossible at this scale
  • The destination environment has to be sized before anything can be tested
  • Nobody can say how much of the data is actually in the database
Who this is for

The people who usually bring us this problem

An owner of a very large community

The migration is right and the practical question is how to execute it without a downtime members will notice.

An operator whose previous attempt failed

It ran out of time, disk or patience, and you need a staged method rather than a faster machine.

A team planning a migration before committing

You need the constraints established — transfer time, change rate, window length — before you decide whether the move is feasible.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

The window is set by the delta, not by the total

Bulk data moves before the cutover, so the window covers only what changed in the meantime. On a busy community the change rate is the constraint that decides window length, and reducing it is more tractable than transferring faster.

Verification at scale has to be by counting

Nobody inspects a million posts. Reconciliation is automated and per content type, and the automated counts become the only evidence the migration is complete — which means the counting has to be built as part of the importer rather than afterwards.

Attachment volume usually dominates, and is usually underestimated

Media is frequently the larger part of a forum's stored data and the part held outside the database. A migration planned around the database size alone discovers the real volume at the point it has least room to react.

A failed attempt costs more than the delay

On a large community, an abandoned cutover means the work is repeated, the members have been disrupted for nothing, and the next attempt has less goodwill behind it.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Volume and shape assessment

Establishing what actually has to move and where it lives: database size by table, attachment and media volume including anything outside the database, and the change rate during a normal day. The shape of the data decides the method more than the total does.

Destination sizing before anything runs

Sizing the target environment against the real volume with headroom for the conversion's own working space. Import processes need room, and a destination sized only for the data it will hold is a destination that fails part-way through.

Staged bulk transfer

Moving the bulk ahead of the window across as many passes as the volume requires, so the cutover handles only the delta. This is the technique that makes a large migration fit a short window, and it is the same at every size — only the number of passes differs.

Chunked and resumable conversion

Converting in segments that record their own progress and can be resumed, so a failure at eighty per cent does not restart from zero. On a large migration this is not a nicety; it is what makes the work completable within a realistic error budget.

Delta management

Tracking what changed between the bulk transfer and the cutover, and reducing the change rate where possible — by scheduling against the community's quiet period, or by briefly limiting write-heavy activity. The window is a function of the delta, so managing the delta is managing the window.

Automated reconciliation at scale

Per-content-type counting that completes in an acceptable time, with sampling for the structures where a full count is impractical. At this size the reconciliation is a deliverable in its own right and it is built alongside the importer.

Cutover sequencing

The final delta, the switch and the redirect activation as one rehearsed sequence with named owners and a defined rollback trigger. The larger the community, the more the cutover needs to be a procedure rather than a judgement call.

Constraint management against the source and destination

Transfer rate, disk, connection limits and the destination's own write capacity, identified per system. What limits a large migration is usually a specific constraint in one place rather than overall slowness, and naming it is what makes the plan realistic.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

  1. Establish the volume and the change rate first

    The two numbers that decide everything: how much has to move, and how much of it changes per hour. No method is proposed, and no window is estimated, before both are measured rather than assumed.

  2. Size the destination with working room

    The target is provisioned for the data plus the conversion's own requirements plus headroom. A destination that exactly fits the content will fail part-way through, and at this scale that failure is expensive.

  3. Move the bulk outside the window

    The slow part happens in advance, across as many passes as needed. This is the same approach as a small migration, applied repeatedly, and it is what turns an impossible window into a short one.

  4. Make every stage resumable

    The conversion records its own progress per segment, so an interruption costs the segment rather than the run. Long processes fail eventually, and the plan assumes that rather than hoping otherwise.

  5. Reconcile by counting, and say when it cannot be exhaustive

    Automated per-type counts, with sampling where a full count is impractical, and an explicit statement of which structures were sampled rather than counted. Claiming exhaustive verification on something that was sampled is worse than stating the sample.

  6. Rehearse the cutover at the real volume

    The rehearsal runs against the full dataset, not a subset, because the timing and the failure modes at scale are not visible in a reduced copy. That makes rehearsal expensive, which is why it is planned for rather than improvised.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Assessment

  • Volume by component: database tables, attachments, media, anything outside the database
  • Change rate during normal community activity
  • The specific constraint that will limit the transfer, named per system
  • Destination sizing with working headroom
  • Estimated window length derived from the delta rather than from the total
  • A statement of which parts cannot be exhaustively counted

Execution

  • Staged bulk transfer across multiple passes
  • Chunked, resumable conversion
  • Delta tracking and change-rate management
  • Automated reconciliation per content type, with sampling stated where used
  • Rehearsal at real volume

Cutover

  • Rehearsed sequence with named owners
  • Final delta applied and verified
  • Switch and redirect activation as one step
  • Tested rollback with a defined trigger
  • Post-cutover verification with real member activity
Under the hood

Architecture and technology

What makes a migration large

  • Attachment and media volume, frequently the dominant component and often outside the database
  • A change rate that produces a meaningful delta over any transfer window
  • A community that is active daily, so no hour is truly quiet
  • Destination sizing that has to accommodate the conversion's working space
  • Reconciliation that takes long enough to need designing
  • A rehearsal that must run at full volume to be meaningful

The constraints that decide the window

  • Transfer rate between source and destination
  • Write capacity at the destination during conversion
  • Disk available for the conversion's own working set
  • The community's change rate over the window being planned
  • Time needed to reconcile and verify before serving members
Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Online communities

Spain's largest forum

Since 2019, Soludome has managed infrastructure and platform operations for one of Spain's largest online communities. The work covers server and performance optimisation, CDN architecture, load balancing and site security under sustained traffic.

Since 2019continuous engagement
Online communities

Arrse.co.uk

An established community platform had accumulated three separate problems that compounded each other: page loads that had degraded with growth, search visibility that had not kept pace with the community's authority, and a security posture that had not been revisited as the platform's profile grew. The engagement addressed all three concurrently.

4%increase in organic clicks
Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

The source is a platform no importer covers

Schema reverse-engineering and derived-value reconstruction.

Custom forum importers

The source is a supported platform

Compatibility positions and staged upgrade paths.

Forum migration to XenForo

Search visibility is the main risk

Redirect mapping at volume and indexation recovery.

Forum SEO migration

The destination infrastructure is the open question

Capacity planning and serving-path design for the migrated community.

Server migration
Questions

Frequently asked

How long will the migration take?

We will not give a duration before measuring the volume and the change rate, because those two numbers decide it and they vary enormously between communities of similar size. A forum with a very large attachment store and a slow transfer path is a different project from one with the same post count and a fast one. What we will give you after the assessment is the constraint that governs it and a window derived from it.

How long will the forum be offline?

The window covers the delta rather than the whole dataset, which is what makes a short cutover possible at this size. Its length follows the change rate over the transfer period, so it is managed by reducing the delta — scheduling against the community's quiet period, or briefly limiting write-heavy activity — rather than by transferring faster. We will not promise zero downtime; the cutover is a real event.

Can you migrate without a staging environment the same size as production?

The rehearsal has to run at real volume to be meaningful, because the timing and the failure modes at scale are not visible on a reduced copy. If a full-size staging environment is not available, that is a constraint to plan around rather than a step to skip — and the honest options are to provision one temporarily or to accept a longer window with more of the verification happening during cutover.

How can you prove nothing was lost at this scale?

By counting, not by looking. Reconciliation is automated and structured per content type, and where a full count is impractical we say which structures were sampled rather than counted. What we will not do is present sampling as exhaustive verification, because at this scale the difference between the two is large and the claim would be the thing you relied on.

What is the most common reason a large migration fails?

Running out of something — time, disk, or the destination's write capacity — rather than a conversion error. That is why the constraints are named per system during the assessment, and why the destination is sized with working room rather than only for the data it will hold. A conversion that is correct but does not fit is still a failed migration.

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.