Search Engineering

Making a XenForo community's archive findable

A forum's value is a decades-deep archive of discussion. Its search problem is that the archive is also its largest source of thin, paginated and near-duplicate pages — and the two facts pull in opposite directions.

The problem

Forums produce exactly the page shapes search handles worst

A forum generates a thread for every conversational fragment, a page for every offset within every thread, a member profile for every account, and tag and prefix listings across all of it. Much of that is the archive that makes the community valuable and much of it is near-duplicate or too thin to stand on its own. Sorting the two apart is the work, and it has to be done per class rather than per URL.

  • Threads rank but the site's category and hub pages do not
  • A large share of indexed pages are pagination, profiles or tag listings
  • Answers buried deep in long threads are never found from search
  • The same question is answered in several threads and they compete with each other
  • Crawl activity concentrates on member profiles and low-value listings
  • Long threads are indexed on page one only, so the substance is unreachable
  • Impressions are spread thin across thousands of near-identical URLs
  • New threads are indexed slowly because crawl is spent elsewhere
Who this is for

The people who usually bring us this problem

A community whose search performance is flat

The forum is active and search traffic does not reflect the depth of what is in it.

A forum losing visibility to aggregators

Content from the community is being surfaced through other sites rather than from the source.

A community whose crawl is spent on the wrong pages

Large numbers of profiles, tag listings and paginated URLs are consuming the crawl the content needs.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

A forum's page shapes are the ones search handles worst

Threads are short and numerous, pagination multiplies every one of them, member profiles repeat without contributing, and the same subject is discussed across several threads. That is not a defect in the community — it is what a community is — and it means the indexation model has to be deliberate rather than default.

The substance is often on page four

A long thread's early pages are questions and the answers emerge later. If pagination is not crawled or not linked coherently, the most useful content in the archive is the least reachable — and it is reachable to members, which is why nobody notices.

Member profiles are the classic crawl sink

Thousands of near-empty pages, each a legitimate URL, collectively consuming a large share of a crawl budget that the threads need. Excluding them is usually correct and is a decision rather than a default, because a profile can carry genuine content on some forums.

Forum duplication is structural rather than incidental

The same question asked twice, a thread split, threads merged, quoted replies restating content. Forums cannot avoid it, and the response is to make the choices — canonical, consolidation, redirect — rather than to hope the search engine picks well.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

URL class inventory and indexation model

Every class XenForo generates — threads, paginated thread pages, forums and sub-forums, member profiles, tags, prefixes, search URLs, what's-new and activity routes — enumerated and decided on. The decision table is the deliverable, because consistency across classes is what prevents the duplication the model exists to control.

Thread and pagination indexation

Making a long thread's later pages crawlable and linked coherently, so answers that emerged on page four can be found. This is where forums most commonly lose their own best content, and the fix is in linking and canonical handling rather than in markup.

Deep archive internal linking

The paths a crawler and a reader take through a decade of threads: related threads, tags, prefixes, forum hierarchies, and the cross-references members already make in replies. Forums have weak generated linking and strong human linking, and surfacing the second is where the archive becomes findable.

Thin and duplicate handling

Short threads, empty forums, single-post tags, duplicate discussion and the near-duplicates produced by merging and splitting. Classified by class, with a decision for each rather than a blanket rule that would discard real content along with the noise.

Template-level audit

Canonicals, structured data, headings, meta and the served HTML across each XenForo template. A theme is where forum search problems most often originate, and a single template defect affects every thread or every profile at once.

Crawl health and rate

How much is being crawled, of what, and how quickly new threads are discovered — including the cases where a slow or unstable response has reduced crawl rate. On a large community this is frequently the real constraint behind a discovery complaint.

Sitemaps for a forum estate

Sitemaps generated for threads and other intended classes at the volumes a large forum reaches, split and prioritised so the crawl reaches content rather than listings.

Q&A visibility

Making the question-and-answer content inside threads legible as such — the discussion structure, the accepted answer where one exists, and the page's own title and headings reflecting the question people actually search for.

Measurement per class

Impressions, clicks and coverage split by threads, forums, profiles, tags and pagination, before and after. On a forum an aggregate figure is close to meaningless, because the change that matters is a shift between classes rather than a total.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

  1. Count the classes before deciding about any of them

    How many threads, how many thread pages, how many profiles, how many tags. The proportions decide the priority, and they are consistently surprising — profiles and pagination usually occupy far more of the index than the threads they exist to serve.

  2. Decide per class, and make the decisions agree

    Indexation, canonical and linking choices are made together because they interact: noindexing a class the site still links prominently wastes crawl on paths declared useless. The decision table is written down so a later change can be checked against it.

  3. Prefer consolidation to exclusion where content is real

    A duplicate discussion of a real question has value and should usually be consolidated toward the stronger thread rather than dropped. Exclusion is right for classes that are genuinely structural — profiles, empty listings — and wrong where it discards community output.

  4. Fix in the theme and the templates

    Forum search problems are usually a theme's canonicals, structured data or linking rather than the platform's configuration. Working at the template level means the fix applies to every thread or every profile, including ones posted after the change.

  5. Check what is actually served

    The HTML delivered to a crawler, the canonical it declares, and the redirects it follows. XenForo's output is theme-dependent, so the same platform produces different search behaviour on different communities and the output has to be read rather than assumed.

  6. Measure by class, not in aggregate

    A forum's total impression count can be flat while threads have improved substantially and profiles have collapsed — which is the outcome the work was for and is invisible in the total.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Audit

  • Every URL class XenForo generates here, with counts
  • The indexation model: a per-class decision table for indexation, canonical and linking
  • Thin and duplicate findings, classified, with the handling chosen for each
  • Template-level defects, grouped by the template that produces them
  • Crawl distribution: what is fetched and what that leaves unfetched
  • Where long threads lose reachability

Implementation

  • Indexation model implemented in the theme and templates
  • Pagination and thread-page linking made coherent
  • Internal linking across the archive surfaced from existing human cross-references
  • Canonicals and structured data corrected per template
  • Sitemaps generated for the intended classes at volume
  • Performance or reliability fixes where they are constraining crawl

Measurement

  • Impressions, clicks and coverage split by class, before and after
  • Discovery timing for new threads
  • Crawl distribution across classes, before and after
  • What changed between classes, and what did not move
  • What remains excluded, and the decision behind it
Under the hood

Architecture and technology

The classes a XenForo community generates

  • Threads, and each page of each thread
  • Forums and sub-forums, and their pagination
  • Member profiles and the account routes attached to them
  • Tags and prefixes, and their listing pages
  • Search result URLs, including those from the site's own search
  • What's-new and activity routes
  • Attachment and media pages

Why forum content is hard to surface

  • Threads are short and numerous, so many carry little distinct content
  • Pagination multiplies the URL count for the same discussion
  • Answers appear several pages into a thread, behind the pages most likely to be crawled
  • The same question recurs, creating near-duplicates the forum cannot avoid
  • Generated internal linking is weak; the strong linking is inside members' replies
  • Profiles and listings consume a large share of crawl for little content
Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Online communities

Spain's largest forum

Since 2019, Soludome has managed infrastructure and platform operations for one of Spain's largest online communities. The work covers server and performance optimisation, CDN architecture, load balancing and site security under sustained traffic.

Since 2019continuous engagement
Online communities

Arrse.co.uk

An established community platform had accumulated three separate problems that compounded each other: page loads that had degraded with growth, search visibility that had not kept pace with the community's authority, and a security posture that had not been revisited as the platform's profile grew. The engagement addressed all three concurrently.

4%increase in organic clicks
Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

The community is moving platform

Preserving visibility through a migration: inventory, redirects and recovery.

Forum SEO migration

The platform needs building or extending

XenForo development, add-ons and templates.

XenForo development

Crawl is constrained rather than misdirected

What limits indexing when the rules and the markup are already right.

Crawl and indexation

The server is the constraint

Performance and reliability at community volumes.

XenForo performance optimisation
Questions

Frequently asked

Should we noindex member profiles?

Usually yes, and it is a decision rather than a rule. Profiles are legitimate pages and on most communities they carry little that is unique — a join date, a post count and a signature — while there may be tens of thousands of them consuming crawl that the threads need. On a community where members post substantial content on their profiles, or where profiles are a genuine destination, the answer can be different. The count matters: the same decision is incidental at a thousand profiles and consequential at two hundred thousand.

How do we stop different threads competing with each other?

By deciding which of a near-duplicate set should rank and making the others point at it — canonical or consolidation — rather than leaving the choice to the search engine. Forums produce duplicates structurally, through repeat questions and through threads being split and merged, and it is not avoidable. What is avoidable is leaving several similar threads in competition so that neither accumulates enough evidence to rank well. The stronger thread keeps the visibility and the others are consolidated toward it.

Our long threads only rank on page one. Can that be fixed?

Often yes, and it is one of the more valuable things to fix on a forum, because the useful answer to a question is frequently several pages in. The problem is usually that pagination is linked inconsistently, canonicalised in a way that collapses the set, or excluded from the sitemap, so the later pages are effectively hidden. Making them properly linked and crawlable does not guarantee they rank, and it does make the substance of the archive reachable at all — which it currently is not.

Do you have experience with forums specifically?

Yes, and the engagements are documented on the case study pages rather than restated here — which is deliberate, because those pages carry the measurement windows and have passed the client approval process, and re-typing figures into new copy is how an unverified number escapes into the site. You can read the engagements and the recorded results there. The short version is that both were large communities with long histories, which is the situation where the archive is the asset and the indexation model is what decides whether it is findable.

Is a forum even capable of ranking in search any more?

Yes, and forums hold something aggregators do not: the discussion itself, with the back-and-forth that resolves a question. What forums typically lack is the technical handling that makes that content legible — coherent pagination, a deliberate indexation model, and internal linking that connects related discussion. That is an engineering gap rather than a model that no longer works, and it is fixable. Where a community's content is genuinely commoditised and duplicated elsewhere, no amount of engineering changes that, and we would say so.

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.