Search Infrastructure

XML sitemaps that are correct rather than merely present

Most sites have a sitemap. Far fewer have one that is complete, accurate and generated from the same source as the pages it describes — and a sitemap that is wrong is worse than none, because it teaches a crawler to distrust the signal.

The problem

A sitemap is an assertion, and wrong assertions are costly

A sitemap says: these URLs exist, they are canonical, here is when each last changed, and this is all of them. Every one of those is a claim a crawler will test. If the URLs are not canonical, if lastmod is a timestamp of the last build rather than the last change, or if whole page types are missing, the sitemap stops being a discovery aid and becomes noise the crawler learns to discount.

The sitemap is generated by a plugin with default settings
Some page types are absent — pagination, archives, facets, records in a custom system
URLs appear that are not canonical, or that redirect
lastmod is the build time rather than the content's change time
The sitemap is over the size or count limits and is not split
New content is not discovered promptly
Search Console reports issues with the sitemap, or slow discovery
Different page types are managed by different systems with no coordination

Who this is for

The people who usually bring us this problem

A large site where the sitemap is auto-generated

The plugin produces something and nobody has established whether it is complete or correct.

A site with several content systems

Different page types come from different places and no single sitemap reflects the estate.

A site where new content is slow to be discovered

Publication happens and indexing follows days or weeks later, which points at discovery.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

A sitemap that disagrees with the site is a liability

Non-canonical URLs, redirecting URLs and dead addresses in a sitemap waste crawl and, more importantly, undermine the reliability of the signal. A crawler that has been sent wrong information stops treating the file as authoritative — which is exactly when it would have been most useful.

lastmod is either trusted or ignored

An accurate lastmod is one of the few signals that makes a large site's re-crawling efficient, because it lets the crawler prioritise what actually changed. A lastmod that changes on every build, or never changes, is detected and discounted within a short period — and once discounted it does not come back easily.

A sitemap generated separately from the pages will drift

If the sitemap is assembled by a different process than the one that produces the routes, the two will disagree — usually quietly, usually at the edges: a new page type, a retired section, a paginated set. Generating both from one source is the only version of this that stays correct without ongoing attention.

Discovery is not the same problem as indexation

A complete and accurate sitemap improves discovery. It does not make a page indexable, and it does not override a canonical decision or a rendering problem. Expecting a sitemap fix to solve a coverage problem is a common and expensive misdirection.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Sitemap estate audit

What exists, what it contains, what it omits, and how it is generated. On a site with several systems or a plugin history this is where the missing page types and the non-canonical entries are found.

Index structure for large estates

A sitemap index over child sitemaps split by page type and by the size and count limits the protocol sets, so that each file is coherent and a failure in one does not affect the rest.

Complete page-type coverage

Every URL class the site intends to be found by: content, section indexes, pagination, author and date archives, tag and category pages, records in a custom system, and whichever of the ones generated by facets are meant to be indexed. The decision about which of these SHOULD be in the sitemap is part of the work.

Canonical agreement

Every URL in the sitemap is the canonical one for its content, matching the canonical the page declares and the address the redirects resolve to. Disagreement between these three is common on sites that have migrated, and it is checkable.

Accurate lastmod

The time the content actually changed, stored with the content rather than derived from the build, and emitted only where it is meaningful. Where a reliable change time does not exist for a page type, omitting lastmod is better than emitting a wrong one.

Generation from the same source as the routes

The sitemap produced from the same data that produces the pages, so a new page type cannot appear on the site without appearing in the sitemap. This is the structural answer and it removes the class of drift entirely.

Splitting, compression and reference correctness

Limits respected with headroom, large files compressed where the protocol allows, and `robots.txt` and internal references pointing at the index rather than a child file. Small details, each of which can make a sitemap ignored.

Verification against the live site

Every URL in the sitemap requested and its response checked against the claim — status, canonical, and agreement with the site's own structure. A sitemap audit that only validates the XML has checked the syntax and not the assertion.

Discovery measurement

Whether new content is found faster as a result, measured through crawl and indexing timings rather than assumed. It is the benefit the sitemap exists for and the one least often verified.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Decide what should be in the sitemap before generating it
Every URL class, and whether it is meant to be discoverable in its own right. Paginated sets, facets, archives and thin variants are decisions rather than defaults — a sitemap containing everything a site can generate is not more complete than one containing what should be found.
Compare the sitemap against the site, both directions
What is in the sitemap and not on the site, and what is on the site and not in the sitemap. The second direction is where missing page types are found, and it requires the site's own inventory rather than the sitemap's.
Check every URL's claim, not the file's syntax
Request each URL and confirm the status, the canonical and the agreement with the sitemap's assertion. Well-formed XML containing wrong information passes every validator and helps nobody.
Emit lastmod only where it is real
A stored content-change time, or nothing. A build timestamp is worse than an absent element, because it is a wrong signal that is specifically designed to be trusted and is therefore discounted once it is not.
Generate from the route source
The structural fix: one source of truth produces the pages and the sitemap, so drift cannot occur. Where that is not possible immediately, the alignment is checked on a schedule instead of assumed.
Measure discovery, not the file
How quickly new content is crawled and indexed before and after, because that is the outcome the work is for. A correct sitemap that made no difference to discovery is a correct sitemap and not a result.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Audit

What the current sitemap estate contains, omits and claims
Every URL class the site serves, with a decision on each: included or deliberately not
Non-canonical, redirecting and dead URLs currently referenced
lastmod accuracy, per page type
Size and count against the protocol's limits
Where generation happens and how it relates to the route source

Engineering

Sitemap index over child sitemaps split by page type
Generation from the same source as the pages
Accurate lastmod, stored with the content, omitted where no reliable value exists
Canonical agreement across sitemap entry, page declaration and redirect
Limits respected with headroom, references pointing at the index

Verification and measurement

Every URL requested, with status and canonical confirmed
Sitemap compared against the site in both directions
Search Console sitemap and coverage review
Discovery timing for new content, before and after
What remains omitted and why

Under the hood

Architecture and technology

What a sitemap can and cannot do

Can: improve discovery, communicate accurate change times, expose page types that internal linking under-connects
Can: reduce crawl waste by describing the estate precisely
Cannot: make an unindexable page indexable
Cannot: override a canonical decision the page itself declares
Cannot: fix a rendering problem that hides content from a crawler
Cannot: compensate for a crawl constraint such as server reliability

The page types a sitemap usually misses

Pagination beyond the first page of a listing
Records in a custom system with its own routing
Section and hub pages generated by a template rather than authored
Archives by date, author or taxonomy
Pages from a second system or an acquired site
Anything added after the sitemap generator was configured

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

The content is time-critical and needs freshness

The news-specific case: a rolling window and per-article eligibility.

News sitemap development

Pages are in the sitemap and still not indexed

Discovery is not the constraint; something else is.

Crawl and indexation

New content is slow to be found

Internal linking, crawl rate and the paths a crawler actually takes.

Technical SEO implementation

Structured data alongside the sitemap

Describing the entities, not only the addresses.

Structured data implementation

Questions

Frequently asked

Do we even need a sitemap?

For a small, well-linked site it matters less, because a crawler will find everything by following links. It becomes valuable in specific situations: a large estate where crawling everything is expensive and an accurate lastmod lets the crawler prioritise; page types that internal linking under-connects, such as records in a custom system or deep pagination; a site whose internal link graph is thin or JavaScript-dependent; and a new or recently migrated site where the linking history does not yet exist. If none of those apply, a correct sitemap is still worth having and it is not the highest-value work on the list.

Should every URL be in the sitemap?

No, and including everything is a common mistake. Paginated sets, facets, thin archive pages and near-duplicate variants are all URLs the site can generate and not all of them should be indexed in their own right. The sitemap is where that decision gets expressed, and it should agree with the site's canonical and robots decisions. A sitemap containing everything the application can produce is not more complete — it is a set of assertions the site does not actually make.

Our plugin has a sitemap option. Is that enough?

It generates a file, and whether it generates a correct one depends on the site. Plugin defaults handle the common page types and the standard WordPress structures, and they frequently miss whatever is specific to the site: records in a custom system, pagination configured in a particular way, a second content source, or a page type added after the plugin was set up. They also usually derive lastmod from the post's modified time, which is close to right for posts and wrong for anything assembled. The audit's job is to establish which of those applies rather than to assume either answer.

Will fixing the sitemap improve our rankings?

Indirectly and not reliably. A sitemap affects discovery and crawl efficiency, which can mean new and changed content is found and re-evaluated sooner. That is a real benefit on a large or fast-moving site and it is not the same as improving position. If pages are already discovered promptly, a better sitemap will change very little — and if pages are discovered promptly and not ranking, the constraint is somewhere else. We will say which of those the measurement shows rather than presenting a tidy file as an outcome.

How do we know the sitemap is accurate over time?

The structural answer is to generate it from the same source that generates the pages, so a page type cannot exist on the site without existing in the sitemap. That removes drift rather than monitoring it. Where the sitemap must be assembled from somewhere else, then it needs a scheduled comparison against the live site in both directions, because the failures are quiet — a new section that never appears, a retired one that stays — and nobody notices either from the file itself.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work