Search Engineering

Custom Google News sitemap engineering for fast-moving editorial sites

A news sitemap is small software that most teams write once and then stop watching. We build it against the publish event, validate it before it deploys, and monitor whether the articles inside it are actually being found — because a news entry is only useful for the two days Google allows it.

The problem

A news sitemap decays the moment nobody is watching it

Google's rule is blunt: include only articles created in the last two days, and once they are older than that, either drop the URLs or drop their news metadata. That turns the feed into a moving part. It has to be right at the moment of publication, not right whenever someone last looked at it. Almost every news sitemap we are asked to fix was built correctly once and then drifted — a publication name that no longer matches what Google holds, timestamps taken from when the file was written instead of when the article was published, or a generator that quietly stopped writing when the publishing pipeline was refactored around it.

The sitemap validates cleanly, but new articles still take days to be discovered
news:publication_date carries the sitemap build time rather than the article's first publication time
The publication name does not exactly match the name Google holds for the site
Articles well past two days are still listed, so the entries that matter are diluted among ones that cannot count
The feed stops updating whenever the publishing pipeline is refactored
The file has accumulated well past 1,000 entries and has never been split
Search Console reports sitemap errors that nobody has triaged
An article is missing from Google News and there is no way to tell whether the feed or the article is at fault

Who this is for

The people who usually bring us this problem

Head of SEO / Technical SEO Lead at a publisher

You own the sitemap in Search Console, you can see discovery is slower than it should be, and you suspect the feed rather than the content. You need someone who can read the generator and change it.

Engineering Lead / Platform Owner

The news sitemap is a generator inside your publishing pipeline that no longer has an owner. It runs, it produces a file, and nobody can say confidently what it is deriving its fields from.

Head of Audience Development / Editorial Operations

Publishing volume has grown past what the feed was designed for, and the gap between an article going live and an article being found has widened with it. You need the constraint explained and removed.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

The discovery window closes before the feed is correct

A news entry covers two days. If the feed is a day stale, or a day late to update, the article has already spent half its news life undiscovered — and no amount of subsequent accuracy gets that back.

Editorial output is capped by something nobody edits

Commissioning more articles does not raise the ceiling if the mechanism that announces them is slow, wrong or partial. The output grows and the discovery does not.

The same feed gets re-diagnosed from scratch each time

Without a specification and a monitor, every symptom becomes a fresh investigation into a generator that is four people's history. That cost recurs indefinitely.

A broken feed presents as a content problem

Per-article discovery failures look like weak headlines or thin coverage. Teams respond by rewriting content, which is expensive and does not touch the cause.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Feed architecture and generation

A generator wired to the publish event rather than a cron job sweeping the database. The distinction is what decides whether an article appears in the feed minutes after publication or on the next scheduled run — and whether a backdated or corrected article is handled correctly.

Eligibility filtering

Deciding, in writing, exactly which content types qualify as news: articles, live blogs, galleries, updated stories, syndicated wire copy. Google asks that news sitemaps contain only news; a feed that includes evergreen pages spends its 1,000 entries on URLs that cannot use the metadata.

Publication metadata correctness

news:name must exactly match the publication name Google holds, minus anything in parentheses. news:language takes an ISO 639 code. news:publication_date must be the original publication time, not the time the file was generated — a subtle inversion that produces a feed which validates and still reports the wrong dates.

Validation before deploy

Namespace and schema checks, date-format checks across every accepted W3C form, XML escaping of ampersands in headlines, one news:news per url element, and a well-formedness gate that fails the deploy rather than publishing a broken feed.

Freshness monitoring and alerting

A generator that stops and a site that has not published are indistinguishable from outside. We instrument both — feed age, newest entry age, entry count, and whether the publish event actually reached the generator — so a silent stop surfaces as an alert rather than as an SEO meeting six weeks later.

Diagnosis of missing news URLs

Working backwards from one article that Google is not surfacing: was it eligible, did the publish event fire, did it reach the feed, was the entry correct, was the feed fetched, was the URL canonical and indexable. Each answer is separable, and separating them is most of the work.

Scaling past 1,000 entries

A news sitemap may carry up to 1,000 news:news elements. Past that it must be split, and splitting introduces a sitemap index, a naming convention and a retention policy for the older files. We build that structure before a busy news day forces it.

Newsroom and engineering handover

A written specification, a runbook for the failures that actually occur, and a handover to the team that will own it — so the next refactor does not take the feed down with it.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Start at the publish event
Before touching XML we establish what actually happens when an article goes live: which service knows first, whether the event is reliable, and what happens to scheduled, corrected, updated and unpublished articles. Every later decision depends on this, and it is the step most often skipped because the sitemap looks like the beginning of the problem rather than the end of a pipeline.
Define eligibility in writing
We agree the content types that qualify and, more usefully, those that do not. An explicit exclusion list is what stops the feed slowly filling with archives, tag pages and evergreen guides as new templates are added by people who never knew the sitemap existed.
Generate from the article's own record
Each field is derived from a named source: the publication timestamp from the article record's original publication time, the title from the article's own title, the URL from the canonical. Deriving fields independently is what makes the feed reproducible and makes a future regression diagnosable rather than mysterious.
Validate inside the deploy pipeline
The checks in the capabilities section run as part of the release, on the generated file, against the schema. A feed failure then blocks a deploy instead of reaching Google, which is the only version of this that holds up under pressure.
Monitor freshness rather than validity
A well-formed file that stopped updating yesterday passes every schema check and is worthless. Monitoring is on the dimensions that move: age of the newest entry in the feed, count of entries, time since the generator last ran, and whether that duration is consistent with publishing volume.
Hand over with a runbook
The engagement ends with the receiving team able to answer four questions without us: what the feed contains, how it is produced, what to check when an article is missing, and what to change when the pipeline moves.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Feed specification

The qualifying content types, stated as an explicit include and exclude list
Field-by-field derivation: which record supplies each news: value
The publication-time semantics in use, and how corrected articles are treated
Retention and splitting policy for feeds beyond 1,000 entries
The publication name and language code as submitted to Google, and why

Implementation

The generator itself, wired to the publish event
Sitemap index and file-naming structure where splitting is required
Routing, caching headers and content type for the feed
robots.txt and Search Console submission changes
Migration of any existing feed without a gap in coverage

Validation harness

Schema and namespace assertions against the generated file
Date-format checks across every accepted W3C form
XML escaping checks for headlines containing reserved characters
One-news:news-per-url and entry-count assertions
A deploy-pipeline hook that fails the release rather than publishing a bad feed

Monitoring and runbook

Feed age, newest-entry age and entry-count instrumentation
Alerts on staleness, emptiness and generator silence
A written procedure for diagnosing a single missing article
The escalation path when a feed failure is the platform team's, not the sitemap's
Handover notes for whoever inherits the generator

Under the hood

Architecture and technology

The element set Google requires

urlset with both the sitemap and news sitemap namespaces declared
One news:news element per url element, never more
news:publication containing news:name and news:language
news:publication_date as the article's original publication time in W3C format
news:title carrying the article's own title

Where the feed sits

A dedicated news sitemap, kept separate from the general XML sitemap
Referenced from the sitemap index, listed in robots.txt, and submitted in Search Console
A URL that is stable across releases, because Search Console retains the submitted path
Content type and cache headers that let Google fetch it frequently without hammering the origin

Limits and scaling

1,000 news:news elements per file is a hard ceiling, not a guideline
Beyond that, split into numbered files and describe them in a sitemap index
Split files need their own retention: an old news file eventually contains only ineligible entries
An empty feed is legitimate and produces a Search Console warning, not an error

Placement in the publishing pipeline

Publication event, from the CMS or the publishing service that owns the article lifecycle
Feed regeneration, so the entry exists before or as the article becomes reachable
Deployment of the regenerated file, with validation gating the release
Fetch, where Google retrieves the feed on roughly the same cadence as the rest of the site
Monitoring, closing the loop on whether the entry was correct and whether the file stayed fresh

This is the structure a correctly formed news sitemap takes, using the required elements and a placeholder publication. The two things worth reading closely are the nesting — one url element per news:news element — and news:publication_date, which carries when the article was first published rather than when this file was written.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:news="http://www.google.com/schemas/sitemap-news/0.9">
  <url>
    <loc>https://www.example.com/markets/quarterly-outlook</loc>
    <news:news>
      <news:publication>
        <news:name>Example Publishing Co</news:name>
        <news:language>en</news:language>
      </news:publication>
      <news:publication_date>2026-09-24T09:15:00+00:00</news:publication_date>
      <news:title>Quarterly outlook: what changed this week</news:title>
    </news:news>
  </url>
</urlset>

An example, written from Google's published news sitemap reference rather than redacted from a client feed — no client's real feed is published here. In production the escaping matters as much as the structure: a headline containing an ampersand must be escaped, and the publication name is the one field that has to match Google's own record character for character.

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

Articles are crawled but never indexed

A correct feed is about discovery. If the article is discovered and still not indexed, the constraint is elsewhere.

Crawl and indexation diagnosis

5xx errors are interrupting Google's crawls

Server faults during a crawl window suppress discovery in a way no sitemap can compensate for.

Platform reliability

The general XML sitemap is wrong too

News entries are a special case of a sitemap problem, so start with the whole estate.

Technical SEO

A migration changed every article URL

If the URLs in the feed moved, the feed is the smallest part of the problem.

Website migrations

The article body is rendered client-side

A feed can point at a URL that renders no content to a crawler.

JavaScript SEO

Questions

Frequently asked

Will a news sitemap get our articles into Google News?

No, and it is worth being precise about this. A news sitemap is a way of telling Google which URLs are articles and when they were published, so discovery is faster and the metadata is unambiguous. It does not force indexing, and it does not by itself determine inclusion in Google News, which is a separate set of criteria. Anyone promising that a sitemap produces coverage is describing something the format cannot do.

Why do we have to keep changing it?

Because Google limits news entries to articles created in the last two days. Once an article is older than that you either remove the URL or remove its news metadata, so a news sitemap is not a document you write — it is an output that has to keep being regenerated. That is why we build it against the publish event rather than as a scheduled export.

Should we extend our existing sitemap or keep a separate news sitemap?

Google accepts both. We default to separate, because it lets you see news discovery independently in Search Console — if the feed is mixed into the general sitemap, a change in news discovery is invisible among ordinary crawl data. A separate file also lets the news feed fail without taking the general sitemap with it.

We publish far more than 1,000 articles in two days. Does that break it?

It breaks a single file. The limit is 1,000 news:news elements per sitemap, so past that you split into numbered files described by a sitemap index, with a retention rule for the older ones. That is a structural decision rather than a fix, and it is much cheaper to build before a busy news day than during one.

Can you take over a news sitemap another team built?

Yes, and that is most of this work. We start by reading the generator rather than rewriting it, then rebuild the field derivations against the article records, add validation, and add monitoring. A rewrite is often unnecessary; the feed usually needs two fields corrected and an owner.

How do we know it is working?

Two ways. Internally, a monitor reports feed age, newest-entry age and entry count, so a generator that stops is loud instead of silent. Externally, Search Console separates news discovery from general crawl data when the feed is a distinct file, which is how you compare discovery before and after rather than inferring it.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work