Search Engineering

What crawlers actually do, from the server's own record

Analytics reports what happened in a browser. A server log reports every request, including the ones that were never rendered — which is where crawl problems live and where no other instrument looks.

The problem

The only complete record of crawler behaviour is on the server

Analytics can only report a page that loaded and executed its script. A crawler that fetched a URL and discarded it, a redirect chain followed on every request, a thousand paginated pages crawled and none indexed, a 5xx that a bot received and a visitor did not — none of that appears in a browser-based tool, and all of it is in the access log. The log is the ground truth, and reading it is the difference between inferring crawl behaviour and knowing it.

Crawl behaviour is being inferred from a search console's summary rather than measured
New content is discovered slowly and the reason is not established
A large share of requests comes from bots and nobody knows which pages they are
Server errors are suspected of affecting crawling and cannot be confirmed
Redirects may be chained and there is no way to tell how often
A site has been restructured and its crawl has not recovered
The site is slow and it is not clear whether that is visitors or crawlers
Nobody can say what proportion of server load is search-engine traffic

Who this is for

The people who usually bring us this problem

A large site with a discovery problem

New content is found late and the cause is not visible in a search console's summary.

A site whose crawl is suspected of being wasted

A large proportion of the URL estate is being fetched and none of it matters.

A site where server reliability and search are connected

Error responses may be reducing crawl rate, and it needs establishing rather than assuming.

What it costs

What this costs while it goes unfixed

Engineering faults are rarely confined to the engineering layer. These are the commercial consequences we see most often.

Logs record requests, not sessions, and that is the point

A browser-based tool aggregates by visitor and by session, which is the right unit for measuring engagement and the wrong one for measuring crawling. A crawler is not a session. The log's unit is the request, and it is the only place where every fetch of every URL by every agent is present without sampling or execution.

Wasted crawl is invisible until it is counted

A site can spend most of its crawl on paginated archives, filtered views, parameter combinations and URLs that redirect — none of which it wants indexed. The proportion is only visible when requests are grouped by URL class, and it is frequently the answer to a discovery problem that looked like a content problem.

Error responses to crawlers do not appear in a search console's totals

A summary reports what was indexed. The log reports what a bot received, including the 5xx responses, the timeouts and the redirect chains that never resolved. Where crawl rate has fallen, that record is where the cause is, and it is attributable to a period rather than inferred from a trend.

The log is the only instrument that can confirm a hypothesis

Every other source produces a signal that could have several explanations. A log produces a count. Where a change has been made and its effect is disputed, the log settles it — which is why reading it is worth doing before a change as well as after.

What we do about it

Capabilities

Each of these is work we carry out, not an area we advise on.

Log format and completeness assessment

What the server records, whether bot requests are captured, whether the log has been rotated and what the retention actually is. A log analysis is bounded by what was kept, and establishing that first prevents a conclusion drawn from an incomplete period.

Crawler identification

Distinguishing verified search-engine crawlers from claimed ones and from other automated traffic, using reverse lookups rather than user-agent strings. A log is full of agents claiming to be a crawler, and treating them all as one produces a crawl picture that is mostly fiction.

Crawl distribution by URL class

What proportion of crawler requests goes to content, to pagination, to filtered or parameterised views, to redirects, to error responses and to URLs that should not be crawled at all. This grouping is the analysis, and it is where wasted crawl becomes a number.

Discovery timing

How long after publication a new URL is first fetched, measured from the log rather than from a console's summary. On a site where discovery is the constraint, this is the metric the whole engagement is judged on.

Response codes as crawlers experienced them

Status codes per URL class and per crawler, including the errors, the redirect chains and the requests that were abandoned. A site can look healthy to a visitor and return errors to a bot, because the two request different things.

Redirect and canonical behaviour in practice

Which redirects are followed repeatedly, how long the chains are, and whether the canonical a page declares is the URL that was actually fetched. Logs show what the site did, which is not always what it was configured to do.

Crawl rate over time

Requests per period by crawler, correlated against site changes, error periods and performance. This is what establishes whether a change improved crawling or whether a decline began at a specific point rather than gradually.

Analysis that survives rotation

Working from retained logs and stating the window explicitly, with the parts that cannot be established from what remains identified as such. A log analysis has a horizon and it is stated rather than implied.

How we work

Engineering methodology

The sequence is deliberate. The order is usually what determines whether the work holds or has to be repeated.

Establish the window before drawing any conclusion
What period the retained logs cover, whether it includes the change being investigated, and whether it is complete. Every finding is bounded by that, and a conclusion drawn from a partial window is the most common error in this work.
Verify crawlers rather than trusting the agent string
Reverse-resolve the addresses and confirm them against the published ranges. An unverified agent string is a claim, and logs contain a great many of them — treating them as crawlers produces a distribution that describes nothing.
Group by URL class before looking at individual URLs
Content, pagination, filters, redirects, errors, and the classes that should not be crawled. A per-URL view is unreadable at scale and the proportion per class is where the actionable finding is.
Measure discovery from the log, not from a summary
The interval between publication and first fetch, per content type. It is the metric that a discovery problem is about, and it is only directly measurable in the log.
Correlate crawl rate with what changed
Requests per period against releases, migrations, error periods and performance incidents. Correlation is not attribution, so the finding is stated as what the record shows rather than as a proven cause.
State the horizon of every finding
The window the analysis covers, what it cannot establish from the retained data, and what would need a longer retention to answer. A log analysis is bounded by what the server kept, and saying so is what makes the rest trustworthy.

Deliverables

What an engagement produces

Documentation is a deliverable, not an afterthought. On most of these engagements a large part of the value is a defect report precise enough for another team to act on.

Assessment

Log format, what is captured, rotation and the actual retention window
Verified crawler traffic, separated from claimed and from other automated traffic
Crawl distribution by URL class, with the proportions
Discovery timing for new content, per type
Response codes as crawlers received them, per class
Redirect chains and canonical behaviour as observed
Crawl rate over the retained window, against site changes

Findings

Where crawl is being spent and where it should be
Which URL classes are consuming crawl without being wanted
Whether errors or redirects are suppressing crawling, and over what period
What the discovery timing is, and what is limiting it
What cannot be established from the retained data, stated plainly

Recommendations

Changes ordered by the crawl they would recover
What to measure afterwards, and on what interval
What retention would be needed to answer the questions the logs cannot
Which findings are the record and which are an interpretation of it

Under the hood

Architecture and technology

What a log shows that analytics cannot

Every request, including the ones that were never rendered
Crawler behaviour, which is not a session and is not measured by session tools
URLs fetched and discarded, and URLs fetched repeatedly
Response codes as bots received them, including errors
Redirect chains as they were actually followed
Crawl rate per period, attributable to a date
The proportion of server load that is automated

What bounds a log analysis

Retention — the horizon of everything the analysis can establish
Whether bot requests are captured at all, or filtered before logging
Whether the site is behind a proxy or CDN that logs elsewhere
Whether the log is sampled, which changes proportions
Whether addresses can be reverse-resolved to verify crawlers
Whether the log includes the hostname, where several sites share a server

Related work

Where we have done this

Engagements where this capability was the substance of the work rather than a line item.

Fintech & capital markets

Search engineering at stockbroking scale

A stockbroking platform publishing at news velocity was losing search visibility to problems that had nothing to do with content quality. Two workstreams ran in parallel: sustaining a high-volume editorial output across business and market categories, and diagnosing the technical faults — a domain safety flag, recurring server errors, and a metadata defect on a templated page type — that were suppressing how much of that output search engines could actually reach.

225Msitewide impressions

All case studies →

Adjacent problems

If this is not quite your problem

These overlap at the edges. Sending you to the right page is more useful than having you work it out.

The crawl constraint is established

What is crawlable, indexable and reachable, and what limits each.

Crawl and indexation

The problem is indexation rather than discovery

Pages are crawled and not indexed, which is a different diagnosis.

Technical SEO audit

Errors are the suspected cause

Reducing 5xx responses and the crawl rate they affect.

SEO server error remediation

The site has declined and the cause is unclear

Establishing which of several possible causes this is.

SEO traffic recovery

Questions

Frequently asked

How long do we need to retain logs for?

Long enough to cover the thing you are investigating, which is the honest answer and frequently longer than the current retention. A change made two months ago cannot be assessed against a log kept for two weeks, and a gradual decline needs a baseline from before it started. For a discovery problem, a month of complete logs is usually enough to establish the distribution; for attributing a change, the window has to include the period before it as well as after. Where retention is the limit, that is stated as the horizon of the analysis rather than worked around.

Do you have log analysis work you can point to?

One published engagement established that recurring server errors and unplanned downtime had reduced crawl rate, which is a log-derived finding and is described on the case study page rather than restated here. What that establishes is that we have read production logs for this purpose; it does not establish a standalone log-analysis service with a portfolio behind it, and this page does not claim one. The analysis itself is described in full above, so you can judge the method rather than the reference.

What can logs tell us that Search Console cannot?

Everything Search Console samples, aggregates or omits. It reports what was indexed; the log reports every fetch of every URL by every agent, including the ones that were discarded, the redirects that were followed, and the error responses a bot received. It also covers the URLs a console never mentions because they are not in its index — which on a site with a large generated estate is most of them. The console is a summary of an outcome; the log is the record of what happened.

Our CDN sits in front of the origin. Where are the logs?

Usually at the CDN, and that is where the analysis has to happen. An origin log behind a CDN records only the requests that reached the origin, which is the opposite of what a crawl analysis wants — most crawler traffic is served from the edge and never appears. Whether the CDN retains request logs, for how long, and in what format is established before the analysis rather than assumed, and where the edge logs are unavailable that is a finding about what can be measured.

Can you tell whether a specific change helped?

Where the log covers the period before and after it, yes — crawl rate and distribution per class, compared across the change. That is the strongest form of evidence available for a search-engineering change, because it is a count rather than a signal with several possible explanations. Where the retention does not reach back far enough, the honest answer is that it cannot be established from what remains, and that is stated rather than inferred from a trend in something else.

Related capabilities and work

Bring us the problem you have not been able to fix

Describe what is happening rather than what you think the cause is. If we are not the right people for it, we will say so.

Discuss your engineering challenge View our work