A crawl-rate decline is almost never caused by what it looks like. The symptom arrives in a search-console report and the instinct is to look at what changed in the content. In practice the cause is usually one of three things below the content layer, and the measurements that distinguish them take an afternoon rather than a quarter.
Step 1: establish that it actually fell
Establish a rate per day over a window long enough to contain the change, and compare against the same period in the previous year if the site is seasonal. A crawl rate that fell 30% in a week where the site published 40% less content has not fallen at all — the ratio has held. Normalise requests against the number of indexable URLs before concluding anything, because growth in the URL space dilutes rate without any reduction in crawl attention.
Step 2: response-code distribution over time
This is the single most informative measurement and the one most often skipped. Group crawler requests by status code, by day. What you are looking for is not the absolute error rate but whether it changed at the same time the crawl rate did.
# Splitting by day is what makes this a diagnosis rather than a snapshot.
awk '$0 ~ /Googlebot/ {
match($4, /\[[0-9]{2}\/[A-Za-z]{3}\/[0-9]{4}/)
day = substr($4, RSTART+1, 11)
codes[day][$9]++
}
END {
for (d in codes) {
printf "%s", d
for (c in codes[d]) printf " %s=%d", c, codes[d][c]
printf "\n"
}
}' access.log | sort
A 5xx rate of 2% sounds tolerable in a monthly average and is frequently fatal in a specific hour. Search engines respond to error rates as a reliability signal, and they respond to the *pattern* — a site that fails predictably at the same time each day is assessed differently from one that fails randomly at the same rate.
Step 3: latency, not just errors
Crawl rate is bounded by response time as much as by errors. A site that starts responding in 4 seconds where it used to respond in 400ms will be crawled less, with no errors anywhere. Compare the 95th percentile rather than the mean: a mean is dominated by the fast majority of requests, and the tail is what a crawler experiences on the pages that matter. This is also the measurement that catches an unindexed database column, a cold cache tier or a new synchronous dependency added by an unrelated feature release.
Step 4: rule out a directive change
Check robots.txt, meta robots and X-Robots-Tag for change against a version-controlled history. This is fast, it is occasionally the entire answer, and it is worth eliminating before spending time on the slow measurements. Two specific patterns are worth checking even if the file looks unchanged: a directive added to a header by a reverse proxy or CDN rather than in application code, and a block on a path prefix that also matches set-level URLs, which reduces crawl rate by removing internal link discovery rather than by blocking the pages themselves.
Step 5: check internal link depth to the priority URLs
A crawl rate can hold steady while discovery collapses, because crawlers still return for known URLs and stop finding new ones. Measure the shortest click depth from the homepage to a sample of priority pages, and compare against a previous crawl. If depth increased — a pagination change, a navigation restructure, a JavaScript-rendered menu — the crawler is spending its budget on the same pages repeatedly rather than reaching new ones.
Step 6: only now look at content
Content is the last hypothesis rather than the first. If the five measurements above are clean, then the question is whether the site published a large volume of near-duplicate or low-value URLs that diluted crawl attention across a growing estate. That is a real cause and a common one on programmatically generated sites, but it presents differently — rate per URL falls while total requests hold — and it is distinguishable from the causes above rather than a catch-all.
What recovery actually looks like
Crawl rate recovery lags the fix, and it lags by longer than most teams expect. A search engine has to observe sustained reliability before it increases its commitment, so the sequence is: errors stop, then the rate begins to recover over weeks. Set the expected window before making the change, with the number you will accept as evidence it worked. Without that, a fix that is working slowly is indistinguishable from one that did not work, and the instinct at that point is to change something else — which resets the clock.