Search Console Crawl Stats Report: How to Read It and Spot Problems

Most Search Console reports tell you what Google decided about your pages. The Search Console crawl stats report tells you what happened when Googlebot asked your server for them. Did the server answer? Was robots.txt reachable? Did DNS resolve? How many requests ended in a redirect, a 404 or a server error? It is one of the few places where you see your site from the crawler's side.

It is also easy to misread. This guide explains each section in plain terms, shows which patterns deserve attention and which usually do not, and gives a short routine. It is written for site owners, developers and technical SEOs with access to a Search Console property.

It will not tell you how to make Google crawl more, because Google says you cannot request an increase in crawl rate. It also does not show rankings or traffic. For those, use the Performance report.

Short answer: The crawl stats report shows how Googlebot's requests to your site went over the last 90 days: how many, how fast, which response codes, which file types, and whether robots.txt, DNS and your server were available. Open Settings in a root-level property, choose Crawl stats, and start with the host status section.

Before you begin

You need a verified Search Console property. The report is only available for root-level properties: a Domain property such as example.com, or a URL-prefix property at the root such as https://example.com. A property for a subfolder will not show it. If you are unsure which type you have, see Domain Property vs URL-Prefix Property in Search Console. If you have no property yet, start with Set Up Google Search Console: Complete Website Owner Checklist.

Google describes this report as aimed at advanced users and says sites with fewer than about a thousand pages generally do not need it. A small site can still use it after an outage or a migration, but the Page indexing report is usually the better first stop; see Page Indexing Report Explained.

Have your server logs or hosting dashboard to hand to confirm any suspicion. Reading the report changes nothing, so there is no risk.

What does the Search Console crawl stats report show?

The report opens with three summary figures and then breaks the requests down by category.

Item	What it counts	Easy-to-miss detail
Total crawl requests	Every crawl request for your URLs, successful or not	Includes failed robots.txt, DNS and connection attempts, and each duplicate request
Total download size	Bytes downloaded while crawling	A resource shared by many pages is fetched once, not per page
Average response time	Mean time across all fetched resources	Each linked resource (script, image, stylesheet) counts separately

Below the summary are the breakdowns: by response, by file type, by purpose, by Googlebot type, and host status. Each shows shares of requests, not of URLs or bytes. A 404 share of 4% means 4% of requests, not 4% of your pages.

The data covers the selected property's domain. Requests Googlebot makes to other domains, including images hosted elsewhere, are not shown, and requests between subdomains appear under the parent domain. For a URL-prefix property, both http and https requests are included in the numbers. Redirect hops count one by one, and client-side redirects are not counted.

How do I read the host status section?

Host status answers a simple question: could Google reach you at all? It combines three details, each with a chart, and the report flags a day as a problem when a chart crosses a dotted red line. Google's example is DNS failures above 5% of requests on a day.

Detail	What it measures
robots.txt fetching	How often requests for robots.txt failed
DNS resolution	Whether the DNS server failed to recognise or answer for your hostname
Server connectivity	Whether the server was unresponsive or returned an incomplete response

The overall status takes one of three values: no significant availability issues in the past 90 days; some availability issues, but not recently (at least one significant issue in the 90 days, but more than a week ago); or a recent availability issue (at least one in the last week). A recent issue is a prompt to check whether it recurs, not proof of damage.

The robots.txt line deserves attention because its failure mode is unusual. A 200 response counts as a success whether the file is valid, invalid or empty, and so do 403, 404 and 410, which mean there is no file. A 429 or any 5xx counts as a failure. Google's page explains that, when robots.txt cannot be fetched, Googlebot stops crawling the site for the first 12 hours but keeps asking for the file, then uses the last copy it fetched successfully for up to 30 days. After that it either crawls without restraint if the home page is available, or stops crawling if it is not. In every case it keeps retrying. See Robots.txt Blocking Google or Bing? Diagnose It Step by Step if the file itself is the problem.

Step-by-step instructions

Step 1: Open the report for the right property

Open Search Console, select the root-level property, click the gear icon for Settings, and choose Crawl stats. If the item is missing, you have selected a property that is not at the root.

What success looks like: You see the crawl requests chart with a 90-day range and the breakdown tables underneath.

Step 2: Check host status first

Read the overall status, then open each of the three details. Note which day each issue happened. If robots.txt fetching or server connectivity failed, check your hosting records for the same dates: a deployment, a traffic spike, a certificate expiry or a firewall change are the usual candidates. DNS failures point at your DNS provider or at a recent nameserver change; see How to Point a Domain to Hosting.

What success looks like: You can say, for each flagged day, what changed on your side or that you found nothing.

Step 3: Read the response breakdown

The "by response" table groups requests as a share of the total. Google sorts the groups like this:

Google's grouping	Examples
Good	OK (200), moved permanently (301 or 308), moved temporarily (302 or 307), moved by meta refresh, not modified (304)
Possibly good	Not found (404)
Bad	robots.txt not available, unauthorized (401 or 407), server error (5xx), other client error (4xx), DNS unresponsive, DNS error, fetch error, page could not be reached, page timeout, redirect error, other error

Google notes that 404s are often correct and need not all be fixed. Concern belongs on server errors, timeouts, fetch errors and redirect errors, and on a 404 share that rises without a reason you can name. Click a response type to see example URLs. Google says these examples are not comprehensive, so a missing URL does not prove Googlebot did not request it.

If the examples show 5xx, start with HTTP 500 and 503 Errors: Find the Real Website Problem. If they show many redirects, see Redirect Chains and Loops. If real pages return 200 but show an error message, that is a soft 404; see Soft 404 Errors: What They Mean and How to Fix Them.

What success looks like: You know the largest "bad" group, if any, and have looked at its example URLs.

Step 4: Look at file types and purpose

The "by file type" table shows HTML, image, video, JavaScript, CSS, PDF, other XML, JSON, syndication feeds (RSS or Atom), audio, geographic data, other file types (redirects are counted here) and unknown (failed requests). Percentages are shares of responses, not bytes. For rendering questions, see JavaScript SEO.

The "by purpose" table splits requests into discovery (a URL Googlebot had not crawled before) and refresh (a recrawl of a known page). Neither ratio is good or bad by itself.

What success looks like: You can explain why the file-type and purpose mix looks the way it does.

Step 5: Check the Googlebot type breakdown

The "by Googlebot type" table separates Smartphone, Desktop, Image, Video, Page resource load (secondary fetches for images, CSS and similar files needed to render a page), AdsBot, StoreBot and other agent types. Google states AdsBot crawls about every two weeks and that spikes there often come from Dynamic Search Ads targets. If you expected mostly smartphone crawling and see mostly desktop, review Mobile-First Indexing Checklist.

What success looks like: The dominant crawler type matches what you expect for your site.

Step 6: Compare with your logs and act

Totals in the report can differ from your server logs. Google says some "page could not be reached" requests never reach your server, and that when robots.txt is unavailable for long, it counts crawls it might have made. For genuine verification of who is crawling, see Verify Googlebot and Bingbot and Log File Analysis for SEO.

If you find a fixable cause, fix it, then re-open the report after a few days. The chart changes only as new crawls happen.

What success looks like: You have either a fix and a date to recheck, or a note that no action is needed.

Which patterns point to a real problem?

Google's own troubleshooting notes on the page map common patterns to causes. Use them as starting points, not diagnoses.

What you see	Causes Google lists	First check
Crawl requests spike	You unblocked a section, added a large section, or added Dynamic Search Ads targets	Did a robots.txt, release or feed change happen that day?
Crawl requests drop	New broad robots.txt rules, slow responses, rising server errors, or content that is low-change or low quality	Compare with host status and average response time
Crawling seems low	Google says you cannot ask for an increase; for small and medium sites it suggests keeping sitemaps current and removing blocks	See Crawl Budget Explained
Crawl rate seems too high	Reduce it using Google's crawl-rate guidance	Return 500, 503 or 429 only briefly

On that last point, Google's separate guidance on reducing crawl rate says to use 500, 503 or 429 only for a short time, such as a couple of hours or one to two days, because longer can mean pages are dropped from the index. It adds that Googlebot will discover fewer new pages and refresh existing ones less often while those codes are returned. To catch problems sooner, see Website Uptime Monitoring: Catch Downtime Before Search Engines Do.

A worked example

Suppose host status shows "Recent availability issue", with server connectivity above the red line on Tuesday and a jump in 5xx responses. Your hosting dashboard shows a deploy that morning. You roll back, confirm key pages return 200, and re-open Crawl stats on Friday: no new spike after Tuesday. The status may still read "Recent availability issue" until a week has passed, because it is defined by the last seven days. You have removed the cause; whether Google crawls more afterwards is its decision.

Crawvix's scan covers some of the same ground from outside, such as whether robots.txt is reachable and whether sampled pages return server errors, but it only sees what a visitor sees at the moment of the scan. The report remains the only view of Googlebot's own experience.

Final verification checklist

You opened the report on a root-level property and the 90-day range loads.
Host status is read, and any flagged day matches a known event or is investigated.
robots.txt fetching shows no repeated failures (429 or 5xx).
The largest "bad" response groups have been opened and their example URLs reviewed.
404 share is understood and does not rise without cause.
File-type and purpose mixes are plausible for your site.
Any difference from server logs is explained or noted.
A date is set to re-check after each fix.

Common mistakes to avoid

Opening it from a subfolder property. The report needs a root-level property. Switch to the Domain or root URL-prefix property.
Treating every 404 as a fault. Google calls 404 "possibly good". Fix those that should exist; leave removed pages alone.
Reading percentages as URL counts. Shares are of requests, so one heavily requested URL can dominate.
Expecting the status to clear immediately. "Recent availability issue" covers the last week; it will not change the moment you fix the cause.
Using 503 or 429 to slow Googlebot for weeks. Google says to keep it short; long runs can lead to dropped URLs.
Assuming a missing example URL was never crawled. Examples are representative, not complete.

Frequently asked questions

Where is the crawl stats report in Search Console?

It sits in the property's Settings, under Crawl stats. It only appears for root-level properties, either a Domain property or a URL-prefix property at the root.

How far back does the crawl stats report go?

Google's page says host status covers the past 90 days. Save screenshots or notes of any numbers you want to compare over a longer period.

Why do the crawl stats not match my server logs?

Google lists several reasons: some requests are not counted yet, some never reach your server, and counts include crawls it might have made while robots.txt was unavailable. Small differences are expected.

Does a drop in crawl requests mean a penalty?

Not necessarily. Google's listed causes are mostly technical or about content change: new robots.txt rules, slow responses, more server errors, or content that rarely changes. Check the Manual Actions report separately; see Search Console Manual Actions.

Official references

Next step: Try the free website sample to see how your own site does on a few of these checks. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.