Log File Analysis for SEO: See How Crawlers Use Your Website

Most SEO tools show you what a crawler of their own finds on your site. Your server log shows something different and more direct: every request that actually reached your server, including those from Googlebot and Bingbot, with the time, the address requested and the status code you returned. Log file analysis for SEO means reading that record to answer questions such as "Which URLs does Googlebot request?", "Is it hitting error pages?" and "Is it spending time on addresses I never wanted indexed?"

This guide is for site owners, developers and SEO practitioners who have access to their own web server or hosting logs. It is practical rather than theoretical: you will locate the logs, isolate crawler requests, check that they are genuine and run a handful of commands that surface real problems.

A caution up front. Logs tell you what crawlers requested. They do not tell you what was indexed or how pages rank. Treat what you find as evidence about crawling, not as a measure of search performance.

Before you begin

Access: you need to be able to download or read the access logs for your site. On shared hosting this is often a control-panel download. If a CDN or proxy sits in front of your server, you may need the logs from that service too, because the origin server may only see the proxy.
Retention: log files grow quickly, and many servers rotate and delete them. Apache's documentation says an access log typically grows 1 MB or more per 10,000 requests, so check how many days your server keeps. Aim to collect at least a few weeks.
Privacy: logs contain visitors' IP addresses. Handle them according to your privacy obligations, keep them off shared drives and delete working copies when you are done.
Tools: a terminal with grep and awk, or a spreadsheet for small files, and Python if you want to verify IP addresses in bulk.
Time and risk: an afternoon. This is read-only work on a copy of the logs, so it carries no risk to the live site. Never edit the originals.

What a log line contains

Most servers can write the Combined Log Format. Apache documents it as the Common Log Format plus the referrer and the user agent. The fields appear in this order: client address, an identity field that is usually -, user name, time, the request line in quotes, status code, response size in bytes, referrer and user agent. nginx's predefined combined format records the same information. An example line:

66.249.64.5 - - [10/Oct/2026:13:55:36 +0000] "GET /guides/a?x=1 HTTP/1.1" 200 2326 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

On IIS, the default format is W3C Extended. Microsoft documents that the fields are chosen in the logging settings, that the default categories include date, time, client IP, method, URI stem, query, status, time taken, user agent and referrer, and that W3C timestamps are recorded in UTC. Read the #Fields line at the top of each IIS file to see the order of columns, because they differ between servers.

Step-by-step instructions

Step 1: Locate and collect the logs

Find where your server writes access logs.

nginx: the access_log directive sets the path. Its documented default is logs/access.log using the combined format, though distribution packages usually point to /var/log/nginx/access.log.
Apache: the CustomLog directive sets the file and format. A relative path is relative to the server root.
IIS: the default directory is %SystemDrive%\inetpub\logs\LogFiles, with a new file created daily by default.
Hosting panels and CDNs: look for "raw access logs", "log download" or a log export feature.

Copy a set of files covering at least two to four weeks into a working folder and decompress them if needed.

You have succeeded when you have a folder of unedited log copies and know which time zone they use.

Step 2: Check that the format has what you need

Open a few lines. You need the client IP, the time, the requested path including the query string, the status code and the user agent. If your server logs the proxy's address rather than the visitor's, you cannot verify crawler IPs from that file; Apache notes that the logged address may belong to a proxy, not the originating machine. In that case, configure the proxy to pass the real client address and log it, or use the proxy's own logs.

You have succeeded when a sample line shows a real client address, a path and a user agent.

Step 3: Isolate crawler requests

Filter by user agent to get the requests that claim to be a search crawler:

cat access.log* | grep -i googlebot > googlebot.log
cat access.log* | grep -i bingbot > bingbot.log
wc -l googlebot.log bingbot.log

Remember that this is a claim, not proof. Anyone can send a Googlebot user agent. Google's documentation says verification is useful because spammers and other troublemakers may claim to be from Google. The next step handles that.

You have succeeded when you have one file per crawler and a rough count of requests.

Step 4: Verify that the requests are genuine

Google documents two methods. The manual method is a reverse DNS lookup on the address from your log, checking that the result is in googlebot.com, google.com or googleusercontent.com, then a forward lookup on that name to confirm it returns the original address. The automatic method compares the address with Google's published IP range files, which use CIDR notation. For Googlebot and other common crawlers the file is common-crawlers.json.

Extract the unique addresses and compare them with the ranges. This script reads the downloaded JSON file and a text file with one address per line:

import ipaddress
import json
import sys

ranges = json.load(open(sys.argv[1]))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in ranges]

for line in open(sys.argv[2]):
    ip = line.strip()
    if ip:
        addr = ipaddress.ip_address(ip)
        ok = any(addr in n for n in nets if n.version == addr.version)
        print(ip, "in published ranges" if ok else "NOT in published ranges")

awk '{print $1}' googlebot.log | sort -u > ips.txt
python3 verify_ips.py common-crawlers.json ips.txt

For Bingbot, Bing's help page lists the same options: a reverse DNS check (the name should end in search.msn.com), a published JSON list and a verification tool. The full procedure for both engines is in our guide to verifying Googlebot and Bingbot. Remove unverified addresses from your crawler files before you analyse them.

You have succeeded when every address left in your Googlebot file is in the published ranges or passes the DNS check.

Step 5: Read the status codes

Count how many crawler requests received each status code (in the Combined format, the status is the ninth field when paths contain no spaces):

awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn

Interpret the result using Google's documented behaviour:

200: normal. Check that these are pages you want crawled.
301 and 302: each hop in a redirect is a separate request. Many redirects mean crawlers are following old links. See redirect chains and loops.
404 and 410: Google treats them as not existing and crawls such URLs less often over time. A few are normal. Many from the same pattern point to broken internal links or old sitemap entries.
5xx and 429: Google documents that these cause it to temporarily slow its crawling. Look for the time and URL pattern, and read HTTP 500 and 503 errors.
401 and 403: Google says not to use these to limit crawling. If crawlers get them, a firewall or login wall may be blocking them by mistake. See Cloudflare blocking Googlebot or Bingbot.

You have succeeded when you can state what share of crawler requests ended in each status class.

Step 6: Find where crawl attention goes

List the most requested paths:

awk '{print $7}' googlebot.log | sort | uniq -c | sort -rn | head -25

Then look for patterns you did not intend:

awk '$7 ~ /\?/ {print $7}' googlebot.log | sort | uniq -c | sort -rn | head -25
awk '$9 == 404 {print $7}' googlebot.log | sort | uniq -c | sort -rn | head -25

Query-string URLs from filters, sort options and session identifiers are a common source of unnecessary requests. Our faceted navigation guide explains how to control them. Compare the list with your sitemap: important URLs that never appear in the log are worth investigating, and so are URLs that appear often but are not in the sitemap. Whether crawl volume matters for a site of your size is covered in crawl budget explained.

You have succeeded when you have a short list of unexpected URL patterns and 404 sources.

Step 7: Cross-check with Search Console and act

Google's Crawl Stats report in Search Console shows total requests, download size, response time, host status, response types, file types, crawl purpose and Googlebot type. It is available for root-level properties only, and Google describes it as aimed at advanced users; sites with fewer than a thousand pages usually do not need it. Google also notes the report may not count every request, so its totals can differ from your server logs. A mismatch is expected and does not mean either is wrong.

Turn your findings into changes: fix broken links, remove dead URLs from the sitemap, add redirects for old addresses that are still requested and reduce parameter-based URLs. Repeat the analysis a few weeks later to see whether the pattern changed.

You have succeeded when each finding has an owner and a fix, and you have noted the date to re-run the analysis.

Final verification checklist

You have unedited copies of logs covering several weeks.
You know the time zone and format of the log files.
Crawler requests were filtered by user agent and then verified by IP range or DNS.
You have counted status codes for verified Googlebot and Bingbot requests.
You have listed the most requested URLs and the most requested 404 URLs.
You have looked for query-string URL patterns.
You compared at least one finding with the Crawl Stats report or your sitemap.
Working copies containing IP addresses are stored securely or deleted.

Common mistakes to avoid

Trusting the user agent. It can be faked. Verify addresses before drawing conclusions.
Analysing too short a period. A few days may miss weekly patterns or a crawl burst. Use several weeks.
Counting CDN or proxy addresses as crawlers. If the log shows a proxy, you cannot verify the crawler. Fix the logging or use the proxy logs.
Ignoring time zones. IIS W3C logs are in UTC, and other servers may use local time. Convert before comparing with an incident time.
Confusing crawling with indexing. A URL in the log was requested. It may not be indexed. Use Search Console for index status.
Chasing every 404. A few are normal. Focus on repeated patterns, links on your own pages and important URLs.
Editing the original files. Always work on copies.

Frequently asked questions

Do I need log analysis for a small website?

Usually not. Google says the Crawl Stats report is aimed at advanced users and that sites under a thousand pages typically do not need it. Log analysis becomes more useful for large sites, sites with many parameter URLs, and when you are investigating a specific crawling problem.

Why do my logs not match the Crawl Stats report?

Google notes that some requests may not be counted and that totals can differ from your server logs. Differences in time zone, caching and CDNs add to the gap. Look for patterns in both rather than expecting identical numbers.

Can I tell which pages Google has indexed from logs?

No. Logs show requests. Indexing is a separate step. Use the page indexing report and the URL Inspection tool in Search Console.

Which log format is best?

The Combined format is widely supported and contains what you need. nginx also supports custom formats, including JSON through log_format with the escape=json parameter, which can be easier to process with scripts.

How long should I keep logs?

Long enough to analyse several weeks of crawling, and no longer than your privacy policy allows. Remember that logs contain IP addresses.

Does a high crawl rate mean good SEO?

No. More requests do not mean better results, and a request pattern full of errors or redirects is a problem rather than a success.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.