Verify Googlebot and Bingbot: Tell Real Crawlers From Impostors

Every request to your website carries a user agent, a line of text that says what software is asking. Search engines put their names in it, and so do scrapers, vulnerability scanners and spammers who want to look welcome. If you decide who may enter based on that text alone, you will either let impostors in or, worse, block the real crawlers that bring your pages into search.

This guide shows how to verify Googlebot and Bingbot properly. It is for site owners, developers and system administrators who read server logs, run a firewall or CDN, or have been told that "Googlebot is hammering the site". You will learn the two checks that work, run them on a real address, and put them to use without breaking legitimate crawling.

A verified address tells you who sent the request. It does not tell you whether the crawler will index your pages or how they will rank. That is a separate question.

Before you begin

Gather IP addresses. Take them from your access logs. If a CDN or proxy sits in front of your server, make sure you have the real client address, not the proxy's. Our log file analysis guide explains how to collect and filter logs.
Tools: a terminal with host or nslookup, or Python 3. Internet access to download the published IP range files.
Time and risk: checking one address takes a minute. Changing firewall rules carries real risk, because a mistake can block genuine crawlers. Make changes on a test rule first, and keep a way to roll back.
Back up your current firewall or CDN rules before you edit them.

How crawlers can be verified

Both engines give you two kinds of checks.

DNS check (reverse, then forward). Look up the host name of the IP address. Check that the name belongs to the search engine's domain. Then look up that name and confirm that it resolves back to the same IP address. The forward step matters because anyone controlling the reverse DNS for their own addresses can set any name they like, but they cannot make the search engine's domain point to their address.
Published IP ranges. The engine publishes lists of address ranges in CIDR format. If the address falls inside a published range, it belongs to that engine.

Use the DNS check for a single address and the range list when you must test thousands of lines.

Step-by-step instructions

Step 1: Pick an address to test

Find a request in your log that claims to be a crawler:

grep -i googlebot access.log | head -n 5
grep -i bingbot access.log | head -n 5

Write down the first field, the client address. Keep a list of the unique addresses for later.

You have succeeded when you have at least one address for each engine.

Step 2: Verify a Googlebot address with DNS

Google's documentation gives this manual method:

Run a reverse DNS lookup on the address.
Check that the returned domain is googlebot.com, google.com or googleusercontent.com.
Run a forward lookup on that domain name.
Confirm that the result matches the original address.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The first command should return a name ending in googlebot.com. The second should return the address you started with. The addresses in this example come from Google's documentation. Replace them with your own.

Google also documents the typical shape of names. Common crawlers look like crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Special-case crawlers resolve to names such as rate-limited-proxy-***-***-***-***.google.com. User-triggered fetchers resolve to ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com.

You have succeeded when the forward lookup returns the original address and the name ends in one of the three domains.

Step 3: Verify a Bingbot address with DNS

Bing's guidance uses the same two-step idea. The reverse lookup should return a name that ends in search.msn.com. A forward lookup of that name should return the original address. Bing's example uses the address 157.55.33.18, whose name is msnbot-157-55-33-18.search.msn.com:

host 157.55.33.18
host msnbot-157-55-33-18.search.msn.com

On Windows, use nslookup in place of host. As with Google, replace the example address with your own.

You have succeeded when the reverse name ends in search.msn.com and the forward lookup matches the original address.

Step 4: Script the DNS check for many addresses

For more than a handful of addresses, use a short Python script. It performs the reverse lookup, checks the domain ending, performs the forward lookup and compares the results.

import socket
import sys

SUFFIXES = {
    "google": (".googlebot.com", ".google.com", ".googleusercontent.com"),
    "bing": (".search.msn.com",),
}

def is_genuine(ip, engine):
    try:
        name = socket.gethostbyaddr(ip)[0].lower()
    except OSError:
        return False
    if not name.endswith(SUFFIXES[engine]):
        return False
    try:
        forward = socket.getaddrinfo(name, None)
    except OSError:
        return False
    return ip in {item[4][0] for item in forward}

if __name__ == "__main__":
    engine, ip = sys.argv[1], sys.argv[2]
    print(ip, "verified" if is_genuine(ip, engine) else "NOT verified")

Run it as python3 dnscheck.py google 66.249.66.1. DNS lookups are slow, so cache the results per address rather than checking every log line. A failed lookup can also be a temporary DNS problem, so recheck an address before you treat it as hostile.

You have succeeded when known-good addresses show as verified and a random address shows as not verified.

Step 5: Use the published IP range files

Google publishes JSON files of address ranges in CIDR format. The page lists these files: common-crawlers.json for crawlers such as Googlebot, special-crawlers.json for crawlers such as AdsBot, user-triggered-fetchers.json and user-triggered-fetchers-google.json for fetchers that act on a user's request, and user-triggered-agents.json. They are served from developers.google.com/static/crawling/ipranges/. Each file has a prefixes list, and each entry holds either an ipv4Prefix or an ipv6Prefix.

Bing also publishes a JSON list, at https://www.bing.com/toolbox/bingbot.json, with the same structure of a prefixes array of ipv4Prefix values.

Compare the log addresses with the ranges:

import ipaddress
import json
import sys

ranges = json.load(open(sys.argv[1]))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in ranges]

for line in open(sys.argv[2]):
    ip = line.strip()
    if ip:
        addr = ipaddress.ip_address(ip)
        ok = any(addr in n for n in nets if n.version == addr.version)
        print(ip, "in published ranges" if ok else "NOT in published ranges")

Download the lists again on a schedule instead of saving one copy for good. The ranges can change, and Google's page does not state how often. Bing's own guidance warns that hard-coding a list of addresses can lead to valid Bingbot requests being blocked with HTTP 403 errors, and suggests watching 4xx errors in the Crawl Information tool in Bing Webmaster Tools to catch this.

You have succeeded when your list of unique crawler addresses is split into "in range" and "not in range".

Step 6: Decide what to do with the impostors

Addresses that claim to be a search crawler but fail both checks are not Googlebot or Bingbot. Typical choices:

Rate-limit or challenge them at your firewall or CDN, using the verified-bot features your provider offers.
Block them if they cause load or scrape content.
Leave them alone if they are harmless and your time is better spent elsewhere.

Always allow verified crawlers. Do not block by user agent alone, and do not block a whole country or hosting provider range without checking that your verified crawlers are not inside it. If crawlers appear in your logs with 403 or 429 responses, read Cloudflare blocking Googlebot or Bingbot, which covers the common causes.

You have succeeded when verified crawler requests return normal status codes and your rule changes affect only unverified traffic.

Step 7: Know which crawlers obey robots.txt

Google groups its crawlers into three categories. Common crawlers, such as Googlebot, always respect robots.txt rules for automatic crawls. Special-case crawlers, such as AdsBot, may or may not respect them. User-triggered fetchers, such as Google Site Verifier, ignore robots.txt rules because a person asked for the fetch. This matters when you see a verified Google address fetching a URL that your robots.txt disallows: it may be a fetcher rather than Googlebot. See robots.txt blocking search engines for what robots.txt controls.

You have succeeded when you can explain why a given verified request does or does not follow your robots.txt.

Final verification checklist

You have tested at least one real log address for each engine by DNS.
The reverse name ends in an approved domain, and the forward lookup returns the same address.
Your crawler address list was compared with a freshly downloaded range file.
Unverified addresses are handled by rate limits or blocks, not by user agent alone.
Verified crawler requests are not blocked by your firewall or CDN.
IP range files are refreshed on a schedule, and nothing is hard-coded permanently.
You know which Google crawler categories ignore robots.txt.

Common mistakes to avoid

Trusting the user agent. It is a claim. Verify the address.
Doing only the reverse lookup. A reverse record can be set by whoever owns the address. Always follow with a forward lookup and compare.
Checking the proxy address. If a CDN or load balancer is in front of your server, the log shows its address. Log and test the real client address.
Hard-coding IP lists. Ranges change. Refresh them or use the DNS check.
Blocking every unknown bot. Real crawlers may use addresses you have not seen before. Test before blocking.
Treating a failed lookup as proof. A timeout or DNS error does not prove an impostor. Retry.
Assuming verified means obedient. A verified user-triggered fetcher can ignore robots.txt by design.

Frequently asked questions

Can I just block everything that is not on the published lists?

You can, but do it carefully. Test the rule against a sample of known-good traffic first, refresh the lists regularly and keep a rollback ready. A blanket rule that goes stale can block real crawlers.

Is the user agent useless, then?

No. It is a useful way to filter logs and see what a client says it is. It is only unreliable as proof of identity.

Why does a verified Google address ignore my robots.txt?

It may be a user-triggered fetcher, which Google says ignores robots.txt rules, or a special-case crawler. Check the reverse name against the patterns above.

Does Google crawl only from US addresses?

Google's locale-adaptive pages documentation says Googlebot's default addresses appear to be US-based but that it also crawls from addresses outside the US, and that you can verify those with reverse DNS. Verify each address rather than assuming a location.

Do other search engines publish similar methods?

Many do, but this guide covers only Google and Bing, whose documentation was checked while writing. Check each engine's own help pages for any others you care about.

Does verifying a crawler help my rankings?

No. It only tells you who is visiting. Allowing genuine crawlers to reach your pages is necessary for crawling, but it does not decide indexing or ranking.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.