Crawl Budget Explained: When It Matters and When It Does Not

"Crawl budget" is one of the most overused phrases in SEO. Forum threads blame it for every page that is slow to appear in search, and plenty of advice tells owners of small sites to optimise it. In reality, Google describes crawl budget as a concern for specific kinds of large or fast-changing sites, and says most other sites do not need to worry about it.

This guide is for site owners and developers who want a straight answer. You will learn what crawl budget is, the two things that set it, how to tell whether you are in the group that should care, and what to do if you are. If you are not, you will learn what to check instead. Nothing here promises that more crawling means more traffic. Crawling is a precondition for indexing, not a ranking factor.

Before you begin

You need access to Google Search Console for your site, ideally as a Domain property or a root URL-prefix property, because those are the properties where the Crawl Stats report is available. You also need a rough count of how many unique, indexable pages your site has and how often they change. Your CMS, database or sitemap can give you the number.

If you can reach server logs, they help, but they are optional. The work is mostly reading reports and removing wasteful URLs, so the risk is low. The main risk is the cleanup itself: blocking or removing URLs you actually want indexed. Keep a list of what you change.

What crawl budget actually is

Google defines the budget as the combination of two things.

Crawl capacity limit is how much crawling your server can handle without being overloaded. Google says every site starts with the same default, conservative limit. The limit rises when the site stays fast and healthy and falls when responses slow down, return 5xx errors or return 429 rate-limit responses. The limit is shared across all of Google's crawlers.

Crawl demand is how much Google wants to crawl your site. It depends on the inventory of URLs Google knows about, how popular they are and how stale the stored copies are. Google also notes that a site move can temporarily raise demand.

Google may crawl less than your server could handle when demand is low. That is why a fast server alone does not "buy" more crawling. Google also treats each hostname as its own site for this purpose, so www.example.com and blog.example.com are measured separately.

Does crawl budget apply to your site?

Google gives rough estimates, not hard thresholds, for who should read its crawl budget guide:

Sites with about a million or more unique pages whose content changes about weekly.
Sites with about 10,000 or more unique pages whose content changes daily.
Sites where many URLs in Search Console are marked "Discovered - currently not indexed".

Google adds that sites without large numbers of rapidly changing pages do not need this guide, and that for most such sites, keeping the sitemap current and watching the Page Indexing report is generally enough.

If a small site has pages that are not indexed, crawl budget is rarely the explanation. Look at quality, duplication, internal links and technical blocks first. The Crawled - currently not indexed and Discovered - currently not indexed guides cover those statuses in detail.

Step-by-step instructions

Step 1: Count your URLs and how often they change

Make three numbers: how many unique pages you want indexed, how many of those change daily or weekly, and how many total URLs a crawler can reach. The third number is often much larger than the first because of filters, sort options, session IDs and calendar pages. A gap between the first and third numbers is the first sign of wasted crawling.

What success looks like: You have a page count, a change rate and a rough estimate of reachable URLs, and you know which side of Google's size estimates you fall on.

Step 2: Read the Crawl Stats report

In Search Console open Settings, then Crawl stats. Google says the report exists only for root-level properties and that sites with fewer than a thousand pages usually do not need it. The report shows:

Total crawl requests, including failed requests and requests for resources such as images and scripts.
Total download size and average response time.
Crawl responses by type, such as 200, redirects, 4xx, 5xx and DNS or connection errors.
File type, crawl purpose (discovery of new URLs or refresh of known ones) and Googlebot type.
Host status, covering robots.txt fetching, DNS resolution and server connectivity over the past 90 days.

Google notes that some requests may not be counted, so numbers can differ slightly from your server logs, and that example URLs are samples, not complete lists.

What success looks like: You can say what share of requests are errors, redirects or non-HTML files, and whether host status shows any issue.

Step 3: Fix availability problems first

Capacity falls when your server struggles. In Crawl Stats, look at host status and the share of 5xx responses and slow response times. A rising error rate or a robots.txt fetch failure is more urgent than any URL cleanup. Google's robots.txt rules explain why: if robots.txt returns a 5xx or cannot be reached, Google stops crawling the site for the first 12 hours and keeps retrying. See the HTTP 500 and 503 guide for how to find the cause.

Faster responses also help efficiency. Google lists speeding up server responses and supporting 304 Not Modified replies, which let it reuse unchanged pages, as ways to make crawling more efficient.

What success looks like: Host status shows no recent issue, and 5xx responses are rare.

Step 4: Shrink the pool of useless URLs

Google's list of best practices for managing URL inventory is short and practical:

Consolidate duplicate content.
Block URLs you do not want crawled with robots.txt.
Return 404 or 410 for pages that are permanently gone.
Fix soft 404 pages (see the soft 404 guide).
Keep sitemaps current, with lastmod on pages that have actually been updated.
Avoid long redirect chains.

For online shops and other filtered lists, the biggest source of waste is usually parameters and filters. The faceted navigation guide shows how to handle them. Remember that robots.txt must be precise: a rule that is too broad blocks pages you want, as the robots.txt guide explains.

What success looks like: The number of reachable URLs falls toward the number of pages you actually want indexed.

Step 5: Avoid the wrong fixes

Google lists tactics that do not do what people expect:

Do not use noindex to manage crawling. Google still requests the page and then drops it, so crawl time is spent anyway.
Do not use robots.txt to temporarily free up budget. Google only reallocates budget to other pages if it is already hitting its capacity limit.
Do not rely on blocking for removed pages. Blocked URLs stay in the crawl queue longer, and a 404 or 410 is a stronger signal not to recrawl.

Also remember that Google ignores crawl-delay in robots.txt, so it will not slow Googlebot.

What success looks like: Your cleanup uses 404 or 410 for removed pages and robots.txt only for URL groups you want crawlers to skip.

Step 6: Let Google know what changed

A current XML sitemap with accurate lastmod values helps Google find new and updated pages. Do not set lastmod to today's date on every build; use it only when a page's content truly changed. If your site is very large, a sitemap index keeps files within limits; see Split Large Sitemaps into a Sitemap Index. If Google reports "Hostload exceeded", Google's guidance is to add server resources, which is a hosting decision, not an SEO tweak.

What success looks like: New pages appear in your sitemap with accurate dates and are discovered without manual requests.

Step 7: Re-check after a few weeks

Return to Crawl Stats after Google has recrawled. Compare the share of 200 responses, redirects and errors with your starting numbers. A change in the mix of responses is the signal to look for. Do not expect a sudden rise in total requests, because crawl demand sets the ceiling as much as capacity does.

What success looks like: A higher proportion of requests go to pages you want indexed, and fewer go to redirects, errors and duplicates.

Final verification checklist

You know your unique page count and how often those pages change.
You know whether you meet Google's rough size estimates for crawl budget guidance.
Crawl Stats shows no recent host status issue for robots.txt, DNS or server connectivity.
5xx and 429 responses are rare in the Crawl Stats response breakdown.
Permanently removed pages return 404 or 410, not 200.
Duplicate and parameter URLs are consolidated or blocked on purpose.
The sitemap lists only canonical, indexable pages with honest lastmod dates.
You wrote down every robots.txt or redirect change you made.

Common mistakes to avoid

Optimising crawl budget on a small site. If you have a few hundred pages, look at content quality and links first.
Using noindex to reduce crawling. The page is still fetched. Use robots.txt for URL groups that should not be crawled.
Blocking removed pages in robots.txt. Serve 404 or 410 so crawlers can see that the page is gone.
Setting every lastmod to the build date. If the dates are not trustworthy, they are less useful as a signal.
Ignoring server errors. Slow responses and 5xx errors reduce capacity, so fix them before fine-tuning URLs.
Reading Crawl Stats as a ranking report. It shows how Google fetched your site, not how it ranks.
Writing broad robots.txt rules without testing them. A pattern that matches more than you meant can block pages you want. Test sample URLs before and after the change.
Assuming these figures apply to other search engines. Google's estimates and report names are Google's. Check each search engine's own documentation.

Frequently asked questions

What is a crawl budget in simple terms?

It is the amount of crawling Google is willing and able to do on your site. Google describes it as the combination of how much your server can handle and how much Google wants to crawl.

Do small websites need to worry about crawl budget?

Usually not. Google says sites without many rapidly changing pages do not need its crawl budget guide, and that a current sitemap and the Page Indexing report are generally enough.

Can I increase my crawl budget?

Google's two levers are capacity and demand. You can add server resources if you see "Hostload exceeded", and you can improve the content users find valuable and unique. There is no setting that requests a larger budget.

Does crawl-delay in robots.txt slow Googlebot?

No. Google ignores unsupported robots.txt fields, including crawl-delay. Reduce the crawl rate through Google's documented options if your server needs relief.

Does a 404 page use crawl budget?

Any request uses some crawling. Google says a 404 or 410 is a stronger signal than a block that the URL should not be recrawled, and that already indexed 404 URLs are removed and crawl frequency for them gradually decreases.

Is the Crawl Stats report available for every property?

No. Google says it is available for root-level properties only, meaning a Domain property or a URL-prefix property at the root.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.