What is crawl budget?

Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given period. It is shaped by how much your server can handle and how much Google wants to crawl your content. Google says it matters mainly for very large or fast-changing sites, and most small sites do not need to worry about it.

Why it matters

If a large site wastes crawling on duplicate, broken or endless URLs, important new pages can wait longer to be fetched. For a typical small business site with a few hundred pages, crawl budget is not the limiting factor, and time is better spent on content and links.

How to check

Understand the two parts. Crawl capacity is how fast Google can crawl without straining your server. Crawl demand is how much Google wants to crawl, based on how popular your pages are and how often they change.
Decide whether it applies to you. Google's guidance is aimed at sites with a very large number of pages, such as a million or more, or sites where a lot of content changes quickly.
Open Search Console, go to Settings and select Crawl stats. The report shows requests per day, response times and the status codes Google received.
Look at the breakdown by response. A high share of 404, 5xx or redirect responses suggests wasted crawling.
Look for sources of endless addresses: filters, sort orders, session identifiers, calendars and internal search pages.
Reduce waste by fixing broken links, removing redirect chains, consolidating duplicates with canonical tags and keeping low-value URLs out of your sitemap.
Keep the server fast. Quicker, steadier responses let Google crawl more with the same effort.

Server logs show every Googlebot request in detail if you want a closer look than Search Console gives.

Common mistakes

Worrying about crawl budget on a small site with a healthy structure.
Blocking pages in robots.txt in the belief that it makes Google crawl the rest more. It can help for truly worthless areas, but it is not a general fix.
Using noindex to save crawling. Noindex pages still have to be crawled to be read.
Leaving redirect chains and soft 404 pages in place on a large site.
Chasing crawl budget when the real issue is that pages are thin, duplicated or not linked from anywhere.

Frequently asked questions

Can I increase my crawl budget?

Not directly. You can make the site faster and more useful, so Google is willing to crawl more, and you can stop wasting what it does spend on duplicates and errors.

Does a sitemap raise crawl budget?

No. A sitemap helps discovery and gives hints about changes. It does not give you a larger allowance.

Do my pages count against a limit?

There is no fixed quota published for each site. The effective amount changes with your server's health and Google's interest in your content.

To see how crawler-friendly your site is, Try the free website sample.