Blocked by robots.txt in Search Console: How to Fix It Safely

You open the Page Indexing report in Google Search Console and find a row labelled "Blocked by robots.txt". Or you find its close relative, "Indexed, though blocked by robots.txt". Both mean the same underlying thing: your robots.txt file tells Googlebot not to crawl these URLs. What you should do about it depends entirely on whether you meant to block them.

This guide is for site owners and developers who need to work out which situation they are in. You will learn how to list the affected URLs, find the exact rule responsible, decide whether it is right or wrong, change the file without causing a bigger problem, and confirm the fix in Search Console. It will not tell you to remove every block. Many of these entries are correct, and the status is information, not always an error.

Before you begin

You need access to Search Console for the property, and the ability to edit your live robots.txt file. That may mean your CMS, an SEO plugin, your hosting file manager, or your deployment process. Open https://example.com/robots.txt in a private browser window and save a copy of what it says now. If your site uses several hostnames, remember that each host needs its own file, because Google applies the rules only to the exact protocol, host and port where the file is served.

Set aside about an hour. Edits are quick, but confirming results takes days. Google says validation typically takes up to about two weeks, and sometimes longer. The risk is real in both directions: unblocking the wrong section can expose pages you meant to keep out of the crawl, and leaving a mistaken block can keep important pages out of the index.

What the two statuses mean

Google's help page for the Page Indexing report defines them this way.

Blocked by robots.txt means your robots.txt file prevents Googlebot from crawling the page. Google adds that a block does not rule out the page being indexed through some other means. If the page should be indexed, remove the rule. If you want it kept out of search, remove the block and use a noindex rule instead.

Indexed, though blocked by robots.txt means Google indexed the URL even though robots.txt blocked crawling. It can happen because other pages link to the URL, and the listing usually has a very limited snippet. The fixes are the same: unblock it if it should be crawlable, or unblock it and add noindex if it should stay out of results.

The reason for that last fix is in Noindex vs Robots.txt. A crawler can only read a noindex rule on a page it is allowed to fetch.

Step-by-step instructions

Step 1: List the affected URLs

In Search Console open Indexing, then Pages. Under "Why pages aren't indexed" click "Blocked by robots.txt". Look at the example URLs. The report lists example URLs, so look for patterns in the paths you see: a folder, a parameter such as ?sort=, a file type.

If you see "Indexed, though blocked by robots.txt", look at it separately, since those URLs are already in the index.

What success looks like: You have a list of URL patterns, not just a count.

Step 2: Sort the URLs into intended and unintended

Go through the patterns one by one and ask whether each URL deserves to be crawled.

Intended blocks. Internal search results, shopping carts, login pages, endless calendar pages and filtered list views are common reasons to block crawling. Leave these alone. Google's guidance for such URLs is that validation may not make sense when robots.txt blocks them on purpose.
Unintended blocks. Product pages, articles, category pages, images or scripts your pages need should not be blocked. These are the ones to fix.
Unclear. Check with whoever owns the content before changing anything.

What success looks like: Each pattern is marked keep blocked, unblock, or unblock and noindex.

Step 3: Find the exact rule

Open the live robots.txt file and read every group. Remember how Google reads it:

A file has groups, each starting with a User-agent line, followed by Allow and Disallow rules.
Google uses only the group for the most specific matching user agent. A group for Googlebot is not merged with the group for *.
The most specific rule, meaning the longest matching path, wins. If an Allow and a Disallow are equally specific, the less restrictive Allow applies.
Paths are case-sensitive, and * and $ work as wildcards.

A rule such as Disallow: /products blocks /products/, /products/shoes and also /products-sale, because it matches anything starting with that text. That pattern, an unintended prefix match, is the most common surprise.

The robots.txt report in Search Console does not show which rule blocks a specific URL. Use the URL Inspection tool to confirm the URL is blocked (the "Crawl allowed?" field reports whether robots.txt allows crawling), then read the file yourself. Developers can also run Google's open-source robots.txt library locally against the file.

What success looks like: For each unintended block you can name the exact line that causes it.

Step 4: Check where the file really comes from

Sometimes the file you edit is not the file that is served. A plugin, a CMS setting, a CDN or a hosting platform may generate or override it. Open the public URL, not a file on disk, and compare. If the public file differs from your repository, find the layer that produces it before editing. Also check that you are looking at the right hostname. The www and non-www versions, and any subdomains, can serve different files.

If a development rule such as Disallow: / was left in place at launch, remove it from the source that deploys the file, or it may come back with the next release. The robots.txt troubleshooting guide covers this case, and if a security layer rather than your file is stopping Googlebot, see Cloudflare Blocking Googlebot or Bingbot.

What success looks like: You know which system serves the live robots.txt and where to change it.

Step 5: Edit the rule narrowly

Change as little as you can. Three common fixes:

Narrow a broad rule so it matches only what you mean:

# Before: blocks everything starting with /products
Disallow: /products

# After: blocks only the internal filter pages
Disallow: /products/filter/

Add an Allow that carves out an exception inside a blocked folder:

User-agent: *
Disallow: /downloads/
Allow: /downloads/public/

Remove the line entirely if nothing under it should be blocked.

Keep the file UTF-8 plain text at the root of the host. Do not add noindex lines to robots.txt, because Google does not support noindex there. For pages you want out of search results, unblock them and add a noindex meta tag or X-Robots-Tag header; the X-Robots-Tag guide covers files that have no HTML head.

What success looks like: The edited file is saved and the public URL shows the new content.

Step 6: Test before and after

Open the public file in a private window to confirm it is served. In Search Console, the robots.txt report shows the files Google found, the fetch status, when they were last checked and the number of parsing issues. Google says errors prevent a rule from being used, while warnings do not. Then inspect one affected URL with the URL Inspection live test. "Crawl allowed?" should now say Yes. The URL Inspection live test guide explains the other fields.

Note how Google treats the file's availability: a 4xx response (other than 429) is treated as no robots.txt, meaning no restrictions, while a 5xx response or network error makes Google stop crawling the site for the first 12 hours and then fall back on its last good copy. A broken deployment can therefore have large effects.

What success looks like: The live test says the URL can be crawled and the robots.txt report shows the file as fetched without errors.

Step 7: Ask for a recrawl and validate

Google caches robots.txt generally for up to 24 hours. After a critical change, such as unblocking important URLs, you can open the robots.txt report, use the more settings icon beside the file and choose "Request a recrawl". Google says you generally do not need to, and that a request does not mean the unblocked URLs are recrawled immediately.

Then go to the issue in the Page Indexing report and press "Validate fix" once every intended fix is made. Do not press it again until the validation succeeds or fails. If it fails, open "See details", fix the URL that caused the failure and start a new validation. For the URLs that should stay blocked, you do not need to validate.

What success looks like: The validation completes, and fixed URLs move out of the blocked list on later checks. They are crawlable at that point, but this does not mean Google will index them.

Final verification checklist

Every URL pattern in the report is marked keep blocked, unblock, or unblock and noindex.
You identified the exact robots.txt line behind each unintended block.
The public robots.txt file, not a local copy, contains your edit.
The file returns a successful status and shows no errors in the robots.txt report.
URL Inspection shows "Crawl allowed? Yes" for a sample of fixed URLs.
Pages that must stay out of search carry noindex and are not blocked in robots.txt.
You pressed "Validate fix" once, after all fixes.
The deployment source for robots.txt is updated so the fix will not be overwritten.

Common mistakes to avoid

Deleting every rule to clear the report. Intended blocks, like cart and search pages, can stay. Remove only the mistaken ones.
Editing a copy that is not served. Check the public URL after every change.
Using a broad prefix rule. Disallow: /blog also blocks /blog-archive. Add a trailing slash or a more specific path.
Blocking a page you want noindexed. The noindex rule is never read. Unblock, then noindex.
Forgetting other hostnames. Each host and port needs its own file.
Overlooking blocked files. Google notes that blocked images, videos and PDFs are excluded from crawling. If the report lists files you want shown in search, review those rules too.
Assuming unblocking means indexing. It only allows crawling. Indexing depends on content and other signals.
Pressing Validate fix repeatedly. Wait for the first validation to finish.

Frequently asked questions

Is "Blocked by robots.txt" always a problem?

No. It is a problem only for URLs you want in search. Cart, login and filtered pages are often blocked on purpose.

What does "Indexed, though blocked by robots.txt" mean?

Google has indexed the URL, usually from links on other pages, although it cannot crawl it. The listing normally has a very limited snippet. Unblock it if you want it crawled, or unblock it and add noindex to remove it from results.

How long until Google sees my new robots.txt?

Google generally caches the file for up to 24 hours. You can request a recrawl in the robots.txt report. Validation of the report entry typically takes up to about two weeks.

Can I test a rule before publishing it?

You can run Google's open-source robots.txt library locally against a draft file. Search Console's robots.txt report works only for files that are already live on your site.

Does robots.txt hide pages from search?

No. Google says it is not a mechanism for keeping a page out of Google. A blocked URL can still be indexed if other sites link to it. Use noindex or a login.

Why does the report show URLs I already fixed?

Reports lag behind crawling. The fix must be crawled and validated. If the entries remain after the validation window, recheck the live file and the exact URL pattern.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.