Disallow in robots.txt tells crawlers not to fetch a URL, while noindex tells search engines not to show a page in results. They do different jobs. To keep a page out of Google, let it be crawled and add noindex, because Google must read the noindex instruction and cannot do so if robots.txt blocks the page.
Why it matters
Many site owners add a Disallow rule and assume the page is gone from search. If other sites link to the address, Google can still list it without ever reading the page. Choosing the right tool prevents pages that linger in results, and prevents pages that you want indexed from disappearing.
How to choose
- Ask what you want. If the goal is to save crawling on an area of low value, use Disallow. If the goal is to keep a page out of search results, use noindex.
- To apply Disallow, add a rule to robots.txt:
User-agent: *
Disallow: /internal-search/
- To apply noindex to an HTML page, put a robots meta tag in the head of the page:
<meta name="robots" content="noindex">
- For files that are not HTML, such as PDFs, send the instruction as an HTTP response header instead:
X-Robots-Tag: noindex
- Confirm that the page is not blocked in robots.txt, then inspect it with the URL Inspection tool in Search Console. The result should say that indexing is not allowed because of the noindex tag.
- Allow time for Google to crawl the page again. The page leaves the results after that crawl.
- If you need to keep people out as well as search engines, add a login. Neither instruction is a security measure.
A quick comparison:
| Disallow | noindex | |
|---|---|---|
| Where it lives | robots.txt | Page or HTTP header |
| What it controls | Crawling | Appearing in results |
| Works if the page is blocked | Yes | No, it must be read |
Common mistakes
- Using both on the same page. The block prevents the noindex from being seen.
- Writing a noindex line inside robots.txt. Google stopped supporting that in 2019.
- Putting noindex pages in your sitemap, which sends mixed signals.
- Relying on either method to protect confidential content.
- Forgetting to remove noindex after a staging site goes live.
Frequently asked questions
Can a Disallowed page still appear in Google?
Yes. Google may show the address, usually without a description, if it learns of the URL from links elsewhere. Disallow stops fetching, not listing.
Which one saves crawl budget?
Disallow does, because the crawler does not request the page. A noindex page still has to be crawled to be read.
Which should I use for a thank-you page?
Usually noindex, so the page stays crawlable but out of results. Disallow is the better fit for large, low-value areas such as endless filter combinations.
Related guides
- noindex vs robots.txt
- Meta robots directives explained
- The X-Robots-Tag HTTP header
- What does a noindex tag do?
To see whether your pages carry the right signals, Try the free website sample.