Noindex vs Robots.txt: Which Keeps a Page Out of Search?

If you want a page to stay out of search results, two tools come to mind first: a noindex rule and a Disallow line in robots.txt. They sound similar, and many site owners use them as if they were interchangeable. They are not. The noindex vs robots.txt question comes down to one idea: robots.txt controls whether a crawler may fetch a URL, while noindex controls whether a page that has been fetched may be stored in the search index.

Getting this wrong has a predictable result. A page you wanted hidden still shows up in search with a bare listing, or a page you wanted indexed vanishes because a leftover rule was never removed. This guide is for site owners, developers and marketers who need to hide a page, tidy up what search engines see, or understand a Search Console status. By the end you will know which tool fits each situation, how to apply it, and how to confirm that it works.

Before you begin

You need a list of the URLs you want to keep out of search results, and a way to change either the page itself (its HTML head or its HTTP response headers) or your robots.txt file. For most sites that means CMS access, a developer, or your hosting control panel.

Allow about an hour for a small site. The change itself is quick. Confirming the result takes longer, because search engines only act on a rule after they next visit the page, and Google notes that a recrawl can take months for some pages.

Before you edit anything, save a copy of your current robots.txt file and note which pages already carry a noindex rule. Both mechanisms can remove important pages from search if you apply them to the wrong URLs. Keep in mind that neither one is a security feature. Anything that must stay private needs a login or a password, not a crawler instruction.

How the two tools differ

Search engines work in stages. A crawler first discovers a URL, then fetches the page, then decides whether to add it to the index and how to show it. Robots.txt acts at the fetch stage. Noindex acts at the indexing stage, and it can only act if the fetch happened.

	robots.txt `Disallow`	`noindex`
Where it lives	One file at the site root	In each page's HTML or HTTP headers
What it controls	Whether a crawler may request the URL	Whether the fetched page may appear in results
Can the URL still appear in results?	Yes, if other pages link to it	No, once the rule has been seen and processed
Works for PDFs and images?	Blocks crawling of them	Yes, using an `X-Robots-Tag` header
Reduces crawling?	Yes	No, the page must still be fetched

Google's own documentation states the key facts plainly. Robots.txt is not a mechanism for keeping a page out of Google, and a blocked URL can still be indexed if other sites link to it, usually showing without a description. To keep a page out of results, Google recommends a noindex rule or password protection. The same documentation explains the catch: if robots.txt blocks a page, the crawler never fetches it, so it never sees the noindex rule, and the page can still appear when other pages link to it.

Bing takes the same position. Its guidance says that when robots.txt does not allow a page, Bing cannot show a description for it, and that adding a noindex robots meta tag to the page source makes the page disappear from its index after it has been recrawled and reprocessed.

Step-by-step instructions

Step 1: Decide what you actually want

Write down the goal for each URL group. There are three common goals, and each points to a different tool.

The page must not appear in search results. Use noindex (or remove the page, or put it behind a login).
The page may appear, but crawlers should not waste time on it. Use robots.txt. Examples are internal search results, calendar pages that never end, or sorted and filtered list views.
The page is private. Use authentication. Neither robots.txt nor noindex stops a person, or a badly behaved bot, from opening the URL.

Many pages need only one of these. If you are unsure, ask what would happen if the URL appeared in a search result. If that is harmless but wasteful to crawl, choose robots.txt. If it would be a problem, choose noindex or authentication.

What success looks like: Every URL group on your list has one stated goal and one chosen tool.

Step 2: Add noindex to pages that must stay out of results

For an HTML page, place a robots meta tag in the <head>:

<meta name="robots" content="noindex">

To target only Google's crawlers, use googlebot as the name instead of robots. Other search engines may read the rule differently, so a Google-only tag will not necessarily affect them.

For files that have no HTML head, such as PDFs, send the rule in the HTTP response instead:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Google treats the meta tag and the header as equivalent. Choose whichever you can set up reliably. In a CMS, look for a per-page "hide from search engines" option before editing templates by hand. The X-Robots-Tag guide explains the header in detail.

What success looks like: Loading the page and viewing its source (or its response headers) shows the rule.

Step 3: Make sure robots.txt does not block those same pages

This is the step people skip. Open your robots.txt file at https://example.com/robots.txt and compare it with your noindex list. If a Disallow rule matches a page that carries noindex, crawlers will not fetch the page and will not see the rule.

For example, this combination does the opposite of what the owner intended:

User-agent: *
Disallow: /private-report/

If /private-report/ also has <meta name="robots" content="noindex">, the tag is never read. Remove the Disallow line, leave the noindex rule in place, and wait for the page to be crawled and dropped. After it is gone from results, you may decide whether to block crawling again.

The robots.txt troubleshooting guide covers how to read and test the file.

What success looks like: No Disallow rule matches any URL that depends on a noindex rule.

Step 4: Use robots.txt only for crawl control

For URL groups you do not mind being known but do not want crawled, add a precise rule. Robots.txt must sit at the root of the host, be UTF-8 text, and use paths that start with a slash:

User-agent: *
Disallow: /search
Disallow: /*?sort=

Rules are case-sensitive, and Google applies the most specific matching rule. Two caveats are worth remembering. Google reads only the group for the most specific matching user agent, so a bot-specific group does not inherit rules from the * group. And Google ignores unsupported fields such as crawl-delay.

Robots.txt is the better choice when your concern is server load or wasted crawling, because a noindex page still has to be fetched. Google's crawl budget documentation says as much: noindex is not a way to manage crawling, since the page is still requested and then dropped. If crawling is your real concern, see the crawl budget guide.

What success looks like: The robots.txt report in Search Console shows the file as fetched, and a test of sample URLs gives the result you expect.

Step 5: Confirm what Google sees

Open the URL Inspection tool in Search Console and inspect a page that should be noindexed. Check the "Indexing allowed?" field, which reports whether indexing is explicitly blocked, for example by a noindex tag. If robots.txt blocks crawling, this field always reads "Yes" because Google cannot see noindex directives, which is a quick way to spot the conflict from Step 3. The "Crawl allowed?" field tells you whether robots.txt lets Google fetch the page.

Run the live test after you make a change. It fetches the page in real time rather than showing the version Google stored earlier. The URL Inspection guide walks through the results.

What success looks like: "Crawl allowed?" shows Yes and "Indexing allowed?" shows the noindex reason for pages you want hidden.

Step 6: Wait, then check the index

Neither tool works instantly. Google only drops a page after it recrawls it and sees the noindex rule. You can ask for a recrawl with the URL Inspection tool, but Google says recrawl timing depends on the page and can be slow. The Page Indexing report lists pages where Googlebot found noindex under the "Excluded by 'noindex' tag" status. If you need a page out of results urgently, use Google's removal tools in addition to noindex.

What success looks like: The URL appears under the noindex status in Search Console, and a search for the exact address no longer returns it after the recrawl.

When to combine them (and when not to)

Using both on one URL is rarely useful. The usual safe pattern runs in sequence: add noindex, let search engines process it, and only then think about blocking crawl. For most pages the second step is unnecessary. A noindexed page costs little to leave crawlable.

A different pattern is perfectly sound: noindex for pages you want removed, robots.txt for separate sections you want crawlers to skip. The two rules cover different URLs, so they do not conflict.

Final verification checklist

Each URL group has a stated goal: hide from results, save crawling, or protect with a login.
Pages meant to stay out of results carry a noindex meta tag or an X-Robots-Tag: noindex header.
No robots.txt Disallow rule matches a URL that relies on noindex.
robots.txt is at the host root, returns a successful status and is plain UTF-8 text.
URL Inspection shows "Crawl allowed? Yes" for noindexed pages.
URL Inspection shows the noindex reason under "Indexing allowed?".
Private pages are behind authentication, not only hidden by a crawler rule.
You have a date to re-check the Page Indexing report.

Common mistakes to avoid

Blocking a noindex page in robots.txt. The crawler never sees the rule. Remove the Disallow line until the page is dropped.
Using robots.txt to hide a page. A blocked URL can still be indexed when other sites link to it. Use noindex or a login instead.
Putting noindex inside robots.txt. Google does not support it there. Use a meta tag or header.
Treating either tool as security. Both are requests to cooperative crawlers. Sensitive content needs authentication.
Leaving a development noindex in place at launch. A site-wide rule from staging can hide a live site. Check the live HTML and headers after every release.
Expecting instant results. Rules are applied after the next crawl. Check again after a few weeks, not minutes.
Using noindex to save crawl time. The page is still fetched. Use robots.txt when crawl load is the concern.
Adding a Google-only tag and assuming every engine obeys it. A googlebot meta name affects Google's crawlers. Use robots when you want the rule to apply broadly.

Frequently asked questions

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing. A blocked URL can still appear in results, usually without a description, if other sites link to it.

Can I use noindex and Disallow together?

You can, but the Disallow prevents the crawler from seeing the noindex, so the combination defeats itself. Use noindex first, wait for the page to be dropped, then decide whether blocking is needed at all.

How long does noindex take to work?

It takes effect after the search engine recrawls the page. Google says this depends on the page and can take months. Requesting a recrawl in URL Inspection may help, and removal tools are available for urgent cases.

Does Bing treat noindex the same way?

Bing says it honours robots meta tags and that a noindex tag will remove a page from its index after the page is recrawled and reprocessed. Bing's guidance also tells owners to allow crawling through robots.txt before using its removal methods.

Is noindex or robots.txt better for PDFs and images?

For keeping them out of results, use an X-Robots-Tag: noindex header on the files, and keep them crawlable so the header can be read. Robots.txt blocks crawling of those files, but a blocked URL can still be indexed if other sites link to it.

Should I noindex pages just to improve my site's quality?

Noindex removes pages from results; it is not a quality fix. Decide page by page whether a page should exist for visitors, and see the thin and duplicate content guide for ideas.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.