Meta Robots Directives Explained: noindex, nofollow, nosnippet

A meta robots tag is a short line in a page that tells search engines how they may use that page. Used well, it keeps a thank-you page out of results or limits how much text is shown in a snippet. Used badly, it quietly removes your best pages from search. One stray noindex left over from a staging site is a classic launch-day disaster.

This guide lists the meta robots directives that Google supports, what each one does in plain English, which popular ones no longer do anything, and how rules combine when they conflict. It is for site owners, developers and SEO leads who need to choose the right directive and prove it works. The facts come from Google's robots meta tag documentation, checked on the day of writing.

By the end you will be able to read any robots tag on your site, write the right one for a given goal, and test that Google sees it.

Before you begin

You need to be able to edit the <head> of your pages (or the HTTP headers of your server), view the rendered HTML of a page, and open Search Console if you have it. Before changing a template, list which page types should be indexable and which should not. Take a backup of the template. A site-wide change to a robots tag affects every page that uses it.

Be clear about one distinction. Robots meta tags control indexing and display. robots.txt controls crawling. They are different tools, and mixing them up is the number one cause of "my noindex does not work". The choice between them is covered in noindex vs robots.txt: Choose the Right Way to Keep Pages Out of Search.

The syntax

The tag goes in the <head>:

<meta name="robots" content="noindex, nofollow">

name="robots" addresses all crawlers that honor it. To address only Google's text results, use name="googlebot". For Google News, use name="googlebot-news". Google says other user-agent values in the meta tag are ignored, and crawlers that are not search crawlers, such as AdsBot-Google, need their own tag.

Rules are separated by commas. You can put several in one tag or use several tags. The name and the content, and the rules themselves, are case-insensitive.

For files that have no <head>, such as PDFs, images and videos, send the same rules in an HTTP header instead:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Google also accepts a user agent name before the rules in the header, and rules without one apply to all crawlers.

The directives Google supports

Directive	What it does
`all`	No restrictions. This is the default, so writing it has no effect.
`noindex`	Keeps the page (or media or file) out of search results.
`nofollow`	Google does not follow the links on the page.
`none`	Same as `noindex, nofollow`.
`nosnippet`	No text snippet or video preview. A static image thumbnail may still appear.
`indexifembedded`	Lets Google index the content when it is embedded in another page, such as in an iframe, even though the page has `noindex`. Only works together with `noindex`.
`max-snippet:[number]`	Caps the text snippet at that many characters. `0` equals `nosnippet`; `-1` lets Google choose.
`max-image-preview:[setting]`	Image preview size: `none`, `standard` or `large`.
`max-video-preview:[number]`	Caps video preview length in seconds. `0` allows at most a static image; `-1` means no limit.
`notranslate`	Do not offer translation of the result's title link and snippet.
`noimageindex`	Do not index images on this page.
`unavailable_after:[date]`	Stop showing the page after that date and time.

Google says nosnippet applies to web search, Images, Discover, AI Overviews and AI Mode, and that it also stops the page being used as direct input for AI Overviews and AI Mode. If your goal is to limit how your text appears, that is the directive to read about closely.

For unavailable_after, Google accepts date formats such as RFC 822, RFC 850 and ISO 8601, and ignores an invalid value. It also says Googlebot reduces its crawl rate after that date.

Directives Google does not use

Plenty of older advice still lists these. Google's documentation says they no longer do anything for Google Search:

noarchive: it no longer controls cached links, because that feature no longer exists.
nocache: not used by Google Search.
nositelinkssearchbox: it no longer does anything, because the sitelinks search box no longer exists.

You may leave them in place if other tools or search engines read them, but do not rely on them to change what Google shows. Other search engines have their own supported lists, so check each engine's own documentation before relying on a directive there. This guide only states what Google documents.

Step-by-step instructions

Step 1: Decide the goal for each page type

Why: the right directive depends on the goal. Write one line per page type.

Should it appear in results at all? If not, use noindex.
Should it appear, but without a long text snippet? Use max-snippet or nosnippet.
Should the links on it not be followed? Use nofollow, sparingly.
Should a PDF or image stay out of results? Use X-Robots-Tag: noindex in the response header.

How to confirm: every page type has one written goal and one chosen rule.

Step 2: Make sure the page can be crawled

Why: Google can only see a robots tag when it fetches the page. Google's documentation says that if robots.txt blocks crawling, the meta tag or header is never read and its rules are ignored. A blocked page can still appear in results if other pages link to it.

Open /robots.txt and check that no Disallow rule matches the URLs where you want noindex to work. Remove the block if you want the page deindexed. The diagnosis steps are in Robots.txt Blocking Google or Bing? Diagnose It Step by Step.

How to confirm: the URL is allowed in robots.txt, and fetching it returns the page.

Step 3: Add the tag or header

Why: this is the actual change. In a template, add one tag in the <head>:

<meta name="robots" content="noindex">

To reduce snippet length and allow large image previews at the same time:

<meta name="robots" content="max-snippet:160, max-image-preview:large">

Google does not enforce placement in the <head> and also respects robots meta tags in the body, but keeping it in the head is clean and predictable. For a PDF served by your web server, configure the server to add X-Robots-Tag: noindex for that file type.

How to confirm: view the source of the live page and find exactly the tag you intended.

Step 4: Check how rules combine

Why: conflicts resolve toward the stricter rule. Google says that when rules conflict, the more restrictive one applies, so max-snippet:50 together with nosnippet gives nosnippet. If you use both a generic robots tag and a googlebot tag, the negative rules are combined: a generic nofollow plus a googlebot noindex means Googlebot sees noindex, nofollow.

Search your templates, SEO plugin settings and server config for every place a robots rule can be set. Plugins, themes and CDNs can add their own.

How to confirm: the final rendered tags and headers, taken together, express exactly one intent for the page.

Step 5: Test with Google's own view

Why: you want to know what Googlebot received. Use the URL Inspection tool in Search Console to fetch the page and look at the HTML Google got. The Page Indexing report shows pages where Google found a noindex rule.

How to confirm: the inspection shows your tag in the HTML and reports the page as excluded by noindex when you intended that. If the page is still indexed, Google may not have revisited it yet. Google notes that less important pages can take months to be recrawled. You can ask for a recrawl, as described in Request Google Indexing the Right Way.

Step 6: Audit for accidental noindex

Why: leftover tags are common after a launch or a theme change. Crawl a sample of your important pages and look for noindex or none in both the HTML and the response headers. Crawvix includes a check named "No accidental noindex" for sampled pages.

How to confirm: every page that should rank has no noindex, none or restrictive snippet rule that you did not choose.

Excluding parts of a page from snippets

If you want to keep one block of text out of snippets rather than the whole page, use the data-nosnippet attribute. Google says it works on span, div and section elements, is a boolean attribute (so any value is ignored), needs valid HTML with properly closed tags, and should not be added or removed with JavaScript after load. If you insert such elements dynamically, include the attribute when you first add them. Structured data inside a data-nosnippet element can still be used for search results.

Final verification checklist

The robots meta tag or X-Robots-Tag header appears in the live page or response, exactly as intended.
Any page where noindex must work is not disallowed in robots.txt.
No important page carries noindex, none or an unintended restrictive snippet rule.
Conflicting rules from plugins, templates and headers have been removed.
Files such as PDFs that must stay out of results use the X-Robots-Tag header.
URL Inspection shows the rule in the HTML Googlebot received.
You are not relying on noarchive, nocache or nositelinkssearchbox to change anything in Google Search.

Common mistakes to avoid

Blocking in robots.txt and expecting noindex to work. Google never sees the tag on a blocked page. Allow crawling until the page has dropped out.
Using noindex in robots.txt. Google does not support a noindex rule in robots.txt. Use the meta tag or header.
Leaving staging rules on production. Check the live site after every launch.
Writing nofollow everywhere. It stops Google following the page's links. Use it only where you mean it.
Expecting noarchive to hide a cached copy. Google says that feature no longer exists.
Adding robots rules through JavaScript only. Prefer putting the rule in the delivered HTML or header so it does not depend on rendering.
Misreading max-snippet:0. It equals nosnippet, so it removes the text snippet.

Frequently asked questions

What is the difference between noindex and nofollow?

noindex keeps the page out of search results. nofollow tells Google not to follow the links on the page. They are independent, and none means both.

How long does noindex take to work?

Google has to recrawl the page to see the tag. Its documentation says less important pages can take months. You can request a recrawl with URL Inspection, but you cannot force a time.

Can I use noindex on a PDF?

Yes. A PDF has no <head>, so send X-Robots-Tag: noindex in the HTTP response header for that file.

Does nosnippet stop my page ranking?

Google describes it as a rule about snippets and previews, not about whether the page is indexed. A page with nosnippet can still be indexed, but it will not show a text snippet.

Do meta robots tags control which pages get crawled?

No. They are read after a page is fetched. To control crawling, use robots.txt. To control what happens after fetching, use robots tags.

Is the robots meta tag case-sensitive?

No. Google says the name, content, rules, header name and user-agent names are case-insensitive.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.