X-Robots-Tag Header: Noindex PDFs, Images and Other Files

A robots meta tag works only inside an HTML page. A PDF price list, a spreadsheet export or a folder of product images has no <head> to put a tag in. If you do not want those files to appear in search results, you need another route, and that route is the X-Robots-Tag HTTP header. The server sends it along with the file, and crawlers read it the same way they read a meta tag.

This guide is for site owners, developers and anyone who manages a web server or a CDN. You will learn what the header does, which rules it accepts, how to add it for Apache, Nginx and IIS, and how to test that it is really being sent. You will also learn the biggest trap: a rule in this header is invisible if robots.txt blocks the file, and it removes nothing from the web, because anyone with the link can still open the file.

Before you begin

You need to know which files you want to keep out of search results and where they live on your server. Make a list by file type (for example all .pdf files) or by folder (for example /downloads/internal/). You also need access to the web server configuration, an .htaccess file, a web.config file or the header rules of your CDN. If a hosting provider manages the server, ask them where custom response headers can be set.

Plan for about 30 minutes of work and a few weeks of waiting, because search engines act on the header only after they next crawl each file. Before editing, copy the existing configuration file. A typing error in a server configuration can take a whole site offline, so test on a staging copy where you can.

Decide whether the files should be indexed or private. A noindex header only asks search engines to leave the file out of results. If the file is confidential, protect it with a login. The header is a request to cooperative crawlers, and it does not control who can download the file.

What the X-Robots-Tag header is

The header is part of the HTTP response for a URL. Google's documentation says any rule that works in a robots meta tag can also be sent as an X-Robots-Tag. It is not part of a formal standard, but it is widely used by search engines. A basic response looks like this:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

You can send more than one X-Robots-Tag header in a response, or list several rules in one header separated by commas. Rules with no crawler name apply to all crawlers. You can also put a crawler name before a rule to limit it to that crawler:

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: otherbot: noindex, nofollow

Header names, crawler names and values are not case-sensitive. If rules conflict, the more restrictive one applies. If you set several crawler-specific rules, Google uses the sum of the restrictive ones, so a general nofollow plus a Googlebot-specific noindex becomes noindex, nofollow for Googlebot.

Which rules can you send

Google documents these indexing and serving rules, and all of them work in the header:

noindex keeps the page, media file or resource out of search results.
nofollow asks Google not to follow links on the resource.
none is the same as noindex, nofollow.
nosnippet blocks text snippets and video previews; a static image thumbnail may still appear.
indexifembedded lets content embedded in an iframe be indexed, but only when used together with noindex.
max-snippet:[number], max-image-preview:[none|standard|large] and max-video-preview:[number] limit previews.
notranslate stops Google from offering a translation.
noimageindex keeps images on the resource out of the image index.
unavailable_after:[date/time] removes the resource from results after a date, written in a widely used format such as RFC 822, RFC 850 or ISO 8601. If the value is invalid, the rule is ignored.

Google states that it ignores noarchive, nocache and nositelinkssearchbox. For most file-hiding jobs you need only noindex. For a compact list of directives and when to use each, see the meta robots directives guide.

Step-by-step instructions

Step 1: Choose the files and the rule

Write down the file pattern and the rule. A typical example is "all PDF files: noindex". Another is "everything in /downloads/internal/: noindex, nofollow". Keep the scope as small as you can. A rule that matches too much, such as every file in the site root, can hide pages you want to rank.

Decide whether the files link out to pages you care about. If the files link to other pages and you still want those links followed, use noindex alone rather than none.

What success looks like: You have a precise list of file types or folders and one written rule for each.

Step 2: Check that robots.txt does not block those files

Crawlers find the header only when they fetch the file. Google states that if robots.txt blocks crawling, meta tags and X-Robots-Tag headers are never seen and are ignored. Open https://example.com/robots.txt and look for Disallow lines that match the PDFs or the folder. If any do, remove them for the files you want to noindex. The noindex vs robots.txt guide explains why the two cannot be combined on one URL.

What success looks like: No Disallow line in robots.txt matches the files that carry the header.

Step 3: Add the header on Apache

On Apache, use .htaccess or the main configuration with the mod_headers module enabled. Google's examples use Files blocks. To noindex all PDFs:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

For common image types:

<Files ~ "\.(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

For a single file, place an .htaccess file in the same directory as that file and name it:

<Files "price-list-2026.pdf">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

Reload Apache if you edited the main configuration. .htaccess changes take effect on the next request.

What success looks like: A request for a PDF returns the header, which Step 6 confirms.

Step 4: Add the header on Nginx

On Nginx, use add_header inside a location block in the site configuration. Google gives these examples:

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

location ~* \.(png|jpe?g|gif)$ {
  add_header X-Robots-Tag "noindex";
}

To hit one exact file, use an exact match:

location = /downloads/price-list-2026.pdf {
  add_header X-Robots-Tag "noindex, nofollow";
}

Test the configuration with nginx -t, then reload Nginx.

What success looks like: nginx -t reports a valid configuration and the header appears on the file.

Step 5: Add the header on IIS or ASP.NET hosting

On IIS, response headers are set with the customHeaders element in web.config. Microsoft's documentation explains that custom response headers are returned in every response in the scope where you set them: a site, an application or a directory. The element has no setting for matching file extensions, so the scope comes from where the file sits. The simplest approach is to put a small web.config in the folder that holds the files you want hidden:

<configuration>
  <system.webServer>
    <httpProtocol>
      <customHeaders>
        <add name="X-Robots-Tag" value="noindex" />
      </customHeaders>
    </httpProtocol>
  </system.webServer>
</configuration>

Place that file in, for example, the downloads/internal folder, not at the site root. If you need rules by file extension, add the header in application code (middleware) or at your CDN, where you can match on the path.

What success looks like: Files in that folder carry the header, and files elsewhere do not.

Step 6: Test the real response

Never assume the header is present. Request the headers of one file:

curl -I https://example.com/downloads/price-list-2026.pdf

Look for X-Robots-Tag: noindex in the output and a 200 status. Also test one file that should not have the header, such as an ordinary page, to confirm the rule is not too broad. If you use a CDN or a cache, test through the public address, because the CDN can strip, add or cache headers. In Search Console, the URL Inspection tool's "View crawled page" shows the HTTP response Google received, and "Indexing allowed?" reports a noindex rule when it finds one.

What success looks like: The matching files return the header, other URLs do not, and URL Inspection shows the rule.

Step 7: Wait for recrawl and monitor

Google drops a page after it recrawls it and sees the rule; it states recrawl time depends on the URL and can take months. Check the Page Indexing report for the "Excluded by 'noindex' tag" status, which lists URLs where Googlebot found a noindex rule. Bing's guidance is similar: a noindex rule removes a page from its index after the page has been recrawled and reprocessed. If a file contains sensitive information, do not wait for crawlers. Remove it or protect it right away.

What success looks like: Search Console shows the files under the noindex status over the next crawl cycles.

Final verification checklist

curl -I on a target file shows X-Robots-Tag with the intended rule.
curl -I on a normal HTML page does not show an unwanted noindex rule.
robots.txt does not block the files that carry the header.
The file still returns HTTP 200, not an error or redirect.
The header still appears when you request the file through the CDN or public address.
URL Inspection shows the noindex rule for a sample file.
Confidential files are protected by a login, not only by the header.
A reminder is set to check the Page Indexing report after a few weeks.

Common mistakes to avoid

Blocking the file in robots.txt as well. The crawler cannot read the header. Remove the Disallow line for those files.
Matching too much. A pattern such as .* hides the whole site. Test pages after every change.
Setting the header only on the origin. A CDN may cache old headers or drop new ones. Test the public address.
Using both the header and a meta tag with different values. The more restrictive rule wins, which can surprise you. Keep one source of truth.
Treating the header as security. It is a request to cooperative crawlers. Use authentication for private files.
Using noarchive, nocache or nositelinkssearchbox and expecting an effect in Google. Google ignores them.
Forgetting to remove the header later. If a file should be indexed again, delete the rule and check the live response.
Using an invalid unavailable_after date. Google ignores the rule when the value is not a valid date and time.

Frequently asked questions

What is the difference between X-Robots-Tag and a robots meta tag?

They carry the same kinds of rules. The meta tag lives in an HTML page's head. The header is part of the HTTP response, so it also works for PDFs, images, video and other files that have no HTML.

Can I use X-Robots-Tag on normal web pages?

Yes. Any rule that works in a meta tag works in the header. It is handy when you set rules centrally on the server or CDN rather than editing each template.

Does Bing support X-Robots-Tag?

Bing's help says it honours robots meta tags, and MDN lists Google, Yahoo and Bing as supporting the header. Bing's own page does not describe the header in detail, so check a sample file in Bing Webmaster Tools over the following weeks.

Will X-Robots-Tag remove a file that is already indexed?

Eventually, once the crawler fetches the file again and sees the header. For an urgent removal, use the search engine's removal tool as well, and keep the file reachable so the rule can be read.

Does the header stop people from downloading the file?

No. Anyone with the URL can still request it. Use a login or access rules for files that must stay private.

Can I put different rules for different crawlers?

Yes. Put the crawler name and a colon before the rule, for example googlebot: noindex. Rules without a name apply to all crawlers.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.