WordPress Discourage Search Engines Setting: Find It and Fix It

One checkbox in WordPress can keep a finished, public website out of search results: "Discourage search engines from indexing this site". It is useful while you build. It is a problem when it is still ticked after launch, which happens often because the setting is stored in the database and travels along when a staging site is copied to production.

If your new WordPress site is not showing up in Google or Bing, or Search Console says pages are excluded by a noindex tag, this setting is one of the first things to check. The check takes a few minutes. The fix is one click. The part that takes care is making sure nothing else is also telling search engines to stay away, and then asking them to look again.

This guide shows you how to find out whether the setting is on, what it actually changes, how to turn it off, how to confirm that every other source of noindex is gone, and how to stop it coming back. It cannot promise when pages will be indexed. That is up to the search engines.

Before you begin

You need a WordPress administrator login. For the checks you also need a browser, and ideally a command line with curl. Allow thirty minutes. The risk is low: unticking the box only removes a request, but if the site is not ready for the public (placeholder text, test pages), finish that first. If you are unsure, read WordPress SEO Checklist: Settings to Check Before and After Launch.

What the setting does

WordPress documents the option in its Reading Settings screen. When it is ticked, several things happen:

A robots meta tag with noindex is added to the page head. Since WordPress 5.3 the tag is output as noindex,nofollow, according to the core announcement. This is a request that search engines do not show the page in results.
robots.txt is not used for this any more. Before 5.3 WordPress sent Disallow: / from its generated robots.txt. The core team removed that, because blocking crawling can still let a site get indexed: a crawler that cannot fetch a page never sees the noindex tag on it.
Update services are not pinged. The documentation says WordPress stops sending pings to ping services when the option is on.
Core sitemaps are disabled. WordPress's sitemap announcement states that the sitemap feature is switched off when the visibility setting discourages indexing, unless a filter overrides it.
Visitors can still get in. WordPress states that the option does not block access to the site; it relies on search engines to honor the request.

Two limits are worth knowing. WordPress cannot add the X-Robots-Tag header to images, PDFs and other files that the web server serves directly, so those files are not covered by the checkbox. And on a multisite network the setting is per site.

Google's documentation explains the key point about noindex: a robots meta tag tells Google not to show the page in results, but it can be read only if crawlers are allowed to fetch the page. That is why the sequence matters when you fix the problem.

Step-by-step instructions

Step 1: Confirm whether the setting is on

In the WordPress admin, go to Settings > Reading. Look for Search engine visibility near the bottom. If the box is ticked, the setting is on.

Do not rely on that screen alone. Confirm from the outside, because what search engines receive is what matters. Open your home page, view the source, and search for robots. Or use:

curl -s https://example.com/ | grep -i "<meta name=.robots"
curl -sI https://example.com/ | grep -i "x-robots-tag"

A line such as <meta name='robots' content='noindex,nofollow' /> is the sign of this setting. The second command shows whether the server or a plugin also sends an X-Robots-Tag header.

What success looks like: You know whether the setting is on and what the page really sends, and you have a copy of the output as "before" evidence.

Step 2: Turn the setting off

Untick Discourage search engines from indexing this site and click Save Changes. If a caching plugin or CDN serves saved copies of your pages, clear its cache so that the old noindex tag is not served from storage. Reload the page source and run the curl command again.

If the box re-ticks itself after saving, something else controls it. Look for a hosting dashboard toggle, a staging tool or a plugin that manages visibility, and change it there.

What success looks like: The robots meta line with noindex is gone from the home page source.

Step 3: Look for other sources of noindex

The checkbox is only one way to say noindex. Check the others:

SEO plugin settings: a plugin can set noindex for the whole site, for specific post types, taxonomies, archives or single pages. Open the plugin's search appearance or indexing settings and review them.
Page-level settings: a single page or post may have noindex set in the editor.
HTTP headers: a web server, host or CDN rule can send X-Robots-Tag: noindex for the entire site, often left over from staging. This is invisible in the page source, which is why the curl -I check matters.
robots.txt: a Disallow: / rule in a physical robots.txt file or from a plugin stops crawlers fetching your pages at all, so they never see that you removed noindex. Fix this first. Robots.txt Blocking Google or Bing? Diagnose It Step by Step explains how.
Password protection: an HTTP login or maintenance plugin may return 401 or 403 to crawlers.

Check a range of pages, not just the home page: a post, a page, a category and a product if you have a shop.

What success looks like: No page that should be indexed carries noindex in either the HTML or the headers, and robots.txt does not block them.

Step 4: Confirm the sitemap and robots.txt

Because core sitemaps are disabled while the setting is on, they should come back once you turn it off. Open https://example.com/wp-sitemap.xml. If you use an SEO plugin, check its sitemap address instead. Then open https://example.com/robots.txt and confirm that it does not block your pages and that it references the sitemap. WordPress adds a reference to the core sitemap index in the robots.txt it generates.

If the sitemap still does not load, a plugin, filter or server rule may be disabling it. See How to Create a Valid XML Sitemap for Your Website.

What success looks like: The sitemap opens and lists your public URLs, and robots.txt allows the pages you want indexed.

Step 5: Ask search engines to look again

Search engines will find the change on their own, but you can help. In Google Search Console, use URL Inspection on the home page and a few key pages. Choose the live test to confirm that the page is indexable now, and request indexing if it is available for the URL. The guide Request Google Indexing the Right Way explains what that button does and does not do. Submit the sitemap in the Sitemaps report so Google sees the full list of URLs. For Bing, add the site in Set Up Bing Webmaster Tools and Verify Your Website.

Google's documentation gives no timetable for how long a page takes to reappear after noindex is removed, and nobody can promise one. Check back over the following days and weeks. If pages are slow to appear, Crawled — Currently Not Indexed: What to Check First and Discovered — Currently Not Indexed: A Practical Troubleshooting Guide cover what the statuses mean.

What success looks like: URL Inspection reports that key pages can be indexed and that no noindex is detected, and the sitemap shows as successfully read.

Step 6: Stop it coming back

Write down a launch procedure: "Before go-live, untick Discourage search engines, clear caches, check the source for noindex." Include it in your deployment checklist. If you copy a production database back to staging, the reverse risk applies: staging may end up indexable, so keep it behind a password. Crawvix's scan checks sampled pages for an accidental noindex, as a meta tag or an X-Robots-Tag header, and its advice for a failure mentions this very setting. A scan right after each launch is a cheap safeguard.

What success looks like: The check is on your launch list and someone owns it.

Final verification checklist

Settings > Reading: "Discourage search engines from indexing this site" is unticked.
The home page source has no robots meta tag containing noindex.
curl -I shows no X-Robots-Tag: noindex on the home page or key pages.
SEO plugin settings do not set noindex for post types, archives or pages you want indexed.
robots.txt does not block your public pages.
The sitemap loads and is referenced in robots.txt.
Search Console URL Inspection shows key pages as indexable.
Caches were cleared after the change.
A launch checklist item exists for this setting.

Common mistakes to avoid

Unticking the box but leaving a plugin noindex. Check SEO plugin settings by post type and archive, not just the global option.
Leaving Disallow: / in robots.txt. Crawlers will not fetch the pages and so will not see that noindex is gone.
Testing only the admin screen. Check the page source and headers, because caches and servers can still serve the old tag.
Forgetting the cache. Page caches and CDNs can keep serving the noindex copy for a while after you save.
Expecting instant results. The change is noticed on the next crawl; there is no published timetable.
Using the checkbox as a privacy tool. It does not block access. Use a password or server access control for private content.
Copying staging to production without a checklist. The setting travels with the database.
Using the setting for files. WordPress cannot apply it to PDFs and images served directly by the web server.

Frequently asked questions

What does "Discourage search engines from indexing this site" actually do?

It adds a robots meta tag with noindex to your pages, stops WordPress pinging update services and disables the core sitemaps. It does not block visitors, and search engines are expected to honor the request.

Does the setting stop Googlebot from crawling my site?

No. Since WordPress 5.3 it no longer adds Disallow: / to robots.txt. Crawlers can still fetch the pages, which is how they find and read the noindex tag.

How long until my site appears in Google after I untick it?

Nobody can promise a time. After the change, request indexing for key pages in Search Console, submit the sitemap and check back. Indexing depends on the search engine's own decisions about each page.

Why does my site still say noindex after I unticked the box?

Another source is probably responsible: an SEO plugin, a page-level setting, an X-Robots-Tag header from the server or CDN, or a cached copy of the page. Check the page source and response headers to find which.

Is the setting reliable for hiding a private site?

No. WordPress's own documentation says it does not block access and that honoring the request is up to search engines. For private content use a login or server-level access control.

Official references

Next step: See how Crawvix checks a website and capture a baseline once scanning opens. This guide provides technical information, not a guarantee of rankings, traffic or index inclusion.