Skip to content

What "Discourage search engines from indexing this site" does in WordPress, and how to turn it off

The box under Settings, then Reading, puts a noindex, nofollow robots tag on every page and switches off the sitemap at /wp-sitemap.xml. It writes nothing into robots.txt. Untick it and save, or run wp option update blog_public 1, then read a page's source to confirm the tag is gone.

By
WP Ministry
Published
Tested on
WordPress 7.1.3, PHP 8.3.35

In short

  • The box is under Settings, then Reading, in the row "Search engine visibility". WordPress stores it as the option blog_public, where 0 is ticked and 1 is not.
  • While it is ticked, every page carries a robots tag of noindex, nofollow, and /wp-sitemap.xml answers 404.
  • It adds no Disallow line to robots.txt, and has not since WordPress 5.3. Looking there for it finds nothing.
  • It is a request, not a lock. Anyone can still open the site, so a staging copy needs a password too.
  • To turn it off, untick it and save, or run wp option update blog_public 1. Then check what a page serves, not only the setting.
  • Google learns of the change when it next crawls the page. Nobody outside Google sets that date.

The box labeled "Discourage search engines from indexing this site" does two things while it is ticked. WordPress adds a robots tag that says noindex, nofollow to every page it serves, and it switches off the sitemap at /wp-sitemap.xml. It does not write a rule into robots.txt, and it does not stop anyone from opening the site.

To turn it off, untick the box under Settings, then Reading, and save. With WP-CLI, run wp option update blog_public 1. Then read a page's source to confirm the tag has gone, because a cache, a plugin or the server can leave the same instruction in place.

Where the setting is

In the dashboard, go to Settings, then Reading. Find the row headed "Search engine visibility". It holds one checkbox, "Discourage search engines from indexing this site", and one line beneath it: "It is up to search engines to honor this request."

The same checkbox is on the screen that installs WordPress, where the site's title and first user are entered. A site that was installed with it ticked keeps it ticked until someone unticks it, and going live does not change it.

WordPress stores the setting as an option named blog_public. The value is 0 while the box is ticked and 1 while it is not. A fresh install has 1 unless the box was ticked on the install screen.

What it does while it is ticked

Box untickedBox ticked
Robots tag on every pagemax-image-preview:largenoindex, nofollow
/wp-sitemap.xmlThe sitemap index, status 200"Page not found", status 404
/robots.txtWordPress's usual rules and a Sitemap: lineThe same rules without the Sitemap: line
A visitor or a crawler asking for a pageGets the pageGets the page

The tag

This is the line in the <head> of every page while the box is ticked:

text
<meta name='robots' content='noindex, nofollow' />

Google's documentation gives the meaning of the two words. noindex asks that the page not be shown in search results. nofollow asks that the links on the page not be followed.

With the box unticked, WordPress still prints a robots tag. It reads max-image-preview:large, which allows large image previews and blocks nothing. So the presence of a robots tag tells you nothing. The word to look for is noindex.

The sitemap

WordPress makes its own sitemap at /wp-sitemap.xml. While the box is ticked, that address and every sitemap file under it answer 404 with the site's own "Page not found" page. The sitemap is not deleted, because it was never a file: WordPress builds it on request and stops doing so while the option is 0. The WordPress sitemap at wp-sitemap.xml covers the sitemap itself.

robots.txt

With no robots.txt file in the site's folder, WordPress answers for that address itself. This is the answer while the box is ticked:

text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

And this is the answer once it is unticked, on a site whose address is https://example.com:

text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The only difference is the Sitemap: line. There is no Disallow: / in either.

What it does not do

  • It writes no rule into robots.txt. The answer above keeps crawlers out of the dashboard and nothing else, ticked or not.
  • It does not block access. The site answers every request as before. The line under the checkbox says as much: the tag is a request, and each search engine decides whether to honor it.
  • It does not cover files the web server sends by itself. The tag is part of a page WordPress builds. An image or a PDF in the uploads folder is sent by the web server without WordPress, so it carries no such instruction. WordPress's developer note on this setting says a header set on the server is the way to cover those.
  • It does not remove pages from Google at once. A page that is already listed stays listed until Google crawls it again and reads the tag.

How this changed, and why older advice finds nothing

WordPress versionWhat changed
Up to 5.2Ticking the box made WordPress's robots.txt answer with Disallow: /.
5.3The Disallow: / was removed. The box now produces a robots tag of noindex,nofollow.
5.5WordPress gained its own sitemap, which is switched off while the box is ticked.
5.7The tag is built through one filter, wp_robots. Sites with the box unticked get max-image-preview:large.

The developer note for 5.3 gives the reason. A rule in robots.txt only stops crawling, and a page that is not crawled can still be listed when other sites link to it. The tag asks for the page to be left out of the results, which is what the box is for.

Much of what is written about this setting still describes the first row of that table. If you were told to open /robots.txt and look for Disallow: /, you will not find it on a current WordPress, whether the box is ticked or not. If you do find Disallow: / there, it did not come from this box. It comes from a robots.txt file in the site's folder or from a plugin, and where robots.txt is in WordPress and how to change it shows how to tell which.

Check whether it is on, three ways

In the dashboard

Open Settings, then Reading, and look at the checkbox. Two other screens report it without your asking:

  • The "At a Glance" box on the Dashboard shows a link that reads "Search engines discouraged" while the box is ticked.
  • Under Tools, then Site Health, the Status tab lists "Search engines are discouraged from indexing this site." among its recommended improvements. WordPress has had this test since version 6.9. On the Info tab, the "WordPress" section has a line "Is this site discouraging search engines?", answered Yes or No.

With WP-CLI

Run this from the site's folder. How to use WP-CLI covers connecting and where to run commands.

bash
wp option get blog_public

It prints 0 if the box is ticked and 1 if it is not.

From outside

This asks the site for its home page and prints its robots tag. Put your own address in place of https://example.com/.

bash
curl -sL https://example.com/ | grep -i -o "<meta[^>]*robots[^>]*>"
text
<meta name='robots' content='noindex, nofollow' />

That output means the instruction is on the page. If the line reads max-image-preview:large and nothing else, it is not.

Without a terminal, open the page in a browser, view its source, and search for noindex. The site health check reads the same tag from an address you give it.

Check a post or the home page, not a search results page. WordPress puts noindex on its own search results whatever this box says.

Turn it off

  1. Step 1: Untick the box and save

    Go to Settings, then Reading. Untick "Discourage search engines from indexing this site" and select "Save Changes".

  2. Step 2: Or set the option with WP-CLI

    This does the same thing from the command line.

    It answers Success: Updated 'blog_public' option. If the option was already 1, it says the value is unchanged, and the box was not your problem.

    bash
    wp option update blog_public 1

  3. Step 3: Clear any page cache

    A caching plugin stores finished pages and serves those copies. A copy stored while the box was ticked still carries the tag. Clear the caching plugin, the host's cache and any CDN that stores whole pages.

  4. Step 4: Read the tag again

    Run the check under "From outside" again, on the home page and on one post. Neither line should contain noindex.

  5. Step 5: Check that the sitemap answers

    This prints the status code of the sitemap's address.

    It prints 200 once the box is unticked and 404 while it is ticked. A 404 with the box unticked has another cause, since a plugin can switch WordPress's sitemap off as well: see the WordPress sitemap at wp-sitemap.xml.

    bash
    curl -sL -o /dev/null -w "%{http_code}\n" https://example.com/wp-sitemap.xml

  6. Step 6: Read robots.txt

    The Sitemap: line should be back. Nothing in the file should say Disallow: / by itself on a line.

    bash
    curl -sL https://example.com/robots.txt

If the box is unticked and the tag is still there

wp option get blog_public prints 1, the cache is clear, and a page still says noindex. Something else is adding it. These are the places to look.

  • An SEO plugin's own setting. A plugin can add robots tags of its own. Look for a setting that says noindex in the plugin's settings, and in its panel on the edit screen of the page that carries the tag.
  • A "coming soon" or maintenance plugin left on from before launch.
  • Code in the theme or a snippet plugin. Any code can add to the tag through the wp_robots filter.
  • A header from the server. Google treats an X-Robots-Tag header the same as the tag in the page. It is set in the server's configuration, not in WordPress. WordPress's developer note recommends this header for development sites, so a server that began as one may still send it. This command prints the header if the page is sent with one:
bash
curl -sIL https://example.com/ | grep -i "x-robots-tag" || echo "no X-Robots-Tag header"

If the site is missing from Google and none of these is the cause, why a WordPress site is not showing up on Google goes through the other causes in order.

What happens at Google's end

Unticking the box changes what your site serves from the next request. It changes nothing at Google until Google asks for the page again. Its documentation says it has to crawl a page to see the tags and headers on it.

  • In Search Console, the Page indexing report lists affected pages under "URL marked ‘noindex’". Inspect one, test the live URL, and, when the test no longer finds a noindex, select "Request Indexing". Google's help page for the report describes these steps.
  • On timing, Google's page on recrawling says crawling can take "anywhere from a few days to a few weeks". It adds that asking for a crawl does not mean a page is included at once, or at all, and that asking again for the same address does not speed it up.
  • For many pages, the same page says to submit a sitemap. Yours answers again now that the box is unticked.

Nobody outside Google can give you a date. What you can do is make sure the instruction is gone everywhere it could come from, and then ask once.

Why a noindex behind a robots.txt block is never read

Google's crawler reads robots.txt before it asks for a page. If robots.txt tells it to stay out, it does not fetch the page, so it never reads the tag on it. Google's documentation says this directly: for noindex to work, the page must not be blocked by robots.txt, and a blocked page can still appear in results if other pages link to it.

This matters in both directions.

  • Hiding a site. Do not add Disallow: / to robots.txt on top of ticking the box. The block stops Google from ever reading the noindex. This is the reason WordPress stopped writing that rule in 5.3.
  • Bringing a site back. If robots.txt still blocks the site, Google cannot see that the noindex has gone. Check robots.txt as well as the tag.

On a staging site

Ticking the box on a staging copy is right, and it is not enough. The tag is a request, and the copy is still open to anyone who has its address. How to set up a WordPress staging site covers the password that goes with it and the other things a copy needs.

The risk is the day the copy becomes the live site and brings the ticked box with it. The WordPress launch checklist has unticking it as a step, with the rest of what to check before a site is announced.

When to get help

  • Ask the host if the X-Robots-Tag header is there and you cannot find where it is set.
  • Hand it over if the option reads 1, the tag or the header is still served, and you cannot find what adds it. A one-time fix from WP Ministry covers one issue on one site and starts with a free diagnosis, which gives you a written cause and a fixed quote.

Common questions

Is the box ticked by default?

No. A new WordPress site has it unticked, with blog_public set to 1. It is ticked only if someone ticked it, on the install screen or under Settings, then Reading, or if a command or a tool set the option. A site that began as a staging copy may have had it ticked on purpose, so check there first.

I unticked it. Why is my site still not on Google?

First confirm that the tag has gone from the page's source, since a cache or a plugin can keep it there. Then it is a matter of Google crawling the page again, which its documentation says can take from days to weeks. If the tag is gone and robots.txt blocks nothing, work through why a WordPress site is not showing up on Google.

Should I look for "Disallow: /" in robots.txt?

Not for this setting. WordPress has not written that rule for this box since version 5.3. Read the page's source for noindex instead. A Disallow: / in robots.txt is a separate problem with a separate cause.

Does ticking the box take my site out of Google?

It asks for that. Google's documentation says that when it crawls a page and reads noindex, it drops the page from its results. That happens page by page as each is crawled again, not on the day you tick the box. Other search engines decide for themselves.

Does it keep people out of a staging site?

No. It changes a tag and switches off the sitemap. Every page still opens for anyone who has the address. Put a password on a site that is not meant to be seen.

More on this subject

Quick Fix, done for you

Quick Fix is $49. One issue, one site, up to about an hour. No fix, no fee. 30-day warranty. It starts with a free diagnosis.

Get a free diagnosis