WordPress robots.txt: where it is, what it says by default and how to edit it

A fresh WordPress has no robots.txt file. When a crawler asks for /robots.txt and no file is there, WordPress writes the answer itself: three lines that keep crawlers out of the dashboard, and a line that says where the sitemap is. That is why the address answers and the folder holds nothing by that name.

There are two real ways to change it. A file named robots.txt in the site's root replaces WordPress's answer entirely. A few lines of PHP on the robots_txt filter add to WordPress's answer and keep the rest. An SEO plugin's editor is one of those two underneath.

Why there is no robots.txt file

WordPress treats /robots.txt as an address it serves, like a post or a feed. A rewrite rule sends the request to WordPress, and a function called do_robots() prints the lines. Nothing is saved to disk.

Three things have to be true for WordPress to answer:

• No real file is in the way. The web server looks for a file first. If robots.txt exists in the site's root, the server sends it and WordPress is never asked.
• Permalinks are not set to Plain. The rule that catches /robots.txt is one of WordPress's rewrite rules, and with Plain permalinks there are none. The section after the next one covers that case.
• WordPress is at the root of the address. WordPress's source adds the rule only when the site's home has no folder in it, so a WordPress at example.com/blog/ writes no robots.txt answer. Crawlers would not look in that folder anyway: Google's documentation says they do not check for robots.txt files in subdirectories.

What a fresh WordPress answers with

This is the whole answer on a new site with nothing added, with example.com standing in for the site's address:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The exception is there because admin-ajax.php is not only for the dashboard. WordPress's plugin handbook sends every Ajax request to that file, from public pages too, and the change that added the line to WordPress gives that as its reason.

The Sitemap line is not part of do_robots(). It is added by WordPress's own sitemap feature, which arrived in WordPress 5.5, and only in two cases together: the sitemap is switched on, and the site is open to search engines. Anything that switches WordPress's sitemap off takes the line with it. The sitemap at wp-sitemap.xml covers the sitemap itself.

Ticking "Discourage search engines from indexing this site" under Settings, then Reading, removes the Sitemap line and changes nothing else here: since WordPress 5.3 that box no longer writes Disallow: / into this answer, and what the box does instead has its own page.

If the address shows your home page or a 404

On a site with Plain permalinks, WordPress does not answer /robots.txt. A new WordPress can start that way, so check before you look for anything else. In the dashboard it is under Settings, then Permalinks: "Plain" is the first choice in the list. Over SSH, WP-CLI prints the structure in use, and prints an empty line when it is Plain.

wp option get permalink_structure

What a crawler receives then depends on the server. On an Apache server whose .htaccess still holds WordPress's rewrite rules, the request reaches WordPress, which redirects /robots.txt to /robots.txt/ and answers with the home page. With no such rules, the server answers with its own 404. In both cases WordPress's rules are still there at /?robots=1, an address no crawler asks for.

Neither outcome blocks anything. Google's documentation says a 404 for robots.txt is treated as no restrictions at all, and that when HTML comes back in place of rules, the lines that are not rules are ignored. What is lost is the Sitemap line and any rule you meant to add with the filter.

To get WordPress's answer back, choose any structure other than Plain. That changes the address of every post on a site that already has visitors, so read how to change permalinks safely first. A real robots.txt file is served whatever the permalinks are.

Three ways to change robots.txt

A real file in the site's root

Create a plain text file named robots.txt, in lowercase, and upload it to the folder that holds wp-admin, wp-content and wp-includes. Google's documentation asks for UTF-8 text made in a text editor, not a word processor, and for the file to sit at the root of the host it applies to.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /example-private/

Sitemap: https://example.com/wp-sitemap.xml

From the moment the file is there, it is the entire answer. WordPress's three lines and its Sitemap line are not merged in, which is why the sample repeats them. Leave them out and they are gone. The Sitemap line has to be the full address of the sitemap your site really serves, and from now on it is yours to keep correct.

Delete the file and WordPress answers again, as before.

The robots_txt filter

A filter lets you change the answer WordPress writes, so its own lines stay and the Sitemap line keeps following the site's settings. The smallest place to put one is a must-use plugin: a single PHP file in wp-content/mu-plugins/ that WordPress loads on every request, with nothing to activate.

The function receives two things: the text WordPress has written so far, and whether the site is open to search engines. Whatever it returns is what a crawler gets, so it must always return the text. Two groups for User-agent: * are not a fault. Google's documentation and the robots.txt standard both say that groups for the same crawler are combined into one.

The file shows on the Plugins screen under "Must-Use" and cannot be switched off there. To undo the change, delete the file. WordPress's documentation notes that must-use plugins get no update notices, so keep a note of why the file exists.

The same add_filter call also works in an ordinary plugin, or in a child theme's functions.php without the opening <?php line and the comment. WordPress loads only the active theme's functions.php, so a rule kept there leaves when the theme is changed.

An SEO plugin's editor

An SEO plugin that offers to edit robots.txt does one of the two things above, and its own documentation says which. Yoast SEO's help page says its file editor creates a robots.txt file and that the file replaces WordPress's default. Rank Math's says its editor changes the answer WordPress writes, and that a real robots.txt file has to be deleted from the root before the editor can be used.

So with any such plugin, look in the site's root first. If a robots.txt file is there, that file is what crawlers get, whatever a settings screen shows.

Which one wins

The file. With a file and a filter both in place, the answer is the file's text alone and the filter's line is absent. The filter only ever changes what WordPress writes, and WordPress is not asked while a file exists.

What robots.txt is for, and what it is not for

robots.txt tells crawlers where they may go. It does not decide what a search engine lists. Google's introduction to robots.txt says the file is mainly for avoiding too many requests to a site, and that "it is not a mechanism for keeping a web page out of Google."

Three consequences, each from Google's documentation:

• A blocked page can still be listed. If other pages link to an address that robots.txt blocks, Google can list the address without visiting it. The result has no description.
• A noindex behind a block is never read. For a noindex to work, the page must not be blocked by robots.txt. If it is, the crawler never sees the tag, and the page can still appear in results.
• It is a request, not a lock. Whether a crawler obeys the file is up to the crawler. Google's advice for anything private is a password.

Two rules that turn up in older advice do nothing for Google: its documentation says a noindex line inside robots.txt is not supported, and neither is crawl-delay.

What not to block on WordPress

• /wp-admin/admin-ajax.php. WordPress allows it on purpose. If you write your own file and keep Disallow: /wp-admin/, keep the Allow line with it.
• /wp-includes/ and /wp-content/. On a fresh WordPress the home page loads its stylesheets and scripts from both folders. Google's documentation says not to block resource files a page depends on, because Google does a poor job of analyzing pages it cannot load them for, and that it does not render JavaScript from blocked files.
• /wp-content/uploads/. That is where your images are. Google's documentation says a robots.txt block on image files keeps them out of Google's search results.
• A page you have marked noindex. The tag has to be read to work.
• Everything. Disallow: / under User-agent: * tells every crawler that follows that group to stay out of the whole site. If a site that should be found has that line, see why a WordPress site is not on Google.

How to test a rule

Read the file as it is served. What counts is what a crawler receives, not what is in an editor. Google's documentation suggests opening the address in a private browsing window. From a command line, this prints the status and headers as well as the text:

curl -s -i https://example.com/robots.txt

A working answer starts with a 200 status and a Content-Type of text/plain, followed by your lines. A 301 with a Location header, or a page of HTML, means something other than a robots.txt answered.

Check what Google fetched. Search Console has a robots.txt report among a property's settings. Google's help page says it shows the robots.txt files Google found for the property's hosts, when each was last crawled, whether the fetch succeeded, the file's size and any lines Google could not parse. You can open the last version Google fetched and the versions from the past 30 days, and ask for a recrawl after an important change. The report is offered for Domain properties and for URL-prefix properties with no path. To find out whether one address is blocked, the same help page points to the URL Inspection tool.

Allow for the delay. Google's documentation says it generally caches a robots.txt file for up to 24 hours, so a change is not acted on at once.

AI crawlers and robots.txt

The companies behind AI products publish the names their crawlers answer to in robots.txt. Each line below is what that company's own page says the name is for.

One rule of the standard matters here. A crawler follows the group that names it, and follows the * group only when no group does. Google's documentation adds that a named group and the * group are not combined. WordPress's default has only a * group, so every crawler that honors robots.txt follows the same three lines. The moment you add a group for one crawler, that crawler stops reading the * group, and the /wp-admin/ lines have to be repeated in its group if you still want them to apply.

A page fetched because a person asked for it is a different case. OpenAI's page says robots.txt rules may not apply to ChatGPT-User, and Perplexity's says Perplexity-User generally ignores them.

Whether to let these crawlers in, and what each choice costs, is a subject of its own. This page covers only where the lines go.

When to get help

• Ask the host if you cannot find the site's root or cannot upload to it.
• Hand it over if the address answers with something you cannot trace to a file, a plugin or the filter. A one-time fix from WP Ministry covers one issue on one site and starts with a free diagnosis, which gives you a written cause and a fixed quote.

robots.txt is one of the things WordPress gives a search engine with no plugin installed. WordPress SEO without a plugin lists the rest.