Skip to content

robots.txt generator for WordPress

Build a robots.txt that starts from WordPress's own rules, names your sitemap, and keeps out the kinds of AI crawler you choose. Made in your browser; nothing is sent.

The sitemap

In full, such as https://example.com/wp-sitemap.xml. WordPress adds this line to its own answer, and a file of your own has to carry it.

The site

WooCommerce adds rules of its own to WordPress's answer: its folders under uploads, and the addresses that put a product in a cart. A file of yours replaces that answer, so it has to carry them.

AI crawlers to keep out

Left unticked, a crawler follows the same rules as every other. Search engines' own crawlers are not in these lists.

GPTBot, ClaudeBot, Google-Extended. Keeping these out does not affect search.

OAI-SearchBot, Claude-SearchBot, PerplexityBot. By each company's own account, keeping its crawler out can keep a site out of that assistant's search answers.

ChatGPT-User, Claude-User, Perplexity-User. One page, fetched because a person asked the assistant about it or gave it the address. OpenAI says robots.txt may not apply to its one, because a person started the request.

robots.txt
# For every crawler that is not named below: the rules WordPress itself answers with.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Built in your browser. It begins with the rules WordPress itself answers with.

Do you need a file at all?

Often not. A WordPress site with no robots.txt in its folder still answers at that address: WordPress writes the answer itself, keeping crawlers out of the admin and naming its own sitemap. A file is worth making when you want something that answer does not say, such as keeping a kind of AI crawler out.

What the file does

  • It begins with WordPress's own rules. A real file replaces WordPress's answer entirely, so the file carries those two lines itself.
  • On a store, it carries WooCommerce's rules too. WooCommerce adds lines to WordPress's answer that keep crawlers out of its own folders and away from the addresses that put a product in a cart. A real file drops them unless it repeats them. It does not block the cart, the checkout or the account pages: those carry a noindex, which a crawler can only read if it is allowed to ask for them.
  • It names your sitemap, if you give its address. WordPress adds that line to its own answer. A file of yours has to carry it.
  • It keeps out the kinds of AI crawler you tick, each by the name its own company gives it. A crawler that a file names follows only the group written for it, so a group that keeps one out says only that.

How to use it

Save the file as robots.txt and upload it to the folder WordPress is installed in, beside wp-config.php. Then open yoursite.com/robots.txt and check that it is what you saved. If an SEO plugin manages robots.txt on your site, change it there instead, so the two do not disagree. Where robots.txt is in WordPress and how to change it covers each way.

What it does not do

robots.txt is a request to crawlers, not a lock. It keeps out the ones that obey it. It also does not take a page out of search: a page that is blocked can still be listed if other pages link to it. And do not use it to hide a site that should stay out of search. For that, WordPress has a setting, and blocking the site here as well stops that setting being read.

Nothing leaves your browser. The file is built on this page, and nothing you type is sent or kept.

Common questions

Should I keep AI crawlers out?
That is your call, and the kinds are different calls. Keeping training crawlers out does not affect search. Keeping an assistant's search crawler out can, by that company's own account, keep your site out of its search answers.
Will this stop my content being used by AI?
It tells the crawlers named in it to stay out, and each company documents robots.txt as the way to manage its crawler. It does nothing about pages already collected, about crawlers that ignore robots.txt, or about copies of your pages on other sites.
I saved the file but my site still shows the old rules. Why?
Either the file is not in the folder WordPress is installed in, or a cache is serving the old answer, or a plugin is answering for robots.txt before the file is reached. Open the address in a private window, and check the plugin's settings.
Why are there no rules for /wp-content/ or /wp-includes/?
Because blocking them stops a search engine loading the styles and scripts it needs to see your pages as visitors do. WordPress's own answer blocks only the admin, and this file does the same.

Rather have it set up for you?

Tell us the address and what you want kept in or out. We set it up on the site, check what each crawler is given, and reply with a fixed quote first. The diagnosis is free.

Get a free diagnosis