Skip to content

robots.txt tester: does this rule block this address?

Paste a robots.txt, give an address and choose a crawler, and see whether it is allowed and which line decides. Worked out in your browser; nothing is sent.

Paste the file as your site serves it. Open yoursite.com/robots.txt in a browser to see it.

A path that starts with a slash, or a full address.

Allowed

Googlebot may ask for /wp-admin/admin-ajax.php

Line 3 decides: "Allow: /wp-admin/admin-ajax.php". It is in the group for every crawler, and it is the longest rule there that matches the address.

How a crawler reads the file

The tester follows the rules as the standard for robots.txt and Google's own documentation give them. They are short, and two of them surprise people.

  • A crawler follows one group. If the file has a group that names the crawler, it follows that group and ignores the one for every crawler. So a file that blocks /private/ for everyone and then has a short group for Googlebot has, without meaning to, let Googlebot into /private/.
  • The longest rule that matches wins, wherever it is in the group, and Allow wins a tie. Order does not matter.
  • A star stands for any characters, and a dollar sign at the end of a rule means the address has to end there.
  • An empty Disallow: blocks nothing.

What it cannot tell you

It tells you what the file says. It does not fetch your site, so paste the file as your site serves it, not as you believe it to be: on WordPress the two can differ, because WordPress answers for robots.txt itself when there is no file, and a plugin can change that answer. It also cannot tell you whether a crawler obeys the file, or whether a page it blocks is listed anyway: a blocked address can still appear in search results when other pages link to it.

Where robots.txt is in WordPress and how to change it explains the file itself. To check a page on a live site, including its robots.txt, use the page indexing check.

What happens to what you paste

Nothing leaves your browser. The answer is worked out on this page, and nothing you type is sent or kept.

Common questions

Why does my rule for everyone not apply to Googlebot?
Because the file has a group that names Googlebot. A crawler that is named follows only its own group. Put the rule in that group as well, or remove the group if it was not needed.
Does blocking a page in robots.txt take it out of Google?
No. It stops the page being crawled. Google's documentation says a blocked address can still be listed, without a description, if other pages link to it. To take a page out of search, let it be crawled and give it a noindex.
Which crawler name should I choose for an AI assistant?
The one the company gives for the job you care about. Search crawlers such as OAI-SearchBot build what an assistant searches; training crawlers such as GPTBot collect pages for models. The AI crawler access check reads a live site's file for all of them at once.

Rules not doing what you expected?

Tell us the address and what you are trying to keep in or out. We look at the site itself and reply with the cause and a fixed quote. The diagnosis is free.

Get a free diagnosis