"Indexed, though blocked by robots.txt" on WordPress
Google indexed the address without fetching it, because robots.txt blocks the crawl and other pages link to it. The help page says robots.txt is the wrong way to keep a page out of Search. Remove the block and use noindex, or remove the block if you want the page fetched.
- By
- WP Ministry
- Published
In short
- The page is in Google's index. robots.txt stopped the crawl, and a link from somewhere else was enough for Google to list it.
- The help page says any snippet will probably be very limited, because Google did not read the page.
- To keep it out of Search, remove the robots.txt block and add noindex. A block prevents Google from seeing a noindex.
- To have the page fetched, delete the rule that matches it. WordPress's own rule for /
wp-admin/ is not this problem. - "Blocked by robots.txt" is the other status. There the address was not indexed.
- The warning clears when Google recrawls. A sandbox cannot file the report.
"Indexed, though blocked by robots.txt" means the page is in Google's index even though your robots.txt told Google not to fetch it. The Page indexing report lists it as a warning, not as a reason for leaving a page out. The help page's account: Google always respects robots.txt, and that does not necessarily prevent indexing if someone else links to the page. Google will not request and crawl it. It can still index it from the linking page. Because of the block, any snippet in results will probably be very limited.
"Blocked by robots.txt" is the other status. There Google obeyed the file and did not index the address. The rule on your site can be the same rule. The difference is that this warning means a link was enough for Google to list the page anyway. That page's section on this warning points here for the steps.
Decide what you want
The help page gives two next steps, and they are opposites.
- You want the page out of Google. robots.txt is the wrong tool. Remove the block and use a
noindexdirective. The help says this in those words. A block stops Google from reading a noindex, so the tag does nothing until the block is gone. - You want the page fetched. Update robots.txt so the address is allowed. The robots.txt tester in Search Console, which the help page names, shows which rule matches.
Google's canonical documentation says the same thing about robots.txt from the other side: do not use it to choose a canonical, because Google may index a disallowed URL without its content.
WordPress's own robots.txt disallows / and allows admin-ajax.php. An address under / in the other status is expected. This warning on a public page or a post is a rule that reaches further than the dashboard.
What causes it
The page was blocked to keep it out of Google, and other sites link to it
CommonGoogle's help says it respects robots.txt and still may index a blocked page from links elsewhere, without requesting it. The snippet is then probably very limited. The help says robots.txt is not how you keep a page out of Search.
A rule blocks a page you want Google to read
SometimesThe help page's other path is that you do not want the page blocked. Update robots.txt to allow it. A rule without a trailing slash matches every address that starts with those characters, which is how a page you want found ends up in this warning.
How to fix it
Remove the block, then noindex the page you want out
- Takes care
- Low risk
- About 20 minutes
Step 1: Find the rule
Open
/. The robots.txt tester tells you whether a line blocks the address in the report. Where robots.txt comes from on WordPress covers a file on disk, which replaces WordPress's answer, and a plugin that adds lines.robots.txt Step 2: Allow the address
Delete or narrow the rule that matches the page. Keep the
/rule. A trailing slash keeps a folder rule to that folder, which the blocked-by-robots.txt page explains.wp-admin/ Step 3: Add noindex on the page
WordPress has no per-page setting for this. The Reading screen's "Discourage search engines" box marks the whole site, and it does not do it with robots.txt. For one page, the tag has to come from a plugin or from code on that page. Confirm it in the page source, or with the page indexing check, after the robots.txt change. "Crawl allowed?
" has to be yes before Google can read the tag. Step 4: Wait for a crawl
The warning is about the last crawl. It changes when Google fetches the page, sees the noindex, and drops the URL. The report then uses the noindex reason instead, which is what you asked for.
Remove the rule if you want the page fetched
- Easy
- Low risk
- About 15 minutes
Step 1: Match the rule to the address
Use the tester on the exact path from the report. If the rule was meant for a different folder, narrow it. If it was a
Disallow: /left from a staging copy, delete it. That leftover is one of the causes on Blocked by robots.txt.Step 2: Fetch robots.txt
The file Google will read should no longer disallow the page. WordPress answers
/itself when no file sits in the root. A file you uploaded is the whole answer.robots.txt Step 3: Inspect the URL
In the URL Inspection tool, the live test should show that crawling is allowed. Indexing after that is a separate decision. Unblocking a page does not place it in results.
A WordPress SEO audit reads robots.txt against the pages you want found, which is this warning and the blocked status together.
Common questions
Can I leave the block and add noindex?
No. Google's help says to remove the robots.txt block and then use noindex. While the block is in place, Google does not request the page, so it does not see the tag. The page can stay indexed from other people's links.
Why is the snippet so short?
The help page says that because of the robots.txt rule, any snippet will probably be very limited. Google indexed the page from the link that pointed at it, not from the page itself.
Is /wp-admin/ a problem if it appears here?
The dashboard is blocked on purpose. Prefer it under "Blocked by robots.txt", and leave WordPress's rule. If the warning is a public URL, the rule is matching more than the dashboard.
I deleted the rule. Why is the warning still there?
Google works from the copy it fetched. The report updates when it crawls the page again. The live test shows what the site answers now, which can be ahead of the report.
To undo it: Put the line back.
When to get help
When the warning lists many pages you want found and the matching rule is written by a plugin or a host, not by a file you can see. The robots.txt tester shows the line. Finding what writes it is the rest.
- GuideBing Webmaster Tools for WordPress: setup, IndexNow, and what it means for ChatGPT and Copilot
- GuideDid AI Overviews cause your traffic drop? What Search Console can show, and what it cannot
- GuideFavicon not showing in Google: the Site Icon WordPress writes
- GuideGoogle shows the wrong site name: what you can set on WordPress
- GuideGenerative engine optimization (GEO): what is documented, what was studied and what is guesswork
- GuideHow to track ChatGPT traffic in Google Analytics 4, and what the count leaves out

