Skip to content

"Crawled - currently not indexed" on a WordPress site: what to leave and what to check

"Crawled - currently not indexed" means Google fetched the address and chose not to index it. If the addresses listed are feeds and archives WordPress makes by itself, that is the right outcome and nothing needs doing. For a page you wrote, it is Google's judgment, and no setting reverses it.

By
WP Ministry
Published
Tested on
WordPress 7.1.3, PHP 8.3.35

In short

  • Google fetched the address and chose not to index it. Nothing on the site blocked it, which is why the report names "Google systems" as the source.
  • A default WordPress serves feeds, list pages and archives that answer 200 with no noindex and are not pages anyone searches for. For those, not indexed is the wanted state.
  • For a page you wrote, the status is Google's judgment of that page. No setting, plugin or request reverses it.
  • Check what the site controls first. The page answers 200, carries no noindex, names itself as canonical, is in the sitemap and is linked.
  • The live test in the URL Inspection tool does not test for this status, so a pass there settles nothing about it.
  • A noindex on an archive moves its address to another reason in the report. The number of pages not indexed stays the same.

"Crawled - currently not indexed" means Google fetched the address and chose not to put it in its index. Nothing on the site blocked it. That is why the Page indexing report gives the source of this reason as "Google systems" and not "Website". Whether it is a problem depends on which addresses are listed. If they are feeds, archives and other addresses WordPress makes by itself, not indexed is the right outcome and nothing needs doing. If they are pages you wrote and want found, the status is Google's judgment of those pages, and no setting, plugin or request reverses it.

So sort the list before changing anything.

What Google says the status means

Google's help for the Page indexing report gives the status two sentences. The first: "The page was crawled by Google but not indexed." The second says the page may or may not be indexed later, and that there is "no need to resubmit this URL for crawling". It names no cause and no remedy.

Four more things from Google's own pages set the frame:

  • The source. The report's table has a Source column. The help says that in general only an issue whose source is the website is one you can fix. This reason is listed with "Google systems".
  • No promise. Google's guide to how Search works says "Indexing isn't guaranteed", and that not every page Google processes is indexed.
  • Not indexed is often right. The help says not to expect every address on a site to be indexed, since some are duplicates or hold nothing meaningful. It asks only that your key pages are.
  • The list is a sample. The examples under a reason are limited to 1,000 rows and do not necessarily show every address with that status.

"Discovered - currently not indexed" is a different status, one step earlier. There Google knows the address and has not fetched it. The help explains it by load on the site, not by the page: Google expected the crawl to overload the site and put it off, which is why those rows show no last crawl date. A page under "Crawled" has been read. A page under "Discovered" has not been read yet.

See what an address answers

One command prints what a crawler is told about an address: the status, the content type, an X-Robots-Tag header if one is sent, where a redirect leads, the robots tag and the canonical link. Run it in a terminal, with an address from the report in place of the example.

bash
curl -s -D - "https://example.com/feed/" | tr -d '\r' | grep -i -o -E '^HTTP/[0-9.]+ [0-9]+|^content-type: [^;]*|^x-robots-tag: .*|^location: .*|<meta name=.robots.[^>]*>|<link rel=.canonical.[^>]*>'

For a feed it prints two lines, and for a post four:

text
HTTP/1.1 200
Content-Type: application/rss+xml
text
HTTP/1.1 200
Content-Type: text/html
<meta name='robots' content='max-image-preview:large' />
<link rel="canonical" href="https://example.com/oak-bench-notes/" />

A line that is missing is something the address does not carry. The word to look for is noindex, in the header or in the tag. With no terminal, the page indexing check reads the status, any noindex and the canonical link for one address.

What a default WordPress serves that can land here

This is what a new WordPress answered with "Post name" permalinks, the Twenty Twenty-Five theme, no plugin, one author and twelve posts. Every address in the table answered 200. None sent an X-Robots-Tag header, and the robots tag on the HTML ones was max-image-preview:large and nothing else. Nothing asks a search engine to leave them out, and none is a page a visitor would search for.

AddressWhat it isContent typeCanonical link
/feed/ and /comments/feed/The site's feeds, linked from the head of every pageapplication/rss+xmlNone
/oak-bench-notes/feed/The comment feed of one post, linked from the head of that postapplication/rss+xmlNone
/category/field-notes/feed/, /author/admin/feed/A feed for each archiveapplication/rss+xmlNone
/page/2/The second page of the list of poststext/htmlNone
/author/admin/An author's archive. With one author it lists the same posts as the home pagetext/htmlNone
/2026/, /2026/02/, /2026/03/14/Archives by year, month and daytext/htmlNone
/category/field-notes/, /tag/oak/A category and a tag. With one post in it, the archive repeats that posttext/htmlNone
/xmlrpc.php?rsdThe Really Simple Discovery file, linked from the head of every pagetext/xmlNone
/oak-bench-notes/harbor-at-dawn/An attachment page, on a site where they are ontext/htmlItself

Three notes on the table:

  • WordPress points at these itself. The feed and discovery addresses are <link> tags in every page's head. The sitemap at /wp-sitemap.xml lists author, category and tag archives.
  • A feed is not a mistake. Google's sitemap documentation accepts a feed in place of a sitemap, as a list of recent addresses.
  • Attachment pages are off on a site installed on WordPress 6.4 or later, where the address redirects to the file. Attachment pages in Google covers the sites where they are still on.

WordPress prints a canonical link on single posts and pages only, so the archives have none. What WordPress does with no SEO plugin has the whole grid.

What WordPress already keeps out

These look like candidates and are not, because the same install marks or redirects each one. By Google's help a page with a noindex is reported under "URL marked 'noindex'", and a redirect under Page with redirect.

AddressWhat it answers
/hello-world/?replytocom=1, the Reply link under a comment200, with noindex, follow in the robots tag and a canonical link to the post
/wp-json/ and everything under it200 as application/json, with the header X-Robots-Tag: noindex
/oak-bench-notes/embed/200, with noindex, follow
/?s=oak, a search200, with noindex, follow. See search pages in Google
/wp-login.php200, with noindex, noarchive
/xmlrpc.php405 as plain text: "XML-RPC server accepts POST requests only."
/?p=1, a post's short link301 to the post's address

If one of these sits under "Crawled - currently not indexed" on your site, run the command on it. Something has changed what WordPress sends.

What does not help

  • Asking again and again. Google says there is a quota for these requests, and that asking several times for the same address does not get it crawled any faster.
  • Changing dates. Google's page on helpful content lists changing a page's date without changing its substance among the signs of a page made for search engines first.
  • Writing to a word count. The same page says Google has no preferred word count.
  • Paying for indexing. Google says it accepts no payment to crawl a site more often. Its Indexing API can only be used for pages that carry job posting or livestream markup.

If the list holds many real pages, or grew after a redesign, what to check when traffic drops after a redesign covers what a new structure does to old addresses. A WordPress SEO audit from WP Ministry is a one-time job that reads the Page indexing report beside the site and fixes what can be fixed on the WordPress side. What Google then indexes stays Google's decision.

What causes it

How to fix it

Feeds and archives: confirm what they are, then leave them

  • Easy
  • No risk
  • About 10 minutes
  • Steps tested on WordPress 7.1.3

  1. Step 1: Open the list of examples

    In the Page indexing report, select the reason in the table headed "Why pages aren't indexed". Search Console shows up to 1,000 example addresses.

  2. Step 2: Sort the addresses by kind

    Match each against the first table: an address ending in /feed/, a numbered list page, an author, a date, a category or a tag archive, an attachment page. Set aside anything that is a post or page you wrote.

  3. Step 3: Run the command on one of each kind

    If an address answers as the table says, it is WordPress's ordinary output and not a fault in your site. There is nothing to change.

For these addresses "not indexed" is the wanted state, so the status describes them correctly and is not a fault. A feed is not a page. A date or author archive repeats posts that have addresses of their own. Google's help says it is fine for an address not to be indexed for the right reasons.

An archive is worth indexing only where you have made it a page in its own right, such as a category with a description and enough posts that someone would look for it.

Do not block these addresses in robots.txt to tidy the report. A blocked address is reported under Blocked by robots.txt instead, and Google's help says a robots.txt rule keeps Google from seeing a noindex at all.

Optional: mark a kind of archive noindex

  • Takes care
  • Low risk
  • About 15 minutes
  • Steps tested on WordPress 7.1.3

Do this only if you would rather state the intent yourself than leave the choice to Google. Know what it changes in the report first. Once Google has crawled the addresses again, they are counted under the reason Google's help calls "URL marked 'noindex'". They are still not indexed, and the total of pages not indexed does not go down.

WordPress has no setting for this. Its robots tag is built by a filter named wp_robots, and a small must-use plugin can add to it: a PHP file in wp-content/mu-plugins/ that loads on every request, with no activation and whatever the theme is.

  1. Step 1: Create the file

    Create the folder wp-content/mu-plugins/ if the site has none, and save this in it as noindex-archives.php.

    wp-content/mu-plugins/noindex-archives.php
    <?php
/**
 * Plugin Name: Noindex author and date archives
 */

// Add noindex to the robots tag of author and date archives.
add_filter( 'wp_robots', function ( $robots ) {
	if ( is_author() || is_date() ) {
		return wp_robots_no_robots( $robots );
	}
	return $robots;
} );

// Take author archives out of WordPress's own sitemap, which lists them.
add_filter( 'wp_sitemaps_add_provider', function ( $provider, $name ) {
	return 'users' === $name ? false : $provider;
}, 10, 2 );

  2. Step 2: Check an archive

    Run the command from the top of this page on an author's archive. The robots line now reads max-image-preview:large, noindex, follow. Posts, pages, categories and tags are unchanged.

Copy the file exactly. A syntax error in a file in this folder takes every page down until the file is corrected or removed.

The second half is there because a sitemap is for addresses you want in search results, and WordPress's sitemap lists author archives. For tags, add || is_tag() to the test. Leave out any archive you want found.

The tag cannot reach a feed, which is not an HTML page. Google's documentation says anything that is not HTML needs the X-Robots-Tag header instead. Leaving feeds as they are is the simpler course.

To undo it: Delete the file.

A real page: check what the site controls

  • Easy
  • No risk
  • About 15 minutes
  • Steps tested on WordPress 7.1.3

Google gives no reason for this status, so first rule out everything that is the site's doing.

  1. Step 1: Make sure Google gets the page you see

    Run the command from the top of this page on the page's own address. It should print 200, text/html, a robots tag with no noindex, and a canonical link that is the address itself. Anything else belongs to a different status: start from the checks for a site that is not on Google. In Search Console, "View crawled page" in the URL Inspection tool shows the HTML Google fetched.

  2. Step 2: Look for a second address with the same text

    Run the command on the address with something added, such as ?utm_source=newsletter. WordPress answers 200 and names the clean address as canonical, which is what the link is for. That is the limit of it. Two posts with the same text each name themselves, and so does /oak-bench-notes/comment-page-1/ on a site that breaks comments into pages. Google's report has reasons of its own for addresses it recognizes as duplicates, such as Alternate page with proper canonical tag. A page that repeats most of another without being its copy is the case to look for here: merge the two and redirect the weaker address.

  3. Step 3: Check that the sitemap lists it

    This prints every address in WordPress's sitemap of posts. Pages are in wp-sitemap-posts-page-1.xml. An SEO plugin may serve a sitemap of its own: the sitemap checker finds it.

    bash
    curl -s "https://example.com/wp-sitemap-posts-post-1.xml" | grep -o '<loc>[^<]*</loc>'

  4. Step 4: Check that something links to it

    This counts the links to the address on the home page. 0 means the home page has none: put the address of the page that should link to it in place of the home page's and run it again. WordPress lists the newest posts on the home page and older ones on /page/2/ and beyond. Google's help asks that every page can be reached by following links from the home page.

    bash
    curl -s "https://example.com/" | grep -c 'href="https://example.com/oak-bench-notes/"'

A real page: what is left is the page itself

  • Advanced
  • No risk

When the site's side is in order, the page was read as you meant it and not indexed. The report does not say why for any one page. Google's guide to how Search works names low quality of the content among the common reasons a page is not indexed. Its page on helpful content gives questions to put to a page, among them whether it offers original information and whether it gives substantial value beside the other pages in the results.

That leaves three courses:

  • Improve the page so it holds something no other page of yours does.
  • Merge it into a stronger page on the same subject and redirect its address.
  • Accept it. Not every page of a site has to be in the index.

Then wait. Google's help promises no time and no outcome: the page may or may not be indexed later.

Request indexing and Validate fix: what each one does

  • Easy
  • No risk
  • About 5 minutes

Request indexing is a button in the URL Inspection tool. By the tool's help, a page that passes a quick check for indexing errors is put in the indexing queue. Google's page on recrawling says crawling can take from a few days to a few weeks, and that a request does not guarantee the page is included at all. Use it once, after a change of substance.

Test live URL fetches the page now. The help lists this status among those the live test cannot test, and says no test can guarantee that a page will be indexed. A pass means only that Google can fetch the page.

Validate fix tells Google you have fixed an issue. Search Console checks a few pages at once, then queues the addresses it knows under that reason to be crawled again and keeps a log. The help says validation "typically takes up to about two weeks" and can take much longer. It is a request and a record, not a switch: whether a page crawled again is indexed is still Google's decision. The count is updated whenever Google crawls the pages, validation or not.

When to get help

Get help when the list holds pages you wrote and want found, not feeds or archives, and there are many of them, or when the count grew after a redesign. Sorting a long list by kind, and finding what a new theme or a changed structure did to those pages, takes reading the report beside the site.

Common questions

Is "Crawled - currently not indexed" an error?

No. It is one of the reasons the report gives for an address not being indexed, and its source is Google's systems, not the website. For a feed or an archive it describes the right outcome. For a page you want found it is a decision you can respond to and cannot overrule.

How long until Google indexes the page?

Google gives no time. Its help says only that the page may or may not be indexed in the future. A change to the page is seen when Google crawls it again, which its documentation says can take from a few days to a few weeks.

Should I remove my feeds or my archives?

Not for the sake of this report. A feed is how feed readers follow a site, and archives are how visitors browse. The second fix above marks a kind of archive noindex if you want to state the intent, and changes nothing a visitor sees.

Why does the report count more addresses than it lists?

The list under a reason is a sample. Google's help limits it to 1,000 rows and says it does not necessarily show every address with that status.

A page was indexed before and is listed here now. What happened?

Google's help on missing pages says a page can be dropped from the index for innocuous reasons, and that Google does not get to every page. Run the checks in the third fix, and compare the page with others on your site on the same subject.

More on this subject

Would you rather we fixed it?

SEO audit is $249. A written audit, each finding with its source, and up to 2 hours of fixes done and tested again. It starts with a free diagnosis.

Get a free diagnosis