How to block AI crawlers on WordPress, or let them in: robots.txt, Cloudflare, hosts and plugins

There is no single "AI crawler" to block or allow. The companies run separate crawlers for separate jobs. OpenAI and Anthropic each have one that collects pages that may be used to train models, one that builds a search index for the assistant's answers, and one that fetches a single page because a person asked about it. Perplexity has the last two. Each answers to its own name in robots.txt, so a WordPress site can refuse the training crawlers and admit the search ones, or the reverse.

A fresh WordPress does neither. Its robots.txt names no AI crawler at all, which leaves every one of them free to crawl. If a site is blocked anyway, the block is in one of three places an owner does not usually look: a line in robots.txt that something else added, a setting at a CDN or a host that stands in front of WordPress, or a plugin.

The crawlers, and what each one is for

Each row is what the company's own page says, read on October 8, 2026. The name is the one to write after User-agent: in robots.txt.

Anthropic's page is dated April 7, 2026. Google's page on its common crawlers was last updated on July 14, 2026, and its page on AI features on December 10, 2025. The pages of OpenAI, Perplexity, Bing and Common Crawl carry no date.

Three details in that table decide most of what follows.

Training and search are separate at OpenAI and Anthropic, and Perplexity says neither of its crawlers is for training. OpenAI's page says each setting is independent of the others, and gives the example this page is about: allow OAI-SearchBot to appear in search results while disallowing GPTBot.

Google does not split them the same way. Google's page on AI features says AI is built into Search, and that robots.txt rules for Googlebot are the control for Search as a whole. There is no crawler name that admits Google Search and refuses AI Overviews. Google-Extended is about Google's other products, and Google's page on its crawlers says of it:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

The switch for AI features inside Google Search is a setting in Search Console, not a crawler name. How to appear in Google AI Overviews covers it.

A fetch that a user asked for is the weak point of robots.txt. Three companies say so about their own fetchers. OpenAI's page says of ChatGPT-User:

Because these actions are initiated by a user, robots.txt rules may not apply.

Perplexity's page says Perplexity-User generally ignores robots.txt rules, for the same reason. Google's page on user-triggered fetchers, last updated on August 19, 2026, says those fetchers generally ignore robots.txt too; its list includes Google-Agent, used by agents on Google's infrastructure that act on a user's request. Anthropic is the exception: its page says its bots, Claude-User among them, honor robots.txt.

Common Crawl is a non-profit that publishes what it crawls for anyone to use. A post on its own blog, dated February 13, 2024, says many AI models are trained on text of which Common Crawl's data is a large portion. The same post says that blocking CCBot also keeps a site out of the search indexes built from that data.

How a rule for one crawler works in robots.txt

A robots.txt file is made of groups. A group is one or more User-agent lines followed by the rules for the crawlers those lines name. Two rules of the standard, RFC 9309, matter for AI crawlers, and Google's documentation states both:

• A crawler follows the group that names it, and only that group. It falls back to the * group only when no group names it. Google's documentation adds that a named group and the * group are not combined.
• Several User-agent lines can share one set of rules. Four names above one Disallow line is one group that applies to all four.

WordPress writes only a * group: three lines that keep crawlers out of /wp-admin/, and a Sitemap line. Every crawler in the table that honors robots.txt therefore follows those three lines and may fetch everything else. The moment a group names a crawler, that crawler stops reading the * group, so its own group has to repeat the /wp-admin/ lines if they should still apply to it.

WordPress robots.txt covers where the file is, what WordPress answers with, and the three ways to change it. The two examples below use one of them: a small must-use plugin on the robots_txt filter, which keeps WordPress's own lines and adds to them.

Two sets of rules you can use

Both examples are one file, saved as ai-crawler-rules.php in wp-content/mu-plugins/. Create the mu-plugins folder if it is not there; a fresh WordPress does not have it. WordPress loads every PHP file in that folder on each request, with nothing to activate. Use one example or the other, not both: they share a file name so that one replaces the other.

Keep training crawlers out, and let search and user crawlers in

This adds one group. It names OpenAI's and Anthropic's training crawlers, Common Crawl's crawler and Google's name for Gemini, and asks all four to stay out of the whole site.

With the file in place, the site answers /robots.txt with WordPress's lines and then the new group. Here example.com stands in for the site's address.

OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot are not named, so they go on following the * group and may crawl the site. Nothing has to be written to let them in.

One line needs a decision. Google says Google-Extended governs grounding in Gemini Apps as well as training, so that line also asks Google not to use the site's pages to ground Gemini's answers. Google offers no name that separates the two. Delete the line if that is not what you want.

Let every crawler in, and say so

A site that wants to be read by all of them needs no extra rule on a fresh WordPress. Naming the crawlers anyway puts the decision on record, and keeps it standing if someone later tightens the * group.

Allow: / and Disallow: /wp-admin/ both match an address inside the dashboard. The standard says the rule with the longer path is the one to use, so the dashboard stays closed and everything else is open.

Googlebot and Bingbot are left out on purpose. They are search engines' crawlers first, and they already follow the * group.

If you use a file

A file named robots.txt in the site's root replaces WordPress's answer entirely, and the filter then does nothing. Either of the two texts above is a complete file: WordPress's three lines, a Sitemap line that must be your site's real sitemap address, and the group. Do not use the plugin file and a real file together.

To undo either example, delete the file you added.

What robots.txt cannot do

It asks. It does not prevent. The standard says its rules are not a form of access authorization. Cloudflare's documentation puts the same thing in practical terms: compliance is voluntary, and the file does not stop a crawler at a technical level.

It does not bind a fetch a user asked for, at OpenAI, Perplexity or Google, as their own pages say above.

It is not read at once. OpenAI's page says its systems can take about 24 hours to adjust to a change in robots.txt, and Perplexity's says up to 24 hours. Google's documentation says it generally caches the file for up to 24 hours.

It says nothing about what stands in front of the site. A firewall decides before WordPress is asked, and a crawler it turns away never sees a page. That is the subject of the next section.

What can block a crawler before WordPress is asked

Cloudflare

A site that uses Cloudflare has settings for AI crawlers that are separate from robots.txt and are enforced at Cloudflare's network. They changed on September 15, 2026, and Cloudflare's own pages do not all describe the same state, so each statement below is given with its page and date.

Three kinds of crawler. Cloudflare's documentation, last updated on July 1, 2026, sorts AI crawlers by what they do:

Cloudflare's post of July 1, 2026, gives ChatGPT-User as an example of the Agent kind.

Four settings. Each kind is set for the whole domain. Cloudflare's post of September 15, 2026, lists the choices:

One consequence is easy to miss. The same post says that from 15 September, "Block" and "Block on pages with ads" under Training also apply to crawlers that do both jobs, and it names Applebot, Bingbot and Googlebot. A site that now chooses "Block" for Training to keep its content out of AI models is, by Cloudflare's account, also turning Google's and Bing's search crawlers away. The setting that refuses training and keeps search is "Disallow AI Training", and the post says domains that had chosen either kind of block for Training before that date were moved to it.

The defaults for a new domain. From September 15, 2026, per the same post, a domain being added to Cloudflare is offered one of two presets, depending on whether the site earns money from advertising. Any of the settings can be changed during setup or later.

Preference Sync is the feature that publishes a preference in robots.txt. Cloudflare finds the pages with ads itself. Its post of July 1, 2025, says it scans the HTML of a page for the code of common ad units, and adds what browsers report loading.

Domains that were already on Cloudflare. Cloudflare has said two things, and they do not agree.

• Its press release of July 1, 2026, said the new defaults would also be applied on 15 September to all existing free customers who had not changed their settings by that date.
• Its post of September 15, 2026, answers the question of what an existing customer needs to do this way:

Nothing, in almost every case. Your current settings carry over on their own.

That post gives a table for domains that never used the three controls. Where the older "Block AI bots" switch was off, all three are now set to Allow. Where it was on, in either of its two forms, Search is Allow, Training is "Disallow AI Training", and Agent is "Block on pages with ads".

Cloudflare's documentation page, meanwhile, was last updated on July 1, 2026. It still describes 15 September as a date ahead, and gives the new default for Training as a block on pages with ads, where the later post gives "Disallow AI Training". The practical conclusion is not to reason from any of these pages about a particular site. Open the domain and read its three settings.

Lines in robots.txt that Cloudflare writes. Since July 1, 2025, Cloudflare has offered a managed robots.txt. Its documentation, last updated on August 3, 2026, says that when the setting is on and the site already answers /robots.txt, Cloudflare puts its own lines in front of the site's and serves both as one file. The lines sit between two comments, # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content. The example on that page disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. The post of September 15, 2026, says the managed robots.txt is being replaced by a feature it calls Bot Preference Sync, and that customers who had it on are moved over. Either way, the robots.txt a crawler receives can hold groups that neither WordPress nor anyone on the site wrote.

Where to look. In the Cloudflare dashboard, open the domain and go to its Security Settings page. The documentation names the controls "Configure AI bot policies", and says the robots.txt preference is found by filtering that page by "Bot traffic". A separate section, AI Crawl Control, has a Crawlers tab that lists each AI crawler with its allowed and unsuccessful requests, and an Overview tab that shows whether the managed robots.txt is on. The documentation says that on the free plan it recognizes AI crawlers by their user-agent strings.

The host

Some hosts filter AI crawlers on their own servers, and the customer's robots.txt has no say in it. Only a host's own documentation is a reason to say what a host does, so this section names the two whose pages were read. For any other host, ask in writing whether AI crawlers are filtered and which ones.

• SiteGround. Its help page, last updated on September 4, 2025, says it allows by default the crawlers that major AI providers use in a person's chat session, and blocks by default crawlers intended for AI model training. A customer who wants the training crawlers let in has to ask through the Help Desk. A second page, last updated on September 1, 2025, lists what is allowed, including ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. Neither page lists the names it counts as training crawlers.
• Kinsta. Its documentation, last updated on June 12, 2026, describes a "Block AI crawlers" switch under a site's Bot protection in MyKinsta. It says the switch blocks AI crawlers entirely, verified ones included, and does not affect search engine crawlers such as Googlebot and Bing's. The page does not list which names it counts as AI crawlers.

SiteGround's default is the same split as the first example above, made for the customer. It becomes a problem only when an owner believes robots.txt settles the matter in either direction.

A plugin

SEO plugins. Two document a setting for AI crawlers.

• Yoast SEO's help page describes "Block unwanted bots" under Settings, then Advanced, then Crawl optimization. It has four toggles: Google AdsBot in the free plugin, and in Yoast SEO Premium also the Google-Extended name, OpenAI's GPTBot and Common Crawl's CCBot. The list has no toggle for ClaudeBot, and none for any search or user crawler. So it cannot block ChatGPT's or Perplexity's search crawler by accident, and it does not cover Anthropic's training crawler.
• All in One SEO's documentation, last updated on August 14, 2026, describes an "Unwanted Bots" tab in its Crawl Cleanup settings, with a box for each crawler and one for all AI crawlers. It says the feature works by adding rules to the robots.txt that WordPress writes, and that it does not write to a real robots.txt file. The page does not list the crawlers.

Where a plugin works through robots.txt, as All in One SEO says it does, its rules show in robots.txt as served. That is why the first check below is to read it. Yoast's page does not say how its toggles are applied.

Security plugins. A security plugin's rate limiting counts requests, not names. The documentation of one widely used security plugin describes limits on how many pages a crawler may request per minute, and says a crawler over the limit is refused for a time with a 503 status. It also warns that strict settings can block legitimate crawlers. No default that singles out AI crawlers is described there. A limit set low by someone years ago is still worth looking for.

How to check what a crawler gets

Work from the outside in. The first two checks take a minute, and the last is the only one that shows what really happened.

The command reads the status from the ninth item on each line. That is where it sits in the "combined" format, which Apache's documentation defines and which nginx uses when no other format is set. A log kept in another format has the status somewhere else on the line.

What you cannot see from your own computer

A block that applies only to a crawler cannot be seen by loading the page as yourself. Your browser is not the crawler, so the page loads, and the dashboard shows nothing wrong.

Sending the crawler's name from your own computer, as in the fourth check, finds only a rule that goes by the name alone. A firewall can also go by where a request comes from. OpenAI and Perplexity publish the addresses their crawlers use and recommend allowing them, Perplexity's page tells site owners to match on the name and the address together, and Cloudflare's documentation says it verifies a bot by a signature, a published list of addresses or reverse DNS. A request from your computer that borrows the name is not the crawler to a firewall of that kind. It may be let through where the real crawler is blocked, or the other way round.

That leaves two records of what really happened. The access log shows every request that reached the server. A crawler stopped at a CDN never reaches the server, so the log has no line for it at all, and the CDN's own report is the place to look: at Cloudflare, the Crawlers tab of AI Crawl Control.

Opting out of AI training

What each company documents as the way to do it:

• OpenAI: disallow GPTBot in robots.txt.
• Anthropic: disallow ClaudeBot in robots.txt, on every subdomain that should be left out. Its page says blocking its addresses in place of a robots.txt rule may not work, because it stops the bot from reading the file.
• Google: disallow Google-Extended in robots.txt. That covers Gemini training, and grounding in Gemini Apps and Vertex AI. Google's help page for the Search Console control says that control does not affect training.
• Microsoft: not a robots.txt name. Bing's post of September 22, 2023, says content on a page tagged noarchive is not used to train Microsoft's models, and that with nocache only the address, title and snippet may be. Those are robots meta tags on the page. The post says a page with either tag still appears in Bing's search results, but there is a cost: Bing's guidelines say noarchive also keeps a page out of Copilot's answers. Cloudflare's post of September 15, 2026, says Microsoft is building a no-training preference in robots.txt, aimed at early 2027; that is Cloudflare's statement, not one found on a Microsoft page.
• Perplexity: its page says neither of its crawlers collects content for training foundation models, and offers no opt-out for training.
• Common Crawl: disallow CCBot in robots.txt. Its FAQ also points to an opt-out registry.

None of these pages says that content already collected is removed. Anthropic's says a restriction signals that the site's future materials should be left out of its training data, and Google's speaks of future generations of Gemini models. Treat an opt-out as applying from the day it is read.

What each choice costs

Letting a crawler in is not the same as being quoted. It lets the site be read, and none of the pages above says that a site its crawler may read will be shown in an answer. What is documented beyond access is in generative engine optimization. An llms.txt file is a different subject: none of the crawler pages above offers it as a way to admit or refuse a crawler.

When to get help

• Ask the host whether AI crawlers are filtered on its servers, and where the access log is.
• Hand it over if robots.txt as served has lines you cannot trace, or the log shows a crawler refused and you cannot find what refuses it. WP Ministry checks crawler access as part of AI search optimization, and a one-time fix covers one issue on one site, starting with a free diagnosis that gives you a written cause and a fixed quote.