Cloudflare blocks GPTBot, ClaudeBot, and PerplexityBot by default on new zones. Check Security > Bots > AI Crawl Control before it costs you AI citations.
Cloudflare has been quietly deleting SaaS blogs from ChatGPT Search and Perplexity since July 2025, and most teams never find out, because the block happens at the edge, before robots.txt is ever read. You can have a perfect on-page setup, a clean robots.txt for AI bots, and a fast, well-structured post, and still never get crawled, because the CDN in front of your origin server said no first.
This is the dashboard-first, Cloudflare-specific audit for that exact failure mode: what's blocked by default, where to check it, the curl test a robots.txt tool can't run, and the new three-way crawler taxonomy that changes the defaults again on September 15, 2026.
If your domain was created on Cloudflare after July 1, 2025, AI crawlers are blocked before you ever touched a setting, and the default doesn't stop at training bots. It catches citation crawlers too, which is the part most teams miss.
Cloudflare announced that "every new domain starts with the default" of blocking AI crawlers, calling itself the first major internet infrastructure provider to flip that switch. The company says it helps manage and protect traffic for roughly 20% of the web, so this was not a niche setting change, it altered crawler access across a fifth of the internet in a single move.
It wasn't a cold start. More than one million site owners had already turned the equivalent one-click blocker on manually before Cloudflare made it automatic, per the same release. CEO Matthew Prince framed the move as a control handoff to publishers: "If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone." Whatever you think of the framing, the practical effect is the one that matters here: if you didn't explicitly choose this, you may already be blocking the crawlers your citation strategy depends on.
The default traces back to a one-click "block all AI bots" toggle Cloudflare shipped on July 3, 2024, and its own announcement of that feature was explicit about the scope: "a brand new one-click to block all AI bots," naming Bytespider, GPTBot, ClaudeBot, and CCBot among the most active crawlers it had detected, plus ChatGPT-User, Claude-Web, and PerplexityBot at lower volume. That's the detail that matters: PerplexityBot and ChatGPT-User are not training crawlers, they're the ones that power citations and live answers, and this list was never scoped to training alone. Cloudflare said the feature "will automatically be updated over time as we see new fingerprints of offending bots," so it isn't a fixed set you can memorize once and trust forever.
Here's the wrinkle worth checking on your own zone. Cloudflare's separate managed-robots.txt feature, the one that writes an actual Disallow file for you, only lists the narrower training-crawler set by name, GPTBot, ClaudeBot, Bytespider, Amazonbot, CCBot, Google-Extended, Applebot-Extended, and meta-externalagent, with no PerplexityBot or ChatGPT-User in that example. So the text file Cloudflare generates can read narrower than what the edge WAF rule actually enforces. A robots.txt that looks training-only is not proof that your search and citation crawlers are getting through. The only way to know for a given crawler is the per-crawler table, covered next.
A block at the CDN layer isn't a suggestion the crawler can weigh; it's a wall the request hits before your application code, your CMS, or your committed robots.txt file are involved at all. AI Crawl Control can deny a request before it reaches your origin, and Cloudflare's Pay Per Crawl feature can return an HTTP 402 Payment Required response to a crawler at the edge, entirely independent of what robots.txt says. Your origin server never logs the visit, because the visit never arrived.
That's also why a robots.txt checker won't tell you the truth here, which is the next problem to solve.
Checking whether Cloudflare is blocking AI crawlers on your site takes two steps: read the per-crawler table in the dashboard, then confirm it live with a request that actually hits the edge. Skip the second step and you're trusting a file that Cloudflare itself can override.
Log into the Cloudflare dashboard, select your domain, and go to Security > Bots > AI Crawl Control, then open the Crawlers tab. Cloudflare's own documentation describes what that tab shows: a name and operator, a category, "the total number of allowed and unsuccessful requests, with trend chart," a count of robots.txt violations, and an action column where you set Allow or Block. This is the ground truth for your zone, and it's the only place that tells you what's actually happening rather than what you assume is happening based on a file you wrote months ago.
If you block a crawler here, you can also configure exactly what it gets back, a 403 Forbidden or a 402 Payment Required, plus a custom message body. That detail matters for diagnosis: a 402 on a crawler you thought you allowed is a strong tell that Pay Per Crawl or an AI Crawl Control rule is intercepting it, not your robots.txt.
A standard robots.txt tester does exactly one thing: it reads the text of your robots.txt file and evaluates the rules written there. That's a real answer to a narrower question than the one you're actually asking. Cloudflare's managed robots.txt doesn't replace your file, it prepends Cloudflare's own rules ahead of your file in the same response, "combining both into a single response." A tester reading that combined output might still show your intended rule further down and call it "allowed," while the crawler itself, obeying the first matching directive, already stopped at Cloudflare's line. Even when the file itself is fully honest, AI Crawl Control's edge enforcement happens independent of what's written there at all. Neither failure mode shows up in a text-only checker, because neither one is a text problem.
The only test that sees what a real crawler sees is a real request. Curl your live URL with the crawler's exact user agent and read the status code back:
curl -sI -A "GPTBot" https://yourdomain.com/ | head -n 1
curl -sI -A "ClaudeBot" https://yourdomain.com/ | head -n 1
curl -sI -A "PerplexityBot" https://yourdomain.com/ | head -n 1A 200 means the request reached your origin. A 403 or 402 means something at the edge, either the managed robots.txt or an AI Crawl Control rule, intercepted it before your server ever saw the request. Run this against production, not staging, since bot rules commonly differ between environments. If you want to go a level deeper and confirm real crawler traffic is hitting your logs rather than being silently dropped, that's the log-side half of this problem, covered in AI crawler log analysis, including how to tell a genuine GPTBot hit from a spoofed one.
The volume at stake is not small. Anthropic's ClaudeBot has a crawl-to-referral ratio of roughly 23,951 to 1 and OpenAI's GPTBot about 1,276 to 1, against Googlebot's 5 to 1, per Cloudflare Radar data compiled by SEOmator and cited in our own crawler log analysis. Those ratios only apply to bots that reach you in the first place. An edge block doesn't lower the ratio, it zeroes out the crawl entirely, with nothing in your logs to flag it.
Cloudflare is retiring the single "block AI or don't" toggle in favor of a three-way split, and the new default that goes live on September 15, 2026 treats each category differently depending on whether your page carries ads.
Cloudflare now classifies AI crawlers into three functions instead of one bucket. Search crawlers collect or index content to answer questions later, the category OAI-SearchBot, Claude-SearchBot, and PerplexityBot fall into. Agent crawlers act in real time on a person's behalf, the ChatGPT-User and similar fetchers that load a page because someone asked an assistant to look at it. Training crawlers take content to train or fine-tune a model, the category GPTBot and ClaudeBot belong to. Same underlying bots as before, but the policy now targets the job a request is doing rather than the vendor that sent it.
On that date, Cloudflare's new default kicks in for pages that display ads: Training and Agent crawlers get blocked while Search crawlers stay allowed. Cloudflare's own reasoning, in its words: "An ad is a signal that a website owner meant for a person to land there and see it, something monetizable that fuels the business." Existing customers can opt out of the new defaults in Security settings before the date arrives, so this isn't forced on live zones without warning, but it will be the automatic setting on new ones.
Here's the trap. Googlebot, Applebot, and BingBot are multi-purpose, doing both Search and Training work, so Cloudflare evaluates them under both policies at once. Block Training crawlers as a category and you block Googlebot too, even while single-purpose Search crawlers stay allowed. A rule aimed at keeping GPTBot out of your training data can take Googlebot out of your organic search traffic in the same click, if you're not reading the category assignment carefully.
One more thing worth knowing before you start adding directives: Cloudflare also ships a Content-Signal robots.txt extension meant to express finer-grained training/search/AI-input preferences inside the file itself. Google's John Mueller said flatly in a July 6, 2026 Reddit thread, reported by Search Engine Roundtable, that "AFAIK none of the crawlers / llms use the 'content-signal' robots.txt directives," adding that it "was made up by a CDN" and "just adds bloat and future maintenance to your robots.txt file." Treat the dashboard toggles as the real control surface and the Content-Signal syntax as, at best, unproven.
The move that protects your citation traffic without opting into unlimited scraping is a three-part rule, not a single switch:
| Category | What it does | Recommended default |
|---|---|---|
| Search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) | Indexes pages so an assistant can cite you later | Allow, this is your citation traffic |
| Training (GPTBot, ClaudeBot, and similar) | Scrapes content to train future models | Block, unless you specifically want to opt in |
| Agent (ChatGPT-User, Claude-User, and similar) | Fetches a page live because a person asked | Case by case, usually harmless since a real user requested it |
Set each row in AI Crawl Control to match, then rerun the curl test from earlier to confirm the edge agrees with the dashboard. This is the same split our robots.txt for AI bots guide argues for at the file level: block the trainers, allow the citers. The difference is that Cloudflare, not your repo, has the final say on whether that rule actually reaches your origin. Getting the crawler in the door is the prerequisite, not the whole job. What happens after that is the subject of getting cited by ChatGPT, Perplexity, and Claude and, for ChatGPT specifically, how to rank in ChatGPT. And letting crawlers in is a different problem from telling them what to prioritize once they're allowed: that's what an llms.txt file is for, a curated reading list layered on top of an open door, not a substitute for one. Once access is confirmed, the next question is whether it converts into an actual citation, which is what AI citation tracking measures.
Fixing a Cloudflare block gets AI crawlers back to your origin. Lyra writes the posts worth crawling and opens each one as a pull request you review and merge.
FAQ
Yes, since July 1, 2025. Cloudflare made blocking AI crawlers the default setting for every new domain onboarding to its network, the first major internet infrastructure provider to do so. More than a million site owners had already turned the option on manually before it became automatic. If your zone predates that date, check Security > Bots > AI Crawl Control directly, since the default only applies going forward.
Because Cloudflare's managed robots.txt prepends its own AI-blocking rules ahead of your file in the same response, and AI Crawl Control can enforce a block at the network layer before a request ever reaches your origin. Your committed robots.txt can say GPTBot is allowed while Cloudflare's edge returns a 403 or 402. A robots.txt tester only reads the text of the file, so it will not catch this; you have to test the live edge with curl.
Log into the Cloudflare dashboard, open your domain, and go to Security > Bots > AI Crawl Control. Every detected crawler gets a row with its category, a request count, and a block or allow toggle, so you can see exactly what is set without guessing. Then confirm it live: curl your production URL with each crawler's user agent and check the status code, since that is the only test that sees an edge-level block.
Search crawlers collect or index content to answer questions later, the category that includes OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Agent crawlers act in real time on a person's behalf, like ChatGPT-User fetching a page someone asked about. Training crawlers, like GPTBot and ClaudeBot, take content to train or fine-tune a model. Starting September 15, 2026, Cloudflare blocks Training and Agent crawlers by default on ad-monetized pages while leaving Search crawlers allowed.
It can. Googlebot, Applebot, and BingBot are multi-purpose crawlers that do both Search and Training, so Cloudflare evaluates them under both policies. If you block Training crawlers, a multi-purpose crawler is blocked too, even while single-purpose Search crawlers stay allowed. Check the AI Crawl Control table for how a specific crawler is classified before you flip a category-level block.
Built by the tool you're reading about
Lyra finds the topics worth ranking for, writes them in your repo's voice, fact-checks every claim, and opens a pull request scored and ready to merge. You review and hit merge. Want to see what she'd write for you? Start free with three posts, no card.
Keep reading
A Framer to git-based blog migration playbook: why Framer's CMS lacks native categories, tags, and author pages, and how to redirect without losing rankings.
A Ghost to git-based blog migration playbook: export the clean JSON, migrate members and Stripe IDs, and protect the rankings your Ghost blog already built.
Bing SEO for AI citations, explained: 87% of ChatGPT citations track Bing's top results and Copilot runs on Bing. The Webmaster Tools checklist to fix today.