Cloudflare content signals policy: what changes September 15

On September 24, 2025, Cloudflare shipped the Content Signals Policy, a new line in robots.txt that let a site owner say, in plain machine-readable terms, how AI companies were allowed to use their content. It was a polite request. Nothing enforced it. On September 15, 2026, that changes: Cloudflare starts blocking the crawlers the policy only used to ask nicely about, and it does it by default on any page carrying an ad.

That's the part worth getting ahead of. The new default doesn't just affect the AI crawlers you meant to keep out. Because of how Cloudflare classifies multi-purpose bots, the same rule can quietly take Googlebot with it. This post walks through what actually changed between the 2025 policy and the 2026 enforcement, how it sits next to a standard robots.txt for AI bots setup, and the exact dashboard steps to check before the deadline.

The Cloudflare content signals policy: how it became three enforced categories

The short version: 2025 added a preference you could write into robots.txt. 2026 adds an actual block that happens whether a crawler reads your robots.txt or not. Those are two different tools solving two different problems, and conflating them is the mistake to avoid.

The 2025 policy was a robots.txt preference: search, ai-input, ai-train

The Content Signals Policy gave site owners three signals to set, each independently, as yes or no. Search covers building a search index and returning links or short excerpts, and specifically excludes AI-generated summaries. ai-input covers using content live in a generative answer, the retrieval-augmented-generation pattern behind AI Overviews and chat answers. ai-train covers using content to train or fine-tune a model outright. No signal for a given use means the site owner stated no preference either way, per the plain-language breakdown of the syntax.

Cloudflare's own default line, shipped automatically to customers on its managed robots.txt, reads Content-signal: search=yes, ai-train=no, use=reference. That fourth field, use, is a separate axis: immediate means a crawler can act on the content in the moment but not retain it, reference allows indexing and excerpting without full reproduction, and full allows summarizing or reproducing the content outright. We pulled a live example straight from the source: curling blog.cloudflare.com/robots.txt today returns Content-Signal: ai-train=yes, search=yes, ai-input=yes, meaning Cloudflare opts its own engineering blog into AI training, alongside search and generative use. The same file frames the whole thing as a legal reservation of rights under Article 4 of the EU's 2019 copyright directive, not just a technical courtesy.

More than 3.8 million domains already run Cloudflare's managed robots.txt, so the policy applied to that base the moment it shipped, no action required. CEO Matthew Prince framed the urgency at launch: "The Internet cannot wait for a solution, while in the meantime, creators' original content is used for profit by other companies." It's a fair complaint about the problem. It says nothing about whether the fix actually works, and that's the catch.

Because here's the reality check. Compliance with the content-signal directive is voluntary, the same as every other robots.txt line, and the search engine most sites care about most has already said it doesn't bother. Google's John Mueller wrote in a July 6, 2026 Reddit thread, reported by Search Engine Roundtable, that "AFAIK none of the crawlers / llms use the 'content-signal' robots.txt directives," adding that it "was made up by a CDN" and "just adds bloat and future maintenance to your robots.txt file." A year after launch, the policy is still a suggestion nobody with real crawl volume is contractually bound to read.

The 2026 change adds enforcement: Search, Agent, and Training crawler categories

If the 2025 policy was a note left in a file nobody has to open, the 2026 change is a lock on the door. On July 1, 2026, in a post titled "Content Independence Day: AI Options," Cloudflare announced it would reclassify AI crawlers into three functional categories: Search (indexing to answer questions later), Agent (acting in real time on a person's behalf, like a fetch triggered by a chat request), and Training (collecting content to train or fine-tune a model). Starting September 15, 2026, on any page where Cloudflare's own ad detection finds an ad unit, Training and Agent crawlers are blocked by default while Search crawlers stay allowed.

Detection is automatic. Cloudflare finds the ad units on a page itself, rather than asking site owners to tag anything manually, and the new default only applies where it finds one. The company's own reasoning for drawing the line at ads: "An ad is a signal that a website owner meant for a person to land there and see it, something monetizable that fuels the business." That's a genuinely different enforcement model from a text signal. This one runs at the edge, ahead of your origin server, whether or not the crawler in question has ever heard of robots.txt.

How this fits your robots.txt for AI bots setup

A content-signal line and a Disallow rule look similar in a robots.txt file and do completely different jobs. Disallow is the directive crawlers that respect robots.txt actually check before requesting a page. Content-signal is metadata about permitted use, read by essentially nobody per Mueller's comment above. If your file only sets ai-train=no and skips the Disallow lines, you've stated a preference and enforced nothing.

The fix is to keep both, doing different work. Our robots.txt for AI bots guide covers the actual enforcement layer: disallow the named training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended), leave the named search crawlers un-disallowed (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and commit the file to your repo so it ships in a reviewable pull request. Add the content-signal line on top of that as a declared-intent layer, not a substitute for it:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow:

User-agent: *
Content-signal: search=yes, ai-input=yes, ai-train=no
Disallow:

The Disallow: / on GPTBot is the rule that matters for compliant crawlers. The Content-signal line on the wildcard group is the paper trail, useful mainly as the express reservation of rights Cloudflare's own file cites, in case a training use later needs a legal basis to challenge. Treat it as a footnote to the real access-control rules, not a replacement for them.

Why the September 15 default change matters if you run ads

If your blog or product pages carry ad units, the September 15 default is not a hypothetical setting somewhere in a dashboard you'll get to eventually. It's the one that decides, automatically, whether GPTBot and ClaudeBot keep reaching those specific pages at all.

The stakes differ a lot by crawler, and knowing which one you're blocking matters. Per SEOmator's read of Cloudflare Radar data, reported by Digital Applied, Anthropic's ClaudeBot had a crawl-to-referral ratio of 23,951 to 1 in Q1 2026, and OpenAI's GPTBot about 1,276 to 1 over the same Jan-Mar window, against roughly 4.9 to 1 for Google's traditional search crawl. Blocking ClaudeBot or GPTBot as Training crawlers costs you almost nothing in return traffic, since they were never sending much back. That's the deliberate, low-cost block the new default is built for.

It also intersects with a decision some site owners are weighing separately: whether to charge Training crawlers instead of blocking them outright through Cloudflare's Pay Per Crawl marketplace. Pay Per Crawl and the September 15 default both key off the same Training and Agent categories, so a page that opts into charging is opting out of the free default block on that category, worth checking if you've set up both.

This is the same access-control discipline that should sit underneath every post a growing blog ships, which is a big part of why we built Lyra to check crawler access as part of the pipeline, not as a separate afterthought someone remembers to do once a year. If you're weighing what that costs against doing it manually, the plans are the honest place to look.

The Googlebot/Applebot overlap risk most site owners miss

Here's where a setting aimed at GPTBot can cost you organic traffic instead. Googlebot, Applebot, and BingBot aren't single-purpose. They do both Search work (indexing for regular results) and Training work (feeding Gemini, Apple Intelligence, and Copilot), so Cloudflare evaluates them under both policies at once. Block the Training category, and a multi-purpose crawler that would otherwise pass the Search check gets blocked anyway, because Cloudflare applies the strictest matching rule when a bot spans categories.

Compare that to Google's crawl-to-referral ratio of roughly 4.9 to 1 above. This is not a bot you can treat like GPTBot. A rule meant to stop AI training scraping can, in the same click, cut the crawler behind your regular search rankings and your eligibility for Google's AI Overviews, which ride Googlebot directly and have no separate crawler or opt-out token. You would not find out from an error message. You'd find out from a slow bleed in Search Console impressions days or weeks later, with nothing in the interface pointing back to a Cloudflare setting you never touched.

The practical takeaway is to never set a Training-category block at the category level without first checking how your specific multi-purpose crawlers are classified. A rule that reads "block Training" in the dashboard is not the same as "block GPTBot."

How to check and configure your settings before the deadline

Two separate places in the Cloudflare dashboard control this, and they can disagree, so check both rather than assuming one implies the other.

Start with Configure AI bot policies and set an explicit choice for each category, since a saved preference is what protects an existing zone from the new default; an unset zone inherits it automatically on September 15, 2026. Then open AI Crawl Control and read the per-crawler table, the same audit our Cloudflare AI crawler blocking guide walks through in more depth, including the curl test that catches a block your robots.txt file never shows. Finally, confirm with your own server logs rather than trusting the dashboard alone. AI crawler log analysis covers verifying that a hit is genuinely GPTBot or ClaudeBot and not a spoofed user agent, and it's the only place an edge-level block shows up as a gap instead of a denial, since a blocked request never reaches your origin to be logged at all.

Once access is settled, the separate question is whether your content is worth crawling in the first place. That's a curated-list problem, not an access-control one, and it's what an llms.txt file is for once the door is correctly open.

Getting crawler access right is the prerequisite; Lyra handles the writing on top of it, opening every new post as a pull request you review before it ships.

Try Lyra → · Talk to the founder