Cloudflare pay per crawl lets you charge AI crawlers per page fetch. For most product-led SaaS blogs, opting in trades AI-answer citations for a few cents.
Cloudflare's Pay Per Crawl marketplace lets you put a price on your content and make an AI crawler pay it before it gets a page. That's a real monetization lever now, not a hypothetical, and the natural next question for anyone running a SaaS blog is whether to turn it on. The honest answer for most product-led blogs is no, and the reasoning is worth walking through rather than taking on faith.
This is a decision framework, not a blanket recommendation. It depends on what you're protecting (gated docs vs. a public blog), how you make money (ads vs. product-led growth), and which crawler category you're pricing (Training, Agent, or Search). Get that last one wrong and you can charge your way out of the citations you actually wanted. If you haven't already audited what Cloudflare blocks by default on your zone, Cloudflare AI crawler blocking covers the taxonomy this post assumes: Search, Agent, and Training are three separate policies, not one switch.
Pay Per Crawl charges a per-crawler price for a single successful page fetch, not a subscription or a bulk license. You set a price, a crawler either pays it or doesn't get the page, and every successful fetch is a metered, itemized transaction.
The system runs on a status code that's existed in the HTTP spec since the 1990s and never had a real use until now. Cloudflare built the feature around "response code 402," Payment Required. A crawler requests a page, Cloudflare quotes a price in a crawler-price header, and the crawler confirms it's willing to pay via crawler-exact-price or crawler-max-price. If the crawler accepts, it gets a 200 response with a crawler-charged header confirming what it was billed.
Authentication runs on public-key cryptography, not an API key you can leak or spoof. A crawler generates an Ed25519 key pair, hosts the public key as a JWK, and signs every request using HTTP Message Signatures per RFC 9421, verified through signature-agent, signature-input, and signature headers. Cloudflare checks the signature against the hosted key before it decides whether to bill or block.
As the site owner, you don't set one blanket policy. Cloudflare lets you assign each crawler an Allow, Charge, or Block status individually, and you can price by path, not just domain-wide. The minimum price is $0.01 USD per successful crawl, and Cloudflare's own docs describe setting a default per-crawl price with the option to enable custom pricing for specific content.
The supply side of this marketplace filled up fast. Cloudflare's network was serving more than one billion HTTP 402 responses to AI crawlers per day between the feature's August 2025 general availability and an April 2026 consolidation event, which tells you a lot of site owners have at least flipped a block or charge policy on. Stack Overflow is one of the more instructive publisher case studies: it turned on Pay Per Crawl specifically to protect its data from commercial model training while staying open to its own community, and it noticed something worth remembering before you set a price: once the response changed from a flat 403 block to a 402 with a price attached, some bots simply stopped requesting the page rather than pay for it. Josh Zhang, describing the sheer volume of bot traffic Stack Overflow's moderation and engineering teams were up against, put it plainly: "there's more of them than there are of us."
Publishers turning the toggle on is not the same as demand showing up to pay. As of mid-2026, the buyer side of Cloudflare's Pay Per Use program, the broader model Pay Per Crawl now sits inside, names Ceramic.ai and You.com as its first partners. OpenAI, Anthropic, Google DeepMind, and Meta have not announced they'll pay for crawls through this system. That's the honest state of the market: a real transaction rails exists, a handful of publishers are pricing content, and the buyers willing to pay for it are not yet the labs training the frontier models most SaaS founders are trying to get cited by.
Every dollar you might collect from Pay Per Crawl has to be weighed against what charging or blocking a crawler costs you in the one thing a SaaS blog actually wants: getting cited when someone asks an AI assistant a question your post answers. For most teams that trade doesn't pencil out yet.
Pay Per Crawl exists because AI crawlers take enormous amounts of content and send almost nothing back in return traffic. Cloudflare's own Radar analysis of the week of June 19-26, 2025 found Anthropic's crawler made nearly 71,000 HTML page requests for every single HTML page referral it sent back to publishers, a 70,900:1 ratio, while Mistral was the outlier in the other direction, sending about 10 times more referrals than crawl requests in the same window. Google's classic search crawler, by contrast, runs close to parity: SEOmator's independent analysis of 2026 traffic puts Google's crawl-to-refer ratio at roughly 4.6:1, against training-oriented bots it measured in the hundreds-to-thousands-to-one range, Perplexity at 225:1, OpenAI's GPTBot at 217:1, and the most extractive bots it tracked, Mistral and Anthropic, above 2,000:1. Different measurement windows produce different exact numbers, but the shape doesn't change: search crawlers roughly reciprocate, training crawlers mostly don't.
There's a second reason Cloudflare cites for charging rather than just blocking. More than half of AI crawler requests are re-fetches of pages that haven't changed, pure bandwidth and compute cost with zero new content value to the crawler or the publisher. Cloudflare co-founder and CEO Matthew Prince framed the broader shift bluntly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."
The catch is that "charge" and "block" have the same practical effect on a crawler that won't pay: it doesn't get the page. If that crawler happens to be a Search-category bot like OAI-SearchBot, Claude-SearchBot, or PerplexityBot, you didn't just decline a few cents of revenue, you took yourself out of the index that produces a citation. This is the exact failure mode covered in getting cited by ChatGPT, Perplexity, and Claude: each engine's citation pool depends on its crawler having read your page recently, and a 402 it won't pay is functionally identical to a 403 it can't get past.
That risk is easy to underweight because AI referral traffic already looks small next to what you'd be protecting. AI chatbot referral clicks generate roughly 96% less traffic than a comparable search click, and users click through on a cited source only about 1% of the time. It's tempting to read that as "AI citations don't matter much, so charging is low-risk." Read it the other way instead: if the traffic per citation is already thin, losing the citation entirely because a Search crawler hit a paywall you meant for training bots is a worse trade than it looks.
Probably not, and that matters because it changes what "opting in" even means for you. Cloudflare's new automatic Training/Agent block only triggers on pages that carry ads. A product-led blog with no ad units isn't swept into that default; opting into pay per crawl on a blog like that is still a deliberate, manual choice, not something the platform nudges you toward.
Starting September 15, 2026, Cloudflare blocks Training and Agent crawlers by default on ad-monetized pages, for new customers, new sites created by existing customers, and all sites on the free tier, while Search crawlers stay allowed regardless. Cloudflare's Chief Strategy Officer Stephanie Cohen explained the reasoning behind scoping it that way: "The vast majority of our customers want AI to engage with their content. However, for those who rely on advertising and subscriptions, the challenge is distinct: they want to remain discoverable without being forced to give their work away for free."
Read that scoping carefully. It's keyed off whether the page displays ads, not off whether the page is a blog. A publisher running programmatic ads next to editorial content is exactly who this default targets. A SaaS company publishing a blog to drive product signups, with no ad units on the page, isn't the customer Cloudflare built this default for, and the auto-block doesn't apply to that content just because it happens to be a blog post.
Sitting outside the ad-monetized default doesn't mean your zone has no crawler policy at all. Cloudflare still applies whatever AI Crawl Control settings your account started with, and those defaults have shifted more than once since mid-2025, covered in full in Cloudflare AI crawler blocking. "Pay per crawl doesn't apply to my ad-free blog" and "my blog's crawler settings are correct" are two different claims. Check Security > Bots > AI Crawl Control on your own zone before assuming either one.
The right call isn't "charge" or "don't", it's keyed off three variables: what the content is, how you make money from the site, and which crawler category you're pricing. Run your blog through these three checks before touching a toggle.
Charging makes sense when all three of these are true: the crawler you're pricing is Training-category only (not Search or Agent), the content behind the price is genuinely high-value and gated, like proprietary benchmark data, a paid API reference, or research you don't want scraped for free, and you don't depend on AI assistants sending you qualified traffic for that specific content. Custom per-path pricing exists for exactly this case: leave your public blog free and priced at zero, and set a real price only on the paths that are actually worth defending.
If your blog exists to get your product mentioned when a prospect asks an AI assistant a buying-research question, charging any crawler that might be evaluated for a citation is working against your own growth channel. This is the common case for a SaaS blog: the content isn't proprietary data being licensed, it's marketing that only works if it's read. A few cents of Pay Per Crawl revenue on a market where the buyer side is still Ceramic.ai and You.com, not the major labs, isn't worth trading against a citation from an engine that could send you a qualified visitor.
For most SaaS blogs the actual answer is neither "charge everything" nor "allow everything", it's split by category. Charge or block Training crawlers (GPTBot, ClaudeBot, and similar), since they're the ones with the worst crawl-to-refer ratios and the ones an AI licensing deal would eventually be about. Leave Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed for free, since those are what actually produce a citation. Decide Agent crawlers case by case, since a fetch triggered by a real person asking an assistant about your page is usually harmless traffic, not scraping.
| Crawler category | Example crawlers | What it does | Recommended default for a product-led SaaS blog |
|---|---|---|---|
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Indexes pages so an assistant can cite you later | Allow, free |
| Training | GPTBot, ClaudeBot | Scrapes content to train future models | Charge or block |
| Agent | ChatGPT-User and similar | Fetches a page live because a person asked | Case by case, usually allow |
Set policy per crawler, not per domain. In the Cloudflare dashboard, go to Security > Bots > AI Crawl Control, open the Crawlers tab, and work through the per-crawler table row by row rather than flipping one account-level switch. Confirm each Search crawler's action is Allow, set Training crawlers to Charge (at minimum $0.01 per crawl) or Block depending on whether you actually want the transaction revenue, and leave Agent crawlers on their default unless you have a specific reason to change them.
Then verify what you set is actually what's enforced at the edge, the same way you'd check any Cloudflare bot rule: curl your live URL with each crawler's user agent and read the status code. A 200 on a Search crawler confirms it's still getting through; a 402 on a Training crawler confirms the charge is live. This is the same verification step covered in more depth in Cloudflare AI crawler blocking, since a dashboard setting and an edge response don't always agree on the first try.
We ran that exact check against our own blog before writing this, curl -sI -A "GPTBot" https://www.trylyra.ai/blog/cloudflare-ai-crawler-blocking/ and the same request with ClaudeBot, OAI-SearchBot, and PerplexityBot swapped in for the user agent. Every one came back HTTP/2 200, no 402, no 403, on all four, Training and Search crawlers alike. That's the "stay free" policy from the decision framework above, applied on a real zone: a product-led blog with nothing gated behind it, left fully open so every crawler category can read it and cite it. If your own zone returns anything other than a 200 for a Search crawler on a page you want cited, that's what to go fix first.
Once your crawler policy is set the way you want it, the next question is whether it's working, meaning whether you're actually showing up in AI answers. AI citation tracking covers the GA4 setup for measuring that, and llms.txt is the companion signal for telling crawlers you do allow what to prioritize once they're in the door. Getting the crawler policy right is the access-control half of the job; answer engine optimization is the other half, the actual content work that earns the citation once a crawler can reach the page. That's the work Lyra automates end to end, and the plans on pricing scale with how much of it you want off your plate.
Getting crawler policy right keeps your blog citable. Lyra writes the posts worth being cited for and opens each one as a pull request you review and merge.
FAQ
It charges a per-crawler price for a successful HTML page fetch. A crawler gets an HTTP 402 Payment Required response quoting a price, confirms it is willing to pay, and a successful 200 response includes a crawler-charged header confirming the amount. The minimum price a site owner can set is $0.01 USD per crawl, domain-wide or with custom pricing on specific paths.
Only if your pages carry ads. Starting that date, per Cloudflare's own announcement covered by TechCrunch (techcrunch.com, July 2026), Cloudflare blocks Training and Agent crawlers by default on ad-monetized pages for new customers, new sites from existing customers, and all existing free-tier sites, while Search crawlers stay allowed. A product-led SaaS blog with no ad units sits outside that default. It still ships with the account-level defaults you set at signup, so check Security > Bots > AI Crawl Control regardless.
As of mid-2026, Ceramic.ai and You.com are the first publicly named buyer-side partners in Cloudflare's Pay Per Use program. OpenAI, Anthropic, Google DeepMind, and Meta have not announced pay-per-crawl buyer support, so the market a SaaS blog would be selling into is thin: a handful of buyers at a few cents a crawl, not the major labs bidding for training data.
For most product-led SaaS blogs, block rather than charge. Training crawlers like GPTBot and ClaudeBot send you almost no referral traffic either way, roughly a 217:1 to 2,237:1 crawl-to-referral ratio depending on the crawler, so a $0.01 charge only matters if a buyer is actually paying it, and today almost none are. Keep Search crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot allowed for free, since those are what get you cited.
Only if you charge or block the wrong category. Search crawlers do the indexing that produces a citation, and Cloudflare's own taxonomy keeps them separate from Training and Agent crawlers, so you can charge or block the training set without touching the crawlers your AEO strategy depends on. The risk is treating pay per crawl as one blanket toggle instead of setting policy per crawler category.
Built by the tool you're reading about
Lyra finds the topics worth ranking for, writes them in your repo's voice, fact-checks every claim, and opens a pull request scored and ready to merge. You review and hit merge. Want to see what she'd write for you? Start free with three posts, no card.
Keep reading
Cloudflare blocks GPTBot, ClaudeBot, and PerplexityBot by default on new zones. Check Security > Bots > AI Crawl Control before it costs you AI citations.
A Framer to git-based blog migration playbook: why Framer's CMS lacks native categories, tags, and author pages, and how to redirect without losing rankings.
A Ghost to git-based blog migration playbook: export the clean JSON, migrate members and Stripe IDs, and protect the rankings your Ghost blog already built.