Crawl budget optimization: when it actually matters

Most crawl budget content treats it as universal, the kind of technical SEO checklist every site should run through. Google's own documentation disagrees. Gary Illyes of Google Search Relations put a number on it directly: "I think most people don't have to worry about it, and when I say most, it's probably over 90% of sites on the internet don't have to worry about it," he said on the Search Off the Record podcast.

So before you spend a week auditing robots.txt rules, the first question is whether your site is even in the 10% that needs to. This matters more than usual for a growing SaaS blog, because programmatic SEO for SaaS is exactly the kind of push that can take a site from a few hundred pages to a few thousand in a matter of months, which is the range where this question starts being worth asking instead of assuming.

What is crawl budget, and when does it actually matter?

Crawl budget is Google's own term, defined precisely: "the set of URLs that Google can and wants to crawl," according to Google's crawl budget management guide. It splits into two parts that get confused constantly.

Crawl capacity limit is about your server. Google defines it as "the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration." If your server slows down or errors out under Googlebot's requests, Google backs off automatically to avoid hurting your site.

Crawl demand is about your content. It varies based on "a site's size, update frequency, page quality, and relevance, compared to other sites," per the same guide. A site Google trusts and finds useful gets crawled more eagerly. A site full of thin or duplicate pages gets deprioritized, capacity limit or not.

Google gives its own size thresholds for when this becomes worth managing, and it's worth quoting them exactly because most crawl budget articles round them up for drama:

Google calls these "a rough estimate to help you classify your site," not a hard cutoff. But they're the closest thing to an official self-diagnosis tool you'll get, and Illyes has backed them up with a specific number elsewhere: "roughly a million URLs is the baseline before a site owner really needs to care about crawl budget," he told Search Engine Journal.

Run the math on a typical SaaS blog. Even an aggressive publishing pace, say four posts a week for three years, lands you around 600 pages. Add a docs section, a handful of landing pages, and a comparison-page cluster, and you might clear a few thousand. That's still an order of magnitude under the 10,000-page threshold, let alone a million. Most SaaS blogs simply aren't large enough for crawl capacity to be their bottleneck. The GSC Crawl Stats report itself makes this explicit: it's "aimed at advanced users" and unnecessary for sites under 1,000 pages, per Google's support documentation.

The two things crawl budget is not about

Two misconceptions do most of the damage here.

It's not a ranking factor. Getting crawled more doesn't make a page rank higher. Crawl budget determines whether Google finds and indexes a page at all, not where that page lands once it's in the index. Confusing the two leads teams to chase crawl frequency instead of the actual quality signals that move rankings.

It's not something small sites need to touch. If your blog has a few hundred or even a couple thousand pages, and you're not seeing a pile-up of unindexed URLs, crawl budget optimization is solving a problem you don't have. The honest answer for most SaaS teams reading a crawl budget guide is: not yet, maybe never.

The warning sign: "Discovered - currently not indexed" in Search Console

Here's the one signal Google says matters regardless of your page count. If you open the Page Indexing report in Search Console and see a growing bucket of URLs labeled "Discovered - currently not indexed," that's worth investigating even on a blog with a few hundred posts.

The status means exactly what it says: Google knows the URL exists, usually because it showed up in your sitemap or got linked from another page, but Google hasn't crawled or indexed it. The URL is in the queue. It just hasn't been served yet, and in some cases never will be.

Why this is usually a crawl demand problem, not a crawl capacity problem, below the thresholds

Below Google's large-site numbers, "Discovered - currently not indexed" is almost never a server-capacity issue. Your site isn't so big that Googlebot can't physically get to every URL. It's a crawl demand signal: Google's quality model looked at the page, or at pages like it on your site, and decided it isn't worth prioritizing yet.

This is the part that should be reassuring, not alarming. A capacity problem needs infrastructure work, more server headroom, faster response times, things that take budget and time. A demand problem is a content problem, and content problems are the kind you can fix directly: consolidate thin pages, strengthen the ones that remain, and stop publishing more of what triggered the deprioritization in the first place.

It also lines up with how aggressively Google has gotten about scaled, low-value content generally. Google said it expected the March 2024 core update, combined with prior efforts, to collectively reduce low-quality, unoriginal content in search results by 40%. The actual number came in higher: "You'll now see 45% less low-quality, unoriginal content in search results versus the 40% improvement we expected," Google reported once the rollout finished in April 2024. A pile of "Discovered - currently not indexed" URLs on a growing programmatic set is often this exact mechanism working as designed, not a bug to route around.

Fixing it: segmented sitemaps, canonicals, and separating docs from marketing

Google is specific about what actually helps, and it's a short list. Beyond capacity, "there are two ways to increase crawl budget," per Google's own crawl budget page: add server resources, or increase the value of your content. Everything below is really about the second lever, plus spending the budget you already have more efficiently.

Consolidate duplicates and canonicalize instead of noindexing

Google's guidance is direct: "block unimportant pages via robots.txt rather than noindex," because a noindex tag still costs a crawl request. Googlebot has to fetch the page to see the tag before it can honor it. If the goal is saving crawl budget, robots.txt disallow rules stop the request before it happens; noindex doesn't.

Duplicate content is the bigger source of waste. Illyes has put a number on how much of the web is duplicate at all: "60% of the internet is duplicate," largely technical variants like http/https, www/non-www, trailing slashes, and URL parameters, according to Ahrefs' writeup of his comments. A SaaS blog rarely has that scale of duplication, but it's the same failure mode at a smaller size: tag pages, paginated archives, and UTM-tagged URLs that all resolve to near-identical content. Canonicalize them to one URL instead of letting Google crawl five versions of the same page. This is also the exact overlap covered in our guide to fixing keyword cannibalization, where near-duplicate pages competing for the same query waste both crawl budget and rankings.

Split sitemaps by content type so lastmod stays meaningful

A single sitemap mixing blog posts, docs pages, and marketing landing pages muddies the one freshness signal Google actually reads: <lastmod>. Docs update on their own cadence, often tied to a product release. Marketing pages barely change. Blog posts update on publish and the occasional content refresh. Blending all three into one sitemap means Google can't tell which section changed just from the timestamp pattern.

Segmenting into sitemap-blog.xml, sitemap-docs.xml, and sitemap-marketing.xml (referenced from a single sitemap index) keeps each section's lastmod trend legible. It's also a chance to keep the tag honest. Google's Gary Illyes has told site owners with unreliable lastmod dates they'd be better off removing the tag entirely rather than leaving it wrong, a point we cover in more depth in our XML sitemap guide for AI crawlers. If your product has real API documentation, it's worth treating as its own crawl-demand surface with its own update cadence; see our guide on getting API documentation cited by AI for what that section needs beyond a segmented sitemap.

Return real 404/410s, kill redirect chains, keep response time down

The rest of Google's checklist is mechanical, which is exactly why it's easy to skip: return a genuine 404 or 410 for pages that are actually gone, rather than a soft 404 that serves a 200 with "not found" text. Fix redirect chains so a URL resolves in one hop, not three. Keep server response times low enough that Googlebot isn't waiting on your infrastructure. None of these grow your crawl budget. They stop it from being spent on dead ends.

Reading Google Search Console's Crawl Stats report correctly

The Crawl Stats report lives under Settings in Search Console, and it's only available on root-level properties, domain properties or URL-prefix properties set at the root, per Google's documentation. It's genuinely built for advanced diagnosis, not a dashboard to check weekly.

Host status, response code breakdown, and what "normal" looks like

Host status is the first thing to check, and it's a simple traffic light: green, yellow, or red, based on whether Googlebot could fetch your robots.txt, resolve your DNS, and connect to your server over the last 90 days. Red or yellow means something upstream of your content is blocking crawls, worth fixing before touching anything else on this list.

The response code breakdown is next. Google states the expectation plainly: "In normal circumstances, the vast majority of responses should be 200 responses." Redirects (301, 302, 307, 308) and 304 Not Modified are also treated as healthy. What should worry you is a rising share of 5xx server errors, 401/407 authentication errors, redirect errors, DNS failures, or timeouts. Any of those climbing over time is the closest thing to a smoke alarm this report gives you.

Crawl purpose and Googlebot type: separating discovery crawls from refresh crawls

The report also splits requests by crawl purpose, discovery (new URLs Google hasn't crawled before) versus refresh (re-crawling URLs it already knows). A healthy growing blog should show a steady stream of discovery crawls tracking your publishing pace. If discovery drops to near zero while you're still publishing, new posts aren't being found quickly, worth checking your sitemap and internal linking before anything else. Orphaned pages, ones with no internal links pointing to them, are a common cause; our guide to internal linking automation covers how to keep new posts from falling into that gap.

Googlebot type (smartphone, desktop, image, video, and others) matters less for a text-heavy SaaS blog, but a sudden shift toward image or video crawling with no corresponding content change is worth a second look, it usually means something on the page is being interpreted differently than intended.

A crawl budget checklist for a SaaS blog scaling into the thousands of pages

Most of this list is dormant until you cross a real threshold. Treat it as a reference to revisit as your page count grows, not a to-do list for this week.

1. Check your actual page count first. Under a few thousand pages with no "Discovered - currently not indexed" pile-up, you're in the 90%+ that Illyes says doesn't need this yet. Confirm before you optimize.
2. Watch the Page Indexing report for "Discovered - currently not indexed." A rising count is the one signal worth acting on early, regardless of size.
3. Canonicalize duplicates instead of noindexing them. Noindex still costs a crawl request; a disallow rule or canonical tag doesn't.
4. Segment sitemaps by content type (blog, docs, marketing) so lastmod stays a meaningful freshness signal per section.
5. Audit your robots.txt to make sure unimportant URL space, filtered views, internal search results, staging paths, is blocked, not just left to noindex. If you're also managing AI crawler access, our robots.txt for AI bots guide covers blocking training crawlers without blocking the ones that drive citations.
6. Return real 404/410s for removed content and collapse redirect chains to a single hop.
7. Check the Crawl Stats report quarterly once you're past the threshold, not before. Host status, response codes, and crawl purpose are the three numbers that matter.
8. When in doubt, remember the only two real levers: server capacity and content value. There's no third lever hiding in a plugin setting.

The honest takeaway is that crawl budget optimization is a scaling problem, not a launch-day checklist. A SaaS blog that's methodically building out programmatic pages and comparison content will eventually earn its way into the group that needs this. Most blogs reading this today aren't there yet, and the more useful move is making sure the content itself clears the bar, so crawl budget never becomes the bottleneck at all.

Lyra fact-checks every claim before it ships and verifies every link resolves, which is the crawl demand problem in miniature: thin, unverified pages are exactly what gets deprioritized at scale.

Try Lyra → · Talk to the founder