AI crawler log analysis: tracking GPTBot and ClaudeBot

Your analytics dashboard has no idea how many times GPTBot or ClaudeBot read your site last week. Not because the number is small, but because both crawlers never run the JavaScript that fires your tracking pixel in the first place. Server access logs are the only record that catches every one of those visits, and most teams have never looked at theirs for this.

This post covers why GA4 and similar tools structurally miss this traffic, the exact user-agent tokens to search for in your logs, how to confirm a hit is genuinely GPTBot or ClaudeBot rather than a spoofed string, and what it means when a page gets crawled constantly but never generates a citation. It's the layer underneath both our robots.txt for AI bots guide, which controls who's allowed to crawl, and AI citation tracking, which measures whether a citation converted into a visit. This post is about the step in between: did the crawl even happen, and was it really them.

Why analytics tools miss most AI crawler traffic

A page view in GA4 requires a browser to load your page and execute a small JavaScript snippet that sends an event back to Google. GPTBot and ClaudeBot request the HTML and stop there. No browser, no script execution, no event, no line in your dashboard. The visit is real. Your analytics tool has no way to know it happened.

Bots fetch HTML, they don't run your tracking pixel

Vercel's engineering team checked this directly against its own network traffic rather than guessing at crawler behavior. Its finding, stated flatly: "none of the major AI crawlers currently render JavaScript," naming OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Meta-ExternalAgent, and Perplexity's PerplexityBot in the same sentence. Every one of those crawlers fetches a URL once, parses the raw response, and moves on. None of them queue a page for a headless-browser render the way Googlebot does, so none of them ever reach the point where a GA4 snippet, a Segment call, or any other client-side tracker would fire.

We cover the mechanics of that gap, and how it affects what these crawlers can read on a client-rendered page, in JavaScript rendering and AI crawlers. This post picks up downstream of that: even on a fully server-rendered page where GPTBot sees every word, your analytics tool still won't log the visit, because the gap isn't about what the crawler can read, it's about how your tracking tool learns a visit happened at all.

How big the blind spot actually is

It's not a fringe amount of traffic. In a recent one-month window across Vercel's own network, GPTBot generated 569 million requests and Claude's crawlers 370 million, a combined volume Vercel puts at about 20% of the 4.5 billion requests Googlebot made over the same period. GPTBot pulled .js files on about 11.5% of its requests, and Claude's crawler pulled JavaScript and image assets too, but Vercel found zero evidence of either crawler executing any of it. They're not ignoring your scripts, they're downloading the file and discarding it unread, the same way you might grab a PDF you never open.

Zoom out further and the traffic these crawlers represent keeps growing against the rest of the web. Automated bot traffic made up 53% of all web traffic in 2025, up from 51% the year before, with human traffic down to 47%, per Imperva's 2026 Bad Bot Report. AI crawlers are a growing slice of that 53%, and unlike a scraper hammering your checkout page, most of it never shows up anywhere in your marketing stack. Server logs are the only place any of it is recorded.

The AI crawler user-agent tokens to search for in your server logs

The reliable way to find this traffic is to search your raw access logs for the literal token each crawler puts in its user-agent header. Below is the current 2026 set across the vendors that matter for a content site, split by what each one is for.

The 2026 token table

GPTBot's full documented user-agent string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and ChatGPT-User's is ...compatible; ChatGPT-User/1.0; +https://openai.com/bot, per OpenAI's crawler documentation. PerplexityBot and Perplexity-User follow the same compatible; <Name>/1.0; +<url> shape, per Perplexity's docs, and CCBot identifies as CCBot/2.0 (https://commoncrawl.org/faq/). Anthropic's support documentation names ClaudeBot, Claude-User, and Claude-SearchBot as three independently controllable agents, ClaudeBot for training, Claude-User for a person's in-app request, Claude-SearchBot for search indexing, but doesn't publish the full literal header string the way OpenAI and Perplexity do. Grep for the token, not the whole string, and you'll catch all of them regardless.

We wrote about the training-versus-search split behind this table, and what it means for robots.txt, in robots.txt for AI bots. This post reuses the same token set for a different purpose: not deciding who to allow in, but confirming who already showed up.

A grep/awk one-liner to pull them out of access logs

Most web servers write a combined log format with the user-agent as the last quoted field. This pulls every AI crawler hit out of a standard access log and tallies them by token:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot" access.log \
  | grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot" \
  | sort | uniq -c | sort -rn

Run that against a week of logs and you get a plain count, something like:

   4218 GPTBot
   3106 ClaudeBot
    891 OAI-SearchBot
    340 ChatGPT-User
    212 PerplexityBot
     47 Claude-SearchBot

If you want the specific pages each crawler is hitting, not just the totals, add the request path into the pipeline:

awk '{print $7}' access.log | paste -d' ' - <(grep -oE "GPTBot|ClaudeBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Claude-SearchBot" access.log) \
  | sort | uniq -c | sort -rn | head -30

That gives you a ranked list of which URLs each crawler actually reads. It's the fastest way to answer a question no analytics dashboard can: which pages does GPTBot keep coming back to, and which ones has it never touched.

Verifying it's really them, not just the user-agent string

A user-agent string is a header any HTTP client can set to whatever it wants. Before you trust a log line as genuine GPTBot or ClaudeBot traffic, especially if you're about to act on the pattern, cross-check the source IP.

The published IP range files

Both major vendors publish a machine-readable list of the IP ranges their crawlers actually use. OpenAI publishes three separate files: GPTBot at openai.com/gptbot.json, OAI-SearchBot at openai.com/searchbot.json, and ChatGPT-User at openai.com/chatgpt-user.json, per OpenAI's bot documentation. Anthropic publishes one combined verification list at claude.com/crawling/bots.json; if a request's source IP appears on that list, it confirms the traffic came from Anthropic's infrastructure, per Anthropic's support documentation.

In practice, that means fetching the file and checking membership:

curl -s https://openai.com/gptbot.json | jq '.prefixes[].ipv4Prefix'
curl -s https://claude.com/crawling/bots.json | jq '.'

Cross-reference the IPs in your log against those ranges. A GPTBot-labeled hit from an IP outside every published prefix is worth a second look; it might be a legitimate infrastructure change the vendor hasn't published yet, or it might be a scraper wearing the label.

Forward-confirmed reverse DNS, because user-agent strings are trivially spoofed

IP range files need updating on your side every time a vendor changes its infrastructure, and they're not the only check available. The other method, long used to verify Googlebot and equally applicable here, is forward-confirmed reverse DNS. Reverse-lookup the source IP to get a hostname, confirm that hostname belongs to the vendor's domain, then forward-resolve that same hostname and confirm it maps back to the identical IP that made the request. As one guide to the technique puts it, "the forward lookup closes the loop," checking whether the hostname actually maps back to the address you started with, because "a reverse hostname can look plausible while still being stale, misleading, or unrelated to the current requester," per this walkthrough of the method. Neither OpenAI nor Anthropic has published a dedicated reverse-DNS domain the way Google has for Googlebot, so treat the published IP list as your primary check for their crawlers and reverse DNS as a supporting signal when a hit looks unusual and you want a second, independent confirmation before you trust it.

What a crawl-but-no-citation pattern tells you about your content

Once you can reliably tell a real ClaudeBot hit from a fake one, the next question is what to do with the pattern in your logs. The honest answer: a crawl by itself tells you almost nothing.

Why a visit from ClaudeBot is not a citation

Per Cloudflare Radar bot and crawler data covering January 1 through March 16, 2026, compiled and reported by SEOmator, ClaudeBot crawled roughly 23,951 pages for every 1 referral it sent back to a site, and GPTBot's ratio was about 1,276:1, against Googlebot's 5:1. Read that against Googlebot: for every 5 pages Google crawls, it sends you 1 visit. For every 23,951 pages ClaudeBot crawls, it sends you 1. That gap isn't a bug in your setup. Anthropic operates no consumer search product that routes clicks back to publishers the way Google Search does, so a ClaudeBot visit in your logs is overwhelmingly a training fetch, not a signal that anyone read your page in a Claude answer. Helen Pollitt, Head of SEO at Getty Images, put the broader case for logs plainly: "Log files are the most accurate recording of how a user or a bot has navigated around your website," as she told Search Engine Journal, and that accuracy cuts both ways: it tells you exactly what happened, and exactly what didn't.

That's also why a rising crawl count deserves attention on its own terms. ClaudeBot's share of identified AI crawler traffic jumped from 12.1% to 20.0% in a single recent month, the largest single-month share gain any crawler has posted, ahead of both Meta-ExternalAgent (10.2%) and GPTBot (9.6%) individually, per a monthly AI crawler traffic analysis. If your logs show ClaudeBot volume climbing that fast, expect the crawl count on every page to climb with it, referral or not.

Repeated crawls with no citation is a diagnosis, not a dead end

Given that ratio, don't read "no referral from ClaudeBot" as a failure signal on any single page; it's the baseline. What's actually worth investigating is a different pattern: a page that used to get crawled weekly and has quietly gone quiet, or a cluster of pages that get crawled constantly but where your citation tracking (the manual prompt log or GA4 setup from our AI citation tracking guide) never turns up a mention anywhere. That combination, sustained crawl interest with zero downstream citation, usually points at a content problem rather than an access problem: no direct answer near the top of the page, no clear date, no cited sources for the model to lean on with confidence. Fixing that is a writing problem, and it's the one our engine-by-engine playbook for getting cited by ChatGPT, Perplexity, and Claude and guide to ranking in ChatGPT both walk through.

A log-analysis workflow you can run this week

Put the pieces together into a routine you can repeat weekly without a paid tool:

1. Pull last week's access logs from your host or CDN (Vercel, Cloudflare, and most hosts expose raw or exportable logs; check your provider's dashboard if you're not sure where they live).
2. Run the grep/awk pipeline above to tally hits by token and, separately, by page.
3. Spot-check unusual IPs against the published IP files (gptbot.json, searchbot.json, chatgpt-user.json, claude.com/crawling/bots.json), or run forward-confirmed reverse DNS on anything that looks off.
4. Compare week over week. A page's crawl count dropping toward zero over several weeks is a stronger signal than any single week's number.
5. Cross-reference against your citation tracking. Pages with high, sustained crawl counts and no citation activity are your content-quality worklist, not your access-control worklist.

That's the whole loop: identify, verify, compare, and only then act. Skipping the verify step is how a spoofed scraper ends up shaping your content roadmap; skipping the compare step is how a single quiet week gets mistaken for a trend.

If step 5's worklist keeps growing faster than your team can rewrite it, that backlog is the problem the Lyra app is built around: she drafts against a sourced brief and opens the fix as a pull request instead of a blind publish, and the plans scale from a free tier up.

Where this sits next to robots.txt and JavaScript rendering

Three separate questions, three separate tools. robots.txt is access control: it decides which crawlers are allowed to fetch your pages at all, covered in robots.txt for AI bots. JavaScript rendering is a comprehension question: once a crawler is allowed in, can it actually read the content on the page, covered in JavaScript rendering and AI crawlers. Log analysis is neither of those. It's verification: proof that a crawl actually happened, confirmation that it was really the vendor it claims to be, and a read on whether that attention is converting into anything downstream. Get all three wrong and you're flying blind on AI search entirely, which is the larger problem our answer engine optimization guide maps out end to end.

Verifying that GPTBot and ClaudeBot actually crawl your pages only matters if what they find is worth citing, and that's the other half of what Lyra writes: server-renderable, source-backed posts shipped as a pull request you review.

Try Lyra → · Talk to the founder