Proving Which AI Crawlers Actually Fetched Your Pages

Your bot log is lying to you, and somebody published the receipts

Duane Forrester ran a verification script over his own server logs and printed the raw counts. Twenty seven of thirty three requests claiming to be an AI assistant, 81.8 percent, came from IP addresses outside the vendors' published ranges. The Googlebot number was worse: of 799 requests carrying a Googlebot user agent, 107 verified and 692 did not. All twenty CCBot requests were impostors.

That is one site and a small sample, and I am not going to dress it up as a population estimate. Forrester's own writeup shows the denominators, which is exactly why it is citable. Read it as an existence proof: unverified bot counts can be wrong by an order of magnitude on a real site.

That matters because the whole AI access argument now runs on log evidence. Confirming the crawlers can reach you is a robots.txt and WAF job. Confirming they did reach you is a log job, and it is the one almost nobody finishes.

Stage one, filter: the agent list, and the two lines most people get wrong

Grep your raw access log, not your analytics. AI crawlers fetch HTML over HTTP and do not run your JavaScript, so client side analytics never fires for them. Vercel and MERJ found GPTBot fetched JavaScript files in 11.50 percent of requests and ClaudeBot in 23.84 percent, and neither executed them. That study is from December 2024 and nobody has replicated it at scale since, so treat the rendering conclusion as dated but directionally unchallenged.

One filter line covers the current field:

• grep -E 'GPTBot|OAI-SearchBot|OAI-AdsBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Google-Extended|Google-CloudVertexBot|Bingbot|Amazonbot|Applebot-Extended|CCBot|Bytespider|meta-externalagent|MistralAI-User' access.log > ai-hits.log

Two lines in that list are the ones I see mis-handled on nearly every audit I inherit.

The first is OAI-SearchBot. OpenAI documents four separate bots, and it is OAI-SearchBot, not GPTBot, that surfaces sites in ChatGPT's search features. Blocking GPTBot is a training decision with zero visibility cost. Blocking OAI-SearchBot removes you from ChatGPT search answers.

The second is Google-Extended. Google states it does not impact a site's inclusion in Google Search nor is it used as a ranking signal, and it has no separate HTTP user agent string of its own. If your log shows Google-Extended as a distinct agent, something in your stack is inventing it.

Stage two, verify: the step that turns a count into evidence

Every major vendor now publishes machine readable IP ranges, and each one tells you to use them. Perplexity's documentation says to combine both User-Agent string matching and IP address verification and calls its JSON endpoints the source of truth for WAF configuration. Anthropic states that if a crawler has a source IP on its published list, that indicates the crawler is coming from Anthropic.

The files you need:

Google is deliberately the odd one out, and this trips people up. Google's documented method is a two way DNS check: run a reverse DNS lookup on the accessing IP, confirm the domain is googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that domain and confirm it returns the original IP. A one way reverse lookup is forgeable. The round trip is not.

Runnable, in the order I actually run it:

Unverifiable is a real third bucket, not a rounding error. Some vendors publish nothing, some requests arrive through proxies, and Forrester's method explicitly kept an unverifiable class rather than forcing a binary. Report it as its own column and stop pretending it is clean.

Stage three, the branch: how big the spoofed pile really is

Here the evidence genuinely conflicts, and the honest move is to say so rather than pick the scarier number.

On one side, DataDome's threat research team reports that when roughly 700,000 reachable sites were tested with a spoofed ChatGPT style user agent, 79.7 percent let it through without blocking or challenging it, that Meta-ExternalAgent drew 16.4 million spoofed requests and ChatGPT-User 7.9 million, and that nearly 2.4 percent of requests claiming to be PerplexityBot were fraudulent. On the other, Screaming Frog's own guidance says spoofing of AI bot user agents is currently rare compared to search engine bots.

Both can be true, and I think they are. DataDome measures a security network where attackers self select toward the traffic that gets waved through, and a 2.4 percent fraud rate on PerplexityBot is a long way from 81.8 percent. Forrester measured one small site. The defensible reading is that the spoofed share on any given site is unknown until you check, plausibly small, and occasionally enormous. That is precisely why the check is not optional.

The strongest single case is not a random scraper. Cloudflare documented Perplexity running an undeclared crawler that presented a generic Chrome user agent, used IPs outside its published range, and rotated ASNs to evade blocks, generating three to six million daily requests across tens of thousands of domains. Cloudflare de-listed Perplexity as a verified bot over it. If a named vendor will do that, an anonymous scraper typing GPTBot into a header is not a stretch.

Stage four, classify: a training crawl has nothing to do with being cited today

Once you have a verified set, the single most useful thing you can do with it is split it by purpose. Every vendor has now converged on the same three way architecture, and Search Engine Land's Lauren Busby lays out the training side plainly.

The volume asymmetry is the punchline. Cloudflare measured training at 79 percent of AI crawler traffic in July 2025, search at 17 percent and user actions at 3.2 percent, and a companion analysis put training at nearly 80 percent of AI bot crawling. Those are 2025 network wide figures, not your site, and they are the best public numbers available.

So an unclassified dashboard that shouts "AI crawler traffic up 300 percent" is, four times out of five, reporting training crawls. That is a bandwidth and licensing story. It is not an AI visibility story, and reporting it as one is how GEO retainers lose credibility in month three.

Use this graphic on your site

Free to republish with a link back to this page. Copy the embed code:

<a href="https://josephtimpson.com/insights/ai-crawler-log-analysis"><img src="https://josephtimpson.com/assets/infographics/ai-crawler-log-analysis.svg" alt="Process diagram of five stages of AI crawler log analysis, from raw log through IP verification to a page level coverage report" width="1200" style="max-width:100%;height:auto"></a><p>Graphic by <a href="https://josephtimpson.com/insights/ai-crawler-log-analysis">Joseph Timpson</a></p>

Stage five, roll up: build the coverage table, not a bot count

The deliverable is a join, not a chart. Take your crawl inventory (every URL you care about) and left join the verified retrieval hits onto it. What you want is the rows with nothing on the right.

The zero rows are the finding. A page no verified retrieval agent has ever fetched is not a content problem, an entity problem, or a schema problem. It is an access or discovery problem, and no amount of writing fixes it.

Status codes deserve their own pass. Vercel and MERJ found ChatGPT's crawler wasted 34.82 percent of its fetches on 404s and Claude's 34.16 percent, against 8.22 percent for Googlebot. Again, December 2024 data. But if a third of what an AI crawler requests on your site is missing, your internal linking and redirect hygiene are costing you retrieval that you have already paid to earn.

What logs cannot tell you, said plainly

A verified fetch is not a citation. It is the necessary condition, and the correlation between fetch volume and citation volume is something nobody has published a credible study on. Anyone selling you "crawl frequency drives citations" is extrapolating.

It also cannot tell you about eligibility. Google states a page must be indexed and eligible to be shown with a snippet to appear in generative AI features, which means a stray nosnippet directive can zero out AI presence on a page your logs show being fetched cleanly every week. Logs and index status answer different questions. Run both.

And the two levers most often sold alongside log analysis do not survive contact with the evidence. llms.txt files are overwhelmingly never requested, and schema does not causally lift AI citations. Your logs will confirm the first one for free: check how many hits your own llms.txt has taken.

The half of this pipeline that is on its way out

IP allowlists are a maintenance treadmill and everyone involved knows it. Cloudflare's own framing is blunt: existing identification methods rely on a combination of IP address range, which may be shared by other services or change over time, and user-agent header, which is easily spoofable.

The replacement is cryptographic. Web Bot Auth has crawlers sign their requests using HTTP Message Signatures and publish their public keys at a declared domain, so verification becomes a signature check instead of a range lookup. Cloudflare's Perplexity post already lists it as one of the ways a well behaved crawler should identify themselves honestly, using a unique user-agent, a declared list of IP ranges or Web Bot Auth integration.

My read: stage two of this pipeline gets simpler within eighteen months and stages one, three, four and five do not change at all. Build the pipeline around the classification and the coverage join, because those survive. Do not build your tooling around a hardcoded CIDR list. This is also the direction that makes agent traffic a first class SEO concern rather than a bot management footnote.

If you genuinely cannot get raw logs

Plenty of clients are on managed hosting with no log export. Three usable fallbacks, in the order I reach for them.

Cloudflare's AI Crawl Control is free on all plans with zero configuration and shows crawler activity and request patterns per operator. It is verification you did not have to build, because Cloudflare is doing the bot identification upstream. The tradeoff is that you get their classification, not yours, and no per URL join.

Google's Search Console generative AI performance reports and Bing Webmaster Tools' AI Performance report cover the citation side rather than the fetch side. They are first party, free, and they answer a different question than logs do. Use them together and stop paying for tools that only sample prompts.

A log analyser closes the gap for everyone else. Screaming Frog's Log File Analyser documents an AI bot monitoring workflow with a not-in-log view that surfaces the pages the AI bots you are monitoring never requested. That single view is the closest off the shelf equivalent to the coverage table above. Note that its IP list verification is documented for search engine bots, so the AI agent verification in stage two is still yours to run.

Whatever you use, the report you hand a client is the same one: verified retrieval fetches, by page, by agent, by date, with the spoofed and unverifiable rows shown rather than quietly dropped. That is what makes it reporting rather than storytelling, and it is the evidence layer everything in the rest of the insights library is built on.