How much Googlebot and GPTBot traffic is fake, and how to tell

Every guide about allowing or blocking AI crawlers carries the same sentence: user agents can be spoofed. The guide then moves on to robots.txt syntax and leaves the interesting question sitting there. On your site, this month, what share of the rows claiming to be GPTBot actually came from OpenAI?

We ran that check on our own site and on six customer sites. Below are the ratios, the three lanes that produce them, and the commands that get you your own version of the number.

What our own logs say

Between 2026-08-25 and 2026-09-14, arrivl.ai took 23,683 requests from 2,478 distinct addresses through Arrivl's own install. We took every request carrying one of seven crawler names and matched its source address against that vendor's published IP ranges, downloaded the same week.

Read the counts alongside the percentages. The Googlebot ratio is 137 matches out of 156 requests, and the GPTBot ratio is 32 out of 146, so the decimal places run finer than the sample supports. The asymmetry between the two names is the durable part, and it holds up on a larger sample: the six-site table further down runs the identical test on more than twenty times the crawler-named volume, and there Googlebot comes out at 90.2% while GPTBot comes out at 7.6%.

That asymmetry is the whole point. The Googlebot row and the GPTBot row came out of one log file, in one format, under names of equal authority. One of them is mostly the vendor. The other one is mostly somebody else.

Three checks produce these tables, and you can run all three yourself.

Lane 1: the vendor's own IP list

Google, OpenAI, Anthropic and Perplexity each publish a JSON file of the address ranges their crawlers use. Fetch them, then ask whether the address in your log falls inside one of the prefixes.

https://developers.google.com/static/crawling/ipranges/common-crawlers.json   # Googlebot, 317 prefixes
https://openai.com/gptbot.json                                                # 21 prefixes
https://openai.com/searchbot.json                                             # 39
https://openai.com/chatgpt-user.json                                          # 213
https://claude.com/crawling/bots.json                                         # 26, creationTime 2026-08-18
https://www.perplexity.com/perplexitybot.json                                 # 8

Prefix counts are what each file contained on 2026-09-16. Google publishes four more files in the same directory for its special crawlers and user-triggered fetchers: special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json. OpenAI's bots page asks site owners to work from these files directly: "we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges below."

Every one of these files has the same shape, so one matcher reads all of them:

{"creationTime": "...", "prefixes": [{"ipv4Prefix": "132.196.86.0/24"}, {"ipv6Prefix": "..."}]}

Know what a match means before you act on it. These lists cover the vendor's own servers, so an agent that a person runs on their own machine arrives from an address the list will never contain. A low match rate opens the investigation, and the next two lanes finish it.

Two names in our own bot registry have an empty ip_ranges_url: bingbot and CCBot. We hold no IP-range file for either one, so for those two the list lane stays empty on our side and the reverse-DNS lane does the work.

Lane 2: reverse DNS, with the forward confirm

Google's page "Verifying Googlebot and other Google crawlers" gives four manual steps. Step three is the one people skip, and it is the one that makes the check sound.

1. "Run a reverse DNS lookup on the accessing IP address from your logs, using the host command."
2. "Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com."
3. "Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name."
4. "Verify that it's the same as the original accessing IP address from your logs."

A PTR record is set by whoever controls the address block, so a forward lookup back to the original address is what closes the loop. Here is a genuine Googlebot address from our logs, both directions:

$ dig +short -x 66.249.79.229
crawl-66-249-79-229.googlebot.com.

$ dig +short crawl-66-249-79-229.googlebot.com
66.249.79.229

Google's crawler masks are crawl-***-***-***-***.googlebot.com and geo-crawl-***-***-***-***.geo.googlebot.com for the common crawlers, rate-limited-proxy-***.google.com for the special-case crawlers, and ***.gae.googleusercontent.com or google-proxy-***.google.com for user-triggered fetchers. Now compare a machine that sent us a Googlebot user agent:

$ dig +short -x 34.148.127.170
170.127.148.34.bc.googleusercontent.com.

Read the label in front of googleusercontent.com. Google's crawler masks use gae. This one says bc, the shape Google Compute Engine gives a rented VM. Anyone with a credit card can run a machine there for an afternoon.

One caution before you write a rule around this. A cloud PTR on its own proves nothing, because several vendors run their fetchers on rented cloud and say so. Anthropic's published list includes 34.162.230.222/32, whose PTR is 222.230.162.34.bc.googleusercontent.com, and Cloudflare vouches for that address as an AI Assistant on every Claude-User row we have from it. A PTR-only rule would have called Anthropic an impostor. Reverse DNS refutes a claim when the vendor publishes its own PTR domain, as Google does, or when you read it together with the list check.

Lane 3: Cloudflare has already checked, if your traffic passes through it

If your site sits behind Cloudflare, a third party has run this verification on every request that reached you. Cloudflare's verified-bots documentation describes a verified bot as one that is "transparent about who it is and what it does", identified by "Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS", and non-abusive about robots.txt and request rate. The verdict arrives as one of Cloudflare's category strings: Search Engine Crawler, AI Crawler, AI Assistant, AI Search, and others. On our rows, Googlebot and OAI-SearchBot come through as Search Engine Crawler, ClaudeBot and GPTBot as AI Crawler, and ChatGPT-User and Claude-User as AI Assistant.

Our own rows carry that verdict per request in a cf_verified_bot_category field from 2026-08-25 onward, which is what the Cloudflare column in the six-site table below counts.

Read an empty value carefully. NULL means Cloudflare declined to vouch for that request, which is real evidence on a Cloudflare-served site and means nothing at all on a site Cloudflare never sees. PerplexityBot is the case where this lane goes quiet for a different reason: 0 of 7,778 PerplexityBot rows across our six customer stores carry any category, because Cloudflare removed Perplexity from its verified-bot list in 2024 after finding an undeclared stealth crawler. For that name, the vendor list is your only vendor-side evidence.

Who is actually wearing the costumes

We reverse-resolved the unlisted addresses on arrivl.ai on 2026-09-16. Five machines account for nearly all of them, and each one wore many crawler names inside a single day.

The top machine sent 946 requests in a single day, more than any one crawler name collected across the whole three weeks of the first table. The last column counts requests whose path matches .env, .git, .aws, credential, secret, .ssh, actuator, master.key or wp-config. What these machines asked arrivl.ai for while calling themselves crawlers: /.env, /.env.production, /.git/config, /.aws/credentials, /.ssh/id_rsa, /actuator/env, /config/master.key, /wp-config.php.bak, /@fs/root/.aws/credentials, /.env.anthropic, /.env.openai, /.codex/config.toml.

Call that what it is. Those are commodity credential scanners, the kind every public site on the internet receives, and the crawler names are camouflage they picked up because crawler names get through. The AI angle sits in two places. The first is that this traffic lands inside your AI-crawler counts and inflates them. The second is in the paths: /.env.anthropic, /.env.openai and /.codex/config.toml are scanners hunting LLM API keys, which is worth forwarding to whoever owns security at your site.

Across the five machines we counted 39 distinct crawler names, including Amazonbot, Applebot, Baiduspider, bingbot, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-User, Discordbot, Googlebot, GoogleOther, GPTBot, GrokBot, KimiBot, LinkedInBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, QwenBot, Slackbot, xAI-Grok and YandexBot. One machine sent all of these on the same day:

Mozilla/5.0 (compatible; GPTBot/1.0; ...)
Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
CCBot/2.0 (https://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)

The same machines also sent plain Chrome, Firefox and Edge user agents against the same paths. The user agent is a text field they fill in, and they fill it in with whatever gets through.

The same measurement across six customer sites

We ran the identical check on six Arrivl customer sites with a local event store. Rows start at 2026-08-25, and each store ends somewhere between 2026-08-29 and 2026-09-15.

The last column tells you how many sites each row rests on. Every figure is a total across the sites that saw that name, two of them for Googlebot and three or four for the rest, and the ratios move a long way between this table and our own site. ClaudeBot came out at 86.1% on arrivl.ai and 46.0% here. Treat these as two independent samples that agree on the ranking of the names and disagree on the magnitudes. Your own logs are the only sample that answers your question, which is what the commands below are for.

Three rows repay a closer look.

Addresses and requests answer different questions, so report both. On the ChatGPT-User row, 399 of 425 addresses are on OpenAI's list while only 4.6% of requests are. Almost all of that volume came from one address, 2a06:98c0:3600::103, sending a browser user agent with the ChatGPT-User token appended to it. Cloudflare's HTTP headers reference explains what that address is: when a Cloudflare Worker makes a cross-zone subrequest, CF-Connecting-IP is set to that single Worker client address, which every Worker on the platform shares. So the rows came from somebody's Worker fetching the site, and the address alone cannot tell you whose. Our own Worker is one of the things it rules out, because Arrivl's probes carry ArrivlVerify/1.0 and never a vendor crawler name. Take the arithmetic lesson too: one shared address can swamp a request total while the address-level picture stays clean.

PerplexityBot shows the rotating-proxy shape: 5,453 distinct addresses, 8 of them on Perplexity's list, and the unlisted ones sending roughly one request each. Reverse DNS on the largest unlisted claimants mostly returns nothing, and the rest are rented cloud in AWS Tokyo and Google Cloud.

The two independent lanes agree where both are available. On Googlebot, Cloudflare verified 593 requests and the IP list matched 628. On OAI-SearchBot both lanes landed on 487. Where Cloudflare's count runs lower, the reason is usually that a site in the group sits outside a Cloudflare zone, so lane three has nothing to say about it.

What the costumes do to a traffic trend

On one B2B site we measure, the crawler traffic split in half. Over two consecutive weeks in August 2026 we reverse-resolved all 96 addresses that had sent a Googlebot or CCBot user agent. 59 returned a PTR record, and those 59 fell into two groups with no overlap.

28 addresses resolved under crawl-*.googlebot.com. They made 513 requests at a steady cadence, and the site answered 282 of them with a 200.

25 addresses resolved under *.bc.googleusercontent.com, rented Google Compute Engine VMs. They made 5,834 requests. The site answered 200 to zero of them and 404 to 3,847 of them, at rates reaching 316 requests in 4.2 seconds. What they asked for while calling themselves Googlebot: /rclone.conf, /web/.env, /settings.py, /config/secrets.yml, /config/master.key, /.streamlit/secrets.toml, /actuator/env, /.pypirc. Real Googlebot indexes pages and leaves credential files alone, which makes the request path the cheapest tell in this whole post.

The same 25 machines also sent 382 requests under a plain Chrome user agent and collected 206 content pages that way. They read the site as a browser and hunt secrets as Googlebot, from the same hosts. Counting four further addresses the same test surfaced, that site's impostor fleet came to 29 machines, 5,877 requests in costume with exactly one answered 200, and 211 pages taken under 65 other user agents.

The effect on the reported numbers is the part that travels. Counted by the user agent as declared, daily crawler requests on that site rose 61% across the pilot. With the impostor fleet removed, they fell 14%. The sign of the trend flipped, so the declared count and the verified count pointed in opposite directions. The genuine, reverse-DNS-verified Googlebot ran a flat 32 to 46 requests a day the whole time.

Running the checks on your own logs

Order them by cost. The first two steps take minutes and usually settle the question.

1. Pull the crawler rows and group them by source address. grep -E 'GPTBot|Googlebot|ClaudeBot|PerplexityBot|OAI-SearchBot|ChatGPT-User|Claude-User' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -50, with the field number adjusted to your log format. A handful of addresses will hold most of the volume.
2. Look at what those top addresses asked for, and at the status codes you returned. A row that wants /.env while calling itself Googlebot has answered your question already, at zero cost.
3. Reverse-resolve the top addresses and run the forward lookup back. dig +short -x <address>, then dig +short <hostname>, then compare the result to the address you started with. A bc.googleusercontent.com PTR under a Googlebot user agent is a rented VM in costume.
4. Match the addresses against the vendor's prefixes. This needs a few hundred CIDR comparisons, which the Python standard library does on its own:

curl -sO https://openai.com/gptbot.json
python3 -c '
import json, ipaddress
nets = [ipaddress.ip_network(v) for p in json.load(open("gptbot.json"))["prefixes"] for k, v in p.items() if k.endswith("Prefix")]
for ip in ["20.171.207.10", "34.148.127.170"]:
    a = ipaddress.ip_address(ip)
    print(ip, "listed" if any(a in n for n in nets) else "not listed")
'

On 2026-09-16 that printed 20.171.207.10 listed and 34.148.127.170 not listed, from the 21 prefixes in the file. Swap the file and the address list for the vendor and the rows you are checking.

1. If your traffic passes through Cloudflare, read its verified-bot category on the same requests and see whether it agrees with the other two lanes.

Then decide allow or block against the verified rows only. Blocking a user agent string in robots.txt is a message to the vendors who identify themselves honestly. A machine that wears 21 crawler names in one day picked those names by what gets through, and robots.txt is addressed to a name.

Those five steps are the answer, and they cost you an afternoon each time you want the number again. That repetition is our product. Arrivl records Cloudflare's verified-bot category on every request of a Cloudflare-served install and shows it in the weekly report as a second lane beside our own reverse-DNS verdict, kept separate rather than averaged into one score, so a crawler that is verified and a crawler that merely claims the name never land in the same count. If you want the log-side version first, our earlier post How to Detect AI Traffic in CDN Logs covers the fields to keep and the queries to run.