How to Detect AI Traffic in CDN Logs

You suspect AI is hitting your site, and your analytics dashboard shows nothing. Both observations are usually correct at the same time. AI crawlers fetch your HTML and leave before any JavaScript runs, so the GA4 tag that would count them never fires. A site can be crawled thousands of times per day and show zero of it in client analytics.

Every one of those requests still lands in one place: the server and CDN log layer, where a request is recorded whether or not it runs a single line of JavaScript. That is where you detect AI traffic, and you can start in the next minute.

Run this first

If you have a raw access log from any server or CDN, these three commands give you your first real numbers:

# count AI-crawler hits by agent
grep -icE "gptbot" access.log
grep -icE "oai-searchbot" access.log
grep -icE "chatgpt-user|claude-user|perplexity-user" access.log

The first line counts OpenAI's training crawler. The second counts its search indexer. The third counts the live fetchers that fire while a real person waits on an AI answer. A nonzero count on any of them means AI traffic is already on your site, and the rest of this guide tells you how to read it.

The three kinds of AI traffic

AI traffic in a CDN log falls into three kinds. Two of them can come from the same company and carry a completely different meaning for your business, so it pays to separate them from the first count onward.

1. AI crawlers

These are the bots. They fetch your pages to train models, to build search indexes, or to answer a live question. The names to match on in the User-Agent header:

Each agent declares itself as a substring of the User-Agent. GPTBot's full string looks like this:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Matching on the substring is enough for counting. OpenAI documents all three of its bots at https://platform.openai.com/docs/bots, and if you steer them with robots.txt, allow time: OAI-SearchBot picks up robots.txt changes in roughly 24 hours. Perplexity also runs PerplexityBot for indexing; match on the substring PerplexityBot.

ChatGPT-User is the row to watch. It fires when a real person asks ChatGPT something that needs your page, so there is a human waiting on the other end of the request. It hits your homepage first, then the target page, and it arrives from Microsoft Azure IPs. In a log full of automated crawling, ChatGPT-User hits are the closest thing to live demand.

The volume skews hard toward one vendor and toward reading over referring. OpenAI bots account for 69% of all AI traffic, per HUMAN Security's 2026 data. Cloudflare observed GPTBot making 1,276 crawls for every visit it refers back, and ClaudeBot at 23,951 to 1, the highest ratio of any major AI bot. The models read your site far more often than they send people to it.

One name keeps showing up in detection lists and deserves a correction: Google-Extended. Google-Extended is a robots.txt control token. The crawling itself is done by Googlebot and GoogleOther, so grepping your logs for it turns up nothing, and disallowing it in robots.txt leaves your Search inclusion untouched.

2. AI-referred humans

The second kind is people: real visitors who clicked through from an AI answer. You detect them by the Referer host, and the hosts to watch are chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com. Anchor the match on the Referer host itself, never on a substring anywhere in the URL.

These visitors run JavaScript, so they can appear in client analytics. Attribution still breaks often: 32% of Perplexity sessions in GA4 get bucketed under a "(not set)" medium, per Workshop Digital. The server-side Referer field gives you the count directly, with the source host attached to every row.

3. Verified bots and impersonators

Every User-Agent string is self-declared, and anyone can send one. A request claiming to be GPTBot counts as verified only when its source IP falls inside OpenAI's published ranges, the JSON files linked in the table above. Each major vendor publishes its ranges at a stable URL, so the check comes down to a download and a comparison.

The honest framing for your own reporting: UA matching gives you the picture, and IP verification tells you which claims are genuine. Both live in server-side data only.

Detecting AI traffic on Cloudflare

Cloudflare gives you two real paths.

The first path needs no code. Cloudflare Logpush streams the HTTP requests dataset to storage you choose, and each record includes the ClientRequestUserAgent field. Push the logs and filter on the UA substrings above. For a quick look, the dashboard's Security → Events view also filters by user agent.

The second path is a tiny Worker. It captures request metadata server-side and passes the request through untouched:

export default {
  async fetch(request, env, ctx) {
    const ua = request.headers.get("user-agent") || "";
    const referer = request.headers.get("referer") || "";
    const path = new URL(request.url).pathname;
    ctx.waitUntil(recordVisit({ ua, referer, path,
      ip: request.headers.get("cf-connecting-ip") }));
    return fetch(request);
  }
};

recordVisit stands in for whatever storage you point it at. This is the same pattern Arrivl's Cloudflare integration uses: read the headers, record them, and return the origin response so the visitor sees zero difference.

Detecting AI traffic on Vercel

Vercel Log Drains stream request logs as NDJSON to an endpoint you choose, and each proxy record carries the request's user agent, path and status. Point a drain at a small collector and filter for the same UA substrings. Arrivl's Vercel integration is built on this pattern.

What the numbers look like at scale

Once you can see this traffic, the natural question is how much of it to expect. Measured across Arrivl's customer sites over a 21-day window (2026-06-20 to 2026-07-11):

• We analysed 2.17 million bot requests.
• 37% of those bot requests served an AI or LLM purpose rather than traditional search.
• Sites received 22.4 AI-purpose crawls for every one AI-referred human visit.
• 92.6% of AI-referred traffic came from ChatGPT.

Every one of those numbers comes from server-side data. AI activity is already more than a third of bot traffic on these sites, and the crawl-to-visit ratio confirms at fleet scale what Cloudflare's per-bot ratios show for individual agents: the reading happens constantly, and the referrals arrive in a thin stream behind it.

From one grep to a standing answer

The commands above answer today's question, and they are yours to keep running. Arrivl does this detection continuously at the edge, identifying the agent, its company and its intent on every request, instead of one-off log greps. If your stack runs on something other than Cloudflare or Vercel, the integrations hub covers the other paths.

Either way, start with the grep. Run it on this month's access log, split what you find into crawlers, referred humans and unverified claims, and you will know exactly who is reading your site.