GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User: what each one actually does on a real site

Your server log holds three OpenAI names and two from Anthropic: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User. Most dashboards roll all five into one row labelled AI traffic. That row throws away the only thing worth knowing about the requests, because each name is doing a different job, and two of the five mean a person was waiting on the other end of the fetch.

We measured all five over three weeks on requests whose bot identity Cloudflare verified at the edge, so a request that merely typed a bot's name stays out of every number below. The shapes are far enough apart that you can tell the jobs apart from a log line.

Five names, three jobs

Training crawls: GPTBot and ClaudeBot. OpenAI's bots page says "GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models." Anthropic's support page says "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content."

A search index: OAI-SearchBot. OpenAI: "OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features." Anthropic documents a matching name, Claude-SearchBot, which "navigates the web to improve search result quality for users." Claude-SearchBot stayed absent from the verified rows in our window, so it appears here as documentation only.

A person, right now: ChatGPT-User and Claude-User. OpenAI: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent," and "ChatGPT-User is not used for crawling the web in an automatic fashion." Anthropic: "Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent."

Those sentences give you the intent. The log gives you the behaviour.

The sample, and what it is worth

The measurement covers nine websites running Arrivl, from 25 August to 15 September 2026. They are a legal-tech vendor, an online school, a B2B software firm, e-commerce and services sites, small and mid-size businesses.

Nine is a small number and they are all our customers, so here is exactly how they were chosen. Forty Arrivl sites took AI-bot traffic across August and September 2026. Thirty-one of them carried no edge verification on their fetches, and the nine are the ones whose requests carry Cloudflare's verification field, which is what separates a genuine GPTBot request from a request that typed the name. Every row below counts verified requests only. That is also the part you can check yourself: if you run a Cloudflare zone, the same field sits on your own requests, and the commands at the end of this post give you your own version of every column.

Two kinds of request run through the tables. A bot either reads your site, or it reads the files that describe your site: /robots.txt, /sitemap*.xml, /llms.txt, /llms-full.txt, anything under /.well-known/. Call the second group agent files. Content pages are everything a reader would open, the home page, posts, product and pricing pages and docs. Assets are images, scripts, styles, fonts and PDFs. A visit means requests from the same bot, the same site and the same source address with no gap of 30 minutes or more between them.

The measured table

Those columns weigh requests. Weighing visits instead answers a different question, the one an SEO usually wants: when this bot shows up at all, does it ever reach a page you wrote? Share of a bot's visits that touch at least one content page, worst to best: ClaudeBot 14.1%, GPTBot 39.9%, OAI-SearchBot 45.0%, Claude-User 78.9%, Googlebot 83.6%, ChatGPT-User 98.7%. Same sample, different denominator, and the spread across it is the whole argument of this post.

The pages Google wants and the AI bots skip

Googlebot spent 42.5% of its requests on product, pricing, demo, signup and contact paths. The five AI names spent between 1.5% and 15.4% of theirs there.

Googlebot also sent more traffic than all five AI names combined, 13,129 verified requests against 11,130, at 84.2 requests per site-day. Whatever you are planning for AI crawlers, the commercial half of your site is still being read mostly by Google.

ClaudeBot: a file walk, and its 404s are free advice

ClaudeBot sent 3,763 requests across all 9 sites and spent 71.4% of them on agent files: robots.txt took 43.1% (1,622 requests) and sitemaps took 27.9% (sitemap.xml 610, sitemap_index.xml 439). Content pages took 17.8%. 93.8% of its visits open at robots.txt or a sitemap, and 14.1% of its visits reach a content page at all.

Its 4xx rate of 13.5% is the highest of the five, and the misses are worth reading. It asked for /sitemap_index.xml 126 times, /project-sitemap.xml 108 times and /page-sitemap.xml 106 times on sites that serve none of them. ClaudeBot guesses common sitemap filenames and takes the error. Grep your own 404 log for those three paths this afternoon; each hit is a sitemap a crawler expected to find.

The reading it does is wide and shallow. It touched 452 distinct content pages, more than any other AI name here, and 92.7% of the pages it fetched were fetched once and never again in the window. Where it did return, the median gap was 3.0 days, and 98.0% of its raw repeat fetches happened inside a single day. It makes one pass, covers more content pages than any other AI name here, and moves on.

OAI-SearchBot: it reads robots.txt more than it reads you

OAI-SearchBot spent 54.3% of its 2,839 requests on /robots.txt. That is 1,541 permission checks against 1,298 requests for everything else on the site. Its visits match: 55.1% start at robots.txt, 96.5% are a single page, and the mean is 1.06 pages per visit, the most single-minded number in the table. Its 4xx responses include 23 requests for a robots.txt file the site answered with an error.

The frequency has a documented consequence. OpenAI's bots page says "For search results, please note it can take ~24 hours from a site's robots.txt update for our systems to adjust," so a rule you change today is read on roughly that clock.

When it does read, it reads posts and it comes back for them. OAI-SearchBot returned to 55% of the pages it fetched, at a median of 3.0 days, and 48.5% of its re-visited pages were blog posts, the highest post share among the six bots in the table above. 84.0% of its repeat fetches of the same page happened within 24 hours, in one burst.

This is the name attached to ChatGPT search, and OpenAI states the cost of a block plainly: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

GPTBot: the quietest of the five, and the thinnest sample

GPTBot sent 474 verified requests across 7 of the 9 sites, which works out at 4.6 requests per site-day. That is a thin sample, so treat what follows as a shape rather than a rate, and leave the decimals out of anything you repeat.

It enters through the sitemap. 60.6% of its visits start at a sitemap file, and all 118 of its agent-file requests are sitemaps (sitemap_index.xml 70, sitemap.xml 48). Then it stops: 88.6% of its visits are a single page. It also leans late, with 51.7% of its requests landing between 18:00 and 24:00 UTC and 37.6% falling at the weekend, the highest weekend share in the table.

One habit here is operational. GPTBot spreads its repeat fetches across days: 30.0% of its raw re-fetches happened within 24 hours, against 84% to 98% for ClaudeBot, OAI-SearchBot and ChatGPT-User, which pack their repeats into one burst. Claude-User sits between the two at 53.0%. GPTBot came back to a page it had already read on 3 of the 9 sites, at a median gap of 2.2 days, and a quarter of those returns took 10 days or more. If you want to know whether GPTBot has seen a page, look across a fortnight of logs rather than a day.

ChatGPT-User: one page, one person, working hours

ChatGPT-User is the row that means somebody typed a question. 98.2% of its 3,410 requests went to content pages, the highest of the six. It reads a median of 1 page per visit, a mean of 1.14, and 91.8% of its 2,914 visits stop at that one page.

It also arrives where a reader would arrive. 44.9% of its visits open on the home page and 32.4% open on a post, with 1.4% starting at an agent file. It skips the discovery ritual and goes straight to the URL it was sent for.

The clock backs up the reading. 33.1% of its requests landed between 12:00 and 18:00 UTC, and 24.1% of them fell at the weekend. Claude-User, the other user-triggered fetcher, sits lower still at 15.4% weekend. The three crawl-side names run between 26.7% and 37.6%. People take Saturdays off, and the crawlers hold the same pace all week. These sites sit in Asia and Europe, so convert the hours before you read a working day into them.

It came back to 61% of the pages it read, at a median of 3.2 days between visits. The address pool is wide: 1,337 distinct source addresses for 3,410 requests, and OpenAI publishes 213 prefixes for this fetcher in https://openai.com/chatgpt-user.json.

Two boundaries are worth stating plainly. OpenAI warns that robots.txt has limited grip on this agent: "Because these actions are initiated by a user, robots.txt rules may not apply." And a ChatGPT-User row records that your page was fetched while a person was asking a question. What the answer then said about you lives outside your logs.

Claude-User: five addresses, and it asks first

Claude-User sent 644 requests to 6 sites from 5 distinct source addresses. Anthropic's published IP list names its fetcher addresses individually as /32 entries, at https://claude.com/crawling/bots.json, which makes this the cheapest of the five to verify on your own side.

It is the politest arrival in the table. 80.7% of its visits open at robots.txt, and among its multi-page visits, 83.6% read robots.txt first and then fetch a content page. The comparable figure is 18.9% for Googlebot and 12.7% for ClaudeBot. After the check it reads: 64.4% of its requests are content pages, at a median of 2 pages and a mean of 2.64 per visit. Its clock matches a person's, with 42.7% of its requests between 12:00 and 18:00 UTC.

All of this rests on 644 requests from 5 addresses, so the counts matter as much as the percentages.

What reads llms.txt

Across the whole window, requests for /llms.txt or /llms-full.txt from these five names came to three. ChatGPT-User made all three. GPTBot, OAI-SearchBot, ClaudeBot, Claude-User and Googlebot made none.

Other verified bots did ask: PetalBot 9 requests, Amazonbot 4, Applebot 1, bingbot 1. Whether each of the nine sites publishes an llms.txt sat outside this measurement, so read the count as what these five names asked for, rather than as proof that a file was missing. On the evidence here, the OpenAI and Anthropic bots find your content through sitemaps and robots.txt, and llms.txt sits well down their list.

The robots.txt decision, per name

Anthropic states that its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and "respect anti-circumvention technologies (e.g., we will not attempt to bypass CAPTCHAs)", and supports the non-standard Crawl-delay extension. Each token below costs you something different.

The two user-triggered names deserve the slowest decision, because a disallow there lands on a person who has already asked about you by name.

Sorting your own log

One pass gives you the two columns that separate the jobs: which names arrived, and how much of each name's traffic was permission checking rather than reading. Match on the user-agent field rather than on the whole line, because a bot name sitting in a referrer will otherwise inflate every count.

awk -F'"' '
  $6 ~ /GPTBot/        {b="GPTBot"}
  $6 ~ /OAI-SearchBot/ {b="OAI-SearchBot"}
  $6 ~ /ChatGPT-User/  {b="ChatGPT-User"}
  $6 ~ /ClaudeBot/     {b="ClaudeBot"}
  $6 ~ /Claude-User/   {b="Claude-User"}
  b == "" {next}
  { class = ($2 ~ /robots\.txt|sitemap.*\.xml|llms.*\.txt/) ? "agent-file" : "content"
    n[b" "class]++; b="" }
  END { for (k in n) print n[k], k }
' access.log | sort -k2

That reads combined log format, where $6 is the user agent and $2 is the request line. Adjust the two field numbers for your format. If your logs arrive as JSON from a log pipeline, pull the user-agent and path fields out first and feed the same two tests. A name whose requests land mostly in agent-file is crawling or indexing you. A name whose requests land almost entirely in content arrived with a URL already in hand.

Read the user-agent string whole while you are in there. OpenAI adds a hint to it: "When fetching robots.txt files, we may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources, especially when logs do not include paths."

Then verify before you trust any of it, because a user-agent string is free to type. Each vendor publishes its crawler address ranges as JSON with the same prefixes shape, and the Python standard library does the matching:

curl -sO https://openai.com/gptbot.json
python3 -c '
import json, ipaddress
nets = [ipaddress.ip_network(v) for p in json.load(open("gptbot.json"))["prefixes"] for k, v in p.items() if k.endswith("Prefix")]
for ip in ["20.171.207.10", "34.148.127.170"]:
    a = ipaddress.ip_address(ip)
    print(ip, "listed" if any(a in n for n in nets) else "not listed")
'

On 2026-09-16 that printed 20.171.207.10 listed and 34.148.127.170 not listed, from the 21 prefixes in the file. Swap the file and the address list for the name and the rows you are checking: https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json, https://claude.com/crawling/bots.json. Claude-User is the fast one, with five addresses doing all of its work in our window.

Our earlier post How to Detect AI Traffic in CDN Logs covers the fields to keep in your pipeline so these queries stay cheap to run. If you would rather have the split standing than re-run it every month, that is our product: Arrivl separates live-answer fetches from index and training crawls on the dashboard, carries the same split per crawler in the weekly report, and keeps the edge-verified lane beside our own verification so a confirmed crawler and a merely claimed one never land in one number.