ArrivlResearch
What agents do, measured
Original analysis built on the agent traffic Arrivl tracks — how AI crawlers, indexers, and live fetchers actually behave on real sites, and what that means for showing up in their answers.
GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User: what each one actually does on a real site
Five names, three jobs. On nine sites between 25 August and 15 September 2026, 98.2% of ChatGPT-User requests went to content pages, while 71.0% of ClaudeBot requests went to robots.txt and sitemaps.
How Often Do AI Crawlers Come Back? Most Pages They Crawl Once
AI crawlers arrive in bursts, they pick a small set of pages to return to, and most pages they crawl they crawl once.
How much Googlebot and GPTBot traffic is fake, and how to tell
We matched every crawler request on our own site and six customer sites against the vendors' published IP ranges. On arrivl.ai, 87.8% of Googlebot requests came from Google and 21.9% of GPTBot requests came from OpenAI.
Does anything actually fetch your llms.txt? Here is what the server logs say
46 days of server logs from 39 business sites: named AI agents asked for /llms.txt 100 times, while five OpenAI and Anthropic fetchers made 195,245 requests to ordinary pages.
OpenAI doesn't run one crawler. It runs several — and they want different things
GPTBot trains. OAI-SearchBot indexes. ChatGPT-User fetches live. Treating them as one 'AI bot' hides the signal that actually matters.
Every analysis here started as someone's dashboard. Get yours.
Start for free