ArrivlBlog
Notes from the agentic web
How AI agents read websites, which ones send customers, and what to fix so you show up in their answers.
How Often Do AI Crawlers Come Back? Most Pages They Crawl Once
AI crawlers arrive in bursts, they pick a small set of pages to return to, and most pages they crawl they crawl once.
How much Googlebot and GPTBot traffic is fake, and how to tell
We matched every crawler request on our own site and six customer sites against the vendors' published IP ranges. On arrivl.ai, 87.8% of Googlebot requests came from Google and 21.9% of GPTBot requests came from OpenAI.
Does anything actually fetch your llms.txt? Here is what the server logs say
46 days of server logs from 39 business sites: named AI agents asked for /llms.txt 100 times, while five OpenAI and Anthropic fetchers made 195,245 requests to ordinary pages.
How to Detect AI Traffic in CDN Logs
Every AI request lands in your CDN logs, including the thousands your analytics never counted.
What AI agents actually see when they visit your site
ChatGPT, Claude, and Perplexity don't browse your site the way people do. Here's what they read, what they skip, and how to tell whether you're showing up.
OpenAI doesn't run one crawler. It runs several — and they want different things
GPTBot trains. OAI-SearchBot indexes. ChatGPT-User fetches live. Treating them as one 'AI bot' hides the signal that actually matters.
Reading about it is one thing. Seeing agents on your own site is another.
Start for free