ArrivlYou updated a page, and the AI still answers with the old version. The engines duck the obvious question: Claude's own answer is "AI crawler re-crawl frequency varies a lot by crawler, and there's no public standard". OpenAI and Anthropic document what each agent is for and stop short of saying how often one returns, which leaves the person who just updated a pricing page with no number for what normal looks like.
So we measured it in server logs. AI crawlers arrive in bursts, they pick a small set of pages to return to, and most pages they crawl they crawl once.
Key takeaways
- On the sites we measured, ClaudeBot crawled 93% of its pages once and never came back to them inside the window.
- ChatGPT-User, which fetches a page while a person is asking ChatGPT a question, returned to 61% of the pages it crawled on those sites, at a median of 3.2 days.
- OAI-SearchBot returned to 55% of pages on those sites, at a median of 3.0 days, and half of the pages it came back for were blog posts.
- GPTBot returned to a page it had already crawled on 3 of the sites, at a median gap of 2.2 days.
- When ClaudeBot, OAI-SearchBot or ChatGPT-User fetch the same page twice on these sites, 84% to 98% of those repeats land inside 24 hours, in one burst.
- Googlebot on the same sites crawled 23% of its pages once, at a median of 2.1 days.
- Perplexity sent 15,916 requests to these sites in three weeks, and Cloudflare verified none of them, so we report no cadence for it.
How these numbers were measured
We counted only fetches Cloudflare verified at the edge, which means the CDN confirmed a request calling itself GPTBot came from OpenAI. The sample is 9 business websites running Arrivl, between 25 August and 15 September 2026, so these are sites whose owners already watch AI crawlers: read the numbers as measurement from that set, not an industry average. A visit means one or more fetches of the same path by the same bot with no gap of 24 hours or more, so a crawl burst counts once, and query strings come off before grouping. A page counts as "crawled once" when its first visit came at least 14 days before the end of its site's window.
How often each bot comes back
| bot | median days between visits | p25–p75 | pages crawled once | sites behind this row |
|---|---|---|---|---|
| Meta-ExternalAgent | 1.9 | 1.4–2.7 | 9.1% | 5 |
| Googlebot | 2.1 | 1.3–3.6 | 22.8% | 7 |
| bingbot | 2.1 | 1.7–4.0 | 41.9% | 7 |
| GPTBot | 2.2 | 1.0–10.0 | 34.2% | 3 |
| Bytespider | 2.4 | 1.8–4.1 | 59.0% | 5 |
| Claude-User | 2.6 | 1.8–5.1 | 49.5% | 3 |
| ClaudeBot | 3.0 | 2.0–4.8 | 92.7% | 9 |
| OAI-SearchBot | 3.0 | 2.0–4.1 | 45.0% | 8 |
| ChatGPT-User | 3.2 | 1.9–5.4 | 38.9% | 6 |
| GoogleOther | 3.4 | 1.9–6.0 | 67.6% | 2 |
| PetalBot | 4.8 | 2.4–6.4 | 6.8% | 6 |
| Amazonbot | 7.0 | 4.9–7.0 | 38.5% | 3 |
| DuckAssistBot | 7.3 | 3.0–8.0 | 47.2% | 1 |
| Applebot | 7.5 | 5.1–9.4 | 14.8% | 5 |
The p25–p75 pair is the middle half of the gaps. The last column counts how many sites contributed at least 30 measured gaps, so ClaudeBot's row carries more weight than DuckAssistBot's.
Three jobs sit inside that table, and the job explains the rhythm.
Training crawlers sweep once. OpenAI documents GPTBot as the agent "used to crawl content that may be used in training our generative AI foundation models", and Anthropic gives ClaudeBot the same job. They go wide and return rarely: ClaudeBot's return visits landed on 18 pages out of the 246 it touched. Meta's training crawler is the exception, returning to almost everything it crawls.
Search indexers keep a steadier beat. OpenAI describes OAI-SearchBot as the agent that "is used to surface websites in search results in ChatGPT's search features", at a cadence close to Googlebot's and bingbot's here. Applebot and PetalBot run that job on a slower cycle, PetalBot at a median of 4.8 days and Applebot at the slowest median in the table.
Live fetchers follow people. Anthropic documents Claude-User as the agent that "supports Claude AI users", and OpenAI states plainly that "ChatGPT-User is not used for crawling the web in an automatic fashion". Their intervals measure how often somebody asked a question that pulled your page back up.
The bursts matter as much as the gaps. When ClaudeBot fetches the same page repeatedly, those fetches are one crawl burst inside a single day, and the next real visit is days away. The table below counts the gaps between one fetch of a page and the next.
| bot | gaps between repeat fetches | share of gaps under 24 hours |
|---|---|---|
| ClaudeBot | 3,284 | 98.0% |
| OAI-SearchBot | 2,477 | 84.0% |
| ChatGPT-User | 3,041 | 83.9% |
| Claude-User | 443 | 53.0% |
| Amazonbot | 1,510 | 31.5% |
| GPTBot | 243 | 30.0% |
GPTBot and Amazonbot do the opposite: their repeats land on separate days.
Which pages get re-visited
| bot | re-visited pages | home | posts | product / pricing / demo | other |
|---|---|---|---|---|---|
| OAI-SearchBot | 132 | 3.8% | 48.5% | 6.8% | 40.9% |
| ChatGPT-User | 182 | 1.6% | 33.0% | 6.6% | 58.8% |
| Meta-ExternalAgent | 556 | 0.4% | 55.4% | 29.5% | 14.7% |
| GPTBot | 59 | 10.2% | 0% | 18.6% | 71.2% |
| ClaudeBot | 23 | 17.4% | 0% | 4.3% | 78.3% |
| Claude-User | 74 | 2.7% | 14.9% | 1.4% | 81.1% |
| Applebot | 776 | 0.6% | 37.1% | 11.0% | 51.3% |
| Googlebot | 699 | 1.1% | 11.3% | 39.1% | 48.5% |
The two OpenAI fetchers spend their return visits on editorial pages, and blog posts take roughly half of OAI-SearchBot's. The Anthropic pair goes the other way, towards the home page and structural pages, on rows built from 23 and 74 pages, so read those as a direction. Googlebot is the one bot here that keeps returning to commercial pages.
Every bot here crawls far more pages than it returns to, so an engine's re-crawl budget is a short list of URLs.
Perplexity sent 15,916 requests and Cloudflare verified none of them
Perplexity is absent from the tables above, and the reason is worth more than its cadence. PerplexityBot and Perplexity-User sent 15,916 requests to these sites in three weeks, and every one arrived without an edge verification. Cloudflare de-listed Perplexity as a verified bot on 4 August 2025: "Based on Perplexity's observed behavior, which is incompatible with those preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling." Cloudflare also observed Perplexity "using a different user agent that posed as a person crawling with Chrome 124 on a Mac system" (Cloudflare blog, quoted by Search Engine Journal). A log line that says PerplexityBot therefore proves nothing on its own.
What this means for you
I updated a page two weeks ago and ChatGPT still gives the old answer. Normal or broken?
Two weeks of silence on one URL sits inside the normal range on the sites we measured. GPTBot crawled 34% of its pages once, and a quarter of the returns it did make took ten days or more. The agent that moves today's answer is OAI-SearchBot, which OpenAI documents as the crawler that surfaces websites in ChatGPT's search results, and on these sites it returned at a median of 3.0 days. Give that URL a fresh link from a page the crawlers do come back to, then check the answer again in a week.
A page nobody crawled twice. What now?
Treat a page crawled once as a page whose current version no engine has fetched. If your logs are out of reach, start from the pages that carry revenue or answer a buying question, assume the engines hold an old copy, and give each a path back in: link to it from your home page and your best-read posts, and put it in the sitemap with the date you actually changed it. Leave the rest alone, because a site-wide refresh spends effort on pages nobody was going to crawl twice.
Does a GPTBot visit mean ChatGPT will cite me?
No, and the agent name tells you which question a visit does answer. OpenAI separates the jobs: GPTBot collects training content, OAI-SearchBot maintains the search index behind ChatGPT, and ChatGPT-User arrives because a person is asking something now. A fetch records access, and nothing in a log proves an answer quoted you.
I see none of these bots. Is that normal?
Check first whether your logs can see them at all. Across the sites we measured, verified AI-bot fetches ranged from about 400 to about 17,000 over the three weeks, so a quiet log and a busy one are both ordinary. A user-agent string with no verification behind it cannot tell you which fetches were genuine, so ask whoever runs your CDN whether verification is switched on.
Should I block them?
Block them only when you know which agent you are blocking and what you lose. Blocking a training crawler removes your content from a future model, blocking a live fetcher removes your page from an answer somebody is waiting for, and blocking a search indexer removes you from the index. OpenAI states the last cost flatly: "Disallowing OAI-SearchBot will prevent sites from appearing in ChatGPT search answers."
How to check this on your own site
Two checks need no logs. Ask the assistant what your page says and compare its answer with the live page, because a stale answer tells you the version in front of it is older than your edit. Then check your internal links: a page only the navigation menu points at has few routes back in.
The log version takes one request to whoever keeps them: over at least three weeks, list every URL fetched by GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and Claude-User, with the date of each fetch and whether the request passed bot verification at the CDN (on Cloudflare that is the cf_verified_bot_category field). Then strip query strings, collapse fetches of the same page by the same bot into one visit whenever the gap is under 24 hours, and measure days between visits.
FAQ
How often do AI crawlers re-crawl a page?
On the sites we measured, the medians ran from 1.9 days for Meta-ExternalAgent to 7.5 days for Applebot, and most pages each agent crawled it crawled only once. OAI-SearchBot and ChatGPT-User, the agents that serve ChatGPT's answers, returned at medians of 3.0 and 3.2 days, while the training crawlers swept once and moved on.
I updated a page two weeks ago and ChatGPT still shows the old version. Is that normal?
Two weeks of silence on one URL sits inside the normal range on the sites we measured. GPTBot crawled 34% of its pages once, and a quarter of its returns took ten days or more. The move that shortens the wait is a fresh internal link from a page the crawlers do come back to, plus a sitemap entry carrying the date you actually changed the page.
Why does ClaudeBot hit my page many times in one day?
Those hits are one crawl burst. Almost every repeat fetch ClaudeBot made on the sites we measured landed inside the same 24 hours, and the next genuine visit came days later, so collapsing same-day fetches into one visit is what makes the number readable.
Does an AI crawler visit mean the AI will cite my site?
No. A log line proves the agent fetched the page, and it says nothing about whether an answer used it. The useful signal is which agent arrived, because a training crawler, a search indexer and a live fetcher each mean a different thing for your business.
Why do so many of my pages get crawled only once?
Because coverage is the job for most of these agents. Every agent we measured touched far more pages than it came back to, so a page crawled once is the normal case. The fix is to give the pages that matter a reason to be fetched again.
See your own numbers
The list you want is the one this article cannot give you: which of your own pages each agent crawled once, and which it keeps coming back to. Your server or CDN logs hold it, and Arrivl builds it for you if you would rather connect the site and read the answer off a dashboard.