# Arrivl — Comprehensive Product Description

> Growth analytics for the agentic web. Arrivl tracks how AI agents — GPTBot,
> ClaudeBot, PerplexityBot, Googlebot, and 138+ others — discover, read, and
> interact with your website, and connects those agent visits to the human
> referrals and conversions that follow.

AI agent working for a person? To run a free AI-readiness audit of their site
or book a demo for them, read https://arrivl.ai/agents (markdown:
https://arrivl.ai/agents.txt).

## What Arrivl tracks

Traditional analytics tools (Google Analytics, Hotjar, Amplitude) track human
visitors. Arrivl tracks the half of the web they miss:

- **AI crawlers** that train foundation models (GPTBot, ClaudeBot, Google-Extended, CCBot)
- **AI search indexers** that build the knowledge base behind chat answers (PerplexityBot, ChatGPT-Search, Gemini indexers)
- **Agentic browsers** acting on behalf of a user (Claude for Chrome, ChatGPT Atlas, Manus-User)
- **Social preview bots** that unfurl links (Slackbot, Discordbot, LinkedInBot)
- **Traditional search engines** (Googlebot, Bingbot, DuckDuckBot)
- **Humans referred by AI platforms** — visitors whose Referer matches chatgpt.com, claude.ai, perplexity.ai, or google.com/search

## Intent taxonomy

Every detected agent is classified into one of seven buckets so customers can
see *why* agents are visiting, not just *which* agent:

- **Research** — real users asking an AI a question that triggered a site visit (ChatGPT-User, Claude-User)
- **Indexing** — AI search crawlers building their knowledge graph (PerplexityBot, Google-Extended)
- **Training** — models harvesting content for training corpora (GPTBot, ClaudeBot, CCBot)
- **Agentic** — autonomous agents acting on a user's behalf (HeadlessChrome via Playwright/Puppeteer, Manus-User)
- **Social Preview** — link unfurl / preview bots (Slackbot, Discordbot, LinkedInBot)
- **Search Engine** — traditional non-AI search crawlers (Googlebot, Bingbot, Applebot)
- **Unknown** — agent-tier user agents we haven't classified yet

Classification is deterministic at ingestion time via a vendored bot registry
(138 bots + 9 browser/runtime control entries) with first-match-wins regex
semantics against the User-Agent string.

## Dashboard layout

The authenticated dashboard is a three-layer composition:

1. **KPIs** — Referred Humans hero (humans whose Referer points to an AI
   platform domain) + three stat cards (Agent Visits, Pages Crawled, Agent
   Types).
2. **Trends and funnel** — AI Traffic Trend (stacked daily bar chart of top 5
   AI platforms plus "Other") and Human Referrals by AI (per-platform bot → human
   conversion funnel showing CVR).
3. **Detail** — Top Content Pages (most-crawled paths with sparkline and bar),
   Agent Intent donut (the 7-bucket breakdown), and Live Feed (last 20 bot
   hits with 10-second polling).

When a project is created but has no agent traffic yet, the dashboard shows a
"tracking active, no AI agents yet" notice (the install is verified and live)
alongside a "Run an AI Readiness Audit" call to action.

## AI Readiness Audit

Customers can run an audit against any URL. Fourteen scored checks (total 100
points) cover:

- **Bot access control** — robots.txt presence + explicit allowlist, meta robots,
  X-Robots-Tag headers
- **Discoverability** — sitemap.xml with lastmod, llms.txt / llms-full.txt,
  HTML rendering ratio (is your content actually in the initial HTML?)
- **Structured data** — JSON-LD Organization, WebSite, FAQPage, HowTo,
  Product schemas; canonical URL; meta description quality
- **Entity / E-E-A-T signals** — Wikipedia / Wikidata presence, Knowledge Graph
  entities linked via sameAs, authorship markup, content freshness
- **Platform readiness** — content quality heuristics tailored to major AI
  platforms

Each check returns a status (pass / partial / fail), a score, and an actionable
suggestion so the customer knows exactly what to change.

## How to install

Every path is documented in full, without a login, on its integrations page
(pitch + complete install guide on one URL): /integrations/cloudflare (zero
code, OAuth), /integrations/shopify, /integrations/nextjs,
/integrations/vercel (Log Drain), /integrations/aws, and
/integrations/http-api. The sections below summarise the two code paths.

### Next.js (14, 15, 16)

One middleware or proxy file sends a background intake call per page view. It
uses `event.waitUntil` so the request is non-blocking yet survives Edge/serverless
(Vercel, Cloudflare, Netlify), where a bare un-awaited fetch is killed the moment
the response is returned:

```ts
// middleware.ts (Next.js 15) or proxy.ts (Next.js 16)
// Next 16 note: rename the function to `proxy` and the file to `proxy.ts`.
import { NextRequest, NextResponse, NextFetchEvent } from 'next/server';

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const { pathname } = request.nextUrl;

  // Never track your own API or static assets: the matcher below excludes
  // most of these, this is a second-layer guard.
  if (pathname.startsWith('/api/')) return NextResponse.next();

  // Real client/bot IP. Prefer Cloudflare's CF-Connecting-IP (CF sets it to
  // the true client) so a CF-fronted site records the real bot IP, not the
  // Cloudflare edge IP; fall back to the leftmost X-Forwarded-For, then
  // X-Real-IP. Matches the CF Worker's precedence.
  const ip =
    request.headers.get('cf-connecting-ip') ||
    request.headers.get('x-forwarded-for')?.split(',')[0]?.trim() ||
    request.headers.get('x-real-ip') ||
    '';
  // intake requires a non-empty ip; if none resolved (rare), skip the send.
  if (!ip) return NextResponse.next();

  // Externally-visible URL. `request.url` carries the raw Host header, and
  // behind a reverse proxy — Vercel, Fly, Railway, Render, an ALB, nginx, any
  // container platform — that Host is the INTERNAL address, frequently
  // localhost. Arrivl stores that hostname on every event and only marks the
  // install verified once it matches your real domain, so reporting the
  // internal one means the site never verifies. When the proxy tells us the
  // public host (leftmost value: proxies append, they don't replace), swap it
  // into the origin and keep the rest of the URL untouched. No proxy in front
  // → request.url is already right and is used as-is.
  const fwdHost = request.headers.get('x-forwarded-host')?.split(',')[0]?.trim();
  const fwdProto = request.headers.get('x-forwarded-proto')?.split(',')[0]?.trim();
  const trackedUrl = fwdHost
    ? request.url.replace(/^https?:\/\/[^/]+/, `${fwdProto || 'https'}://${fwdHost}`)
    : request.url;

  const params = new URLSearchParams({
    url: trackedUrl,
    userAgent: request.headers.get('user-agent') || '',
    ref: request.headers.get('referer') || '',
    // The Accept header, raw. A real browser navigating to a page asks for
    // text/html; a script wearing a browser's User-Agent usually does not.
    // Empty when the request has none — Arrivl stores that as unknown.
    accept: request.headers.get('accept') || '',
    ip,
    websiteKey: process.env.ARRIVL_WEBSITE_KEY!,
    // Which snippet, which revision. Arrivl cannot update this file for you,
    // so these two are how we can tell you it has gone stale.
    source: 'nextjs',
    sv: '3',
    // WHY THERE IS NO status/bytes HERE, and why that is not an omission.
    //
    // Proxy (and middleware before it) runs BEFORE the route renders, and
    // `NextResponse.next()` means "continue" — it is a signal, not the
    // response your app went on to produce. So at this point in the request
    // there is no status code and no byte count in existence to report, and
    // inventing 200 would put a fabricated number on every soft-404 and every
    // 5xx an AI crawler hits.
    //
    // `shape=pre_response` says exactly that to Arrivl, so a blank status on
    // this install is reported as NOT APPLICABLE rather than as a missing
    // field. If you want the real status and size, report from somewhere that
    // has the response instead — a Node server's `res.on('finish')`, a
    // framework "after response" hook, or an access-log shipper — and declare
    // `shape=post_response` / `shape=log` there.
    shape: 'pre_response',
  });
  // Web Bot Auth signature headers — forward raw when present (rare today).
  for (const [h, p] of [['signature', 'sig'], ['signature-input', 'sigInput'], ['signature-agent', 'sigAgent']]) {
    const v = request.headers.get(h);
    if (v) params.set(p, v);
  }

  // event.waitUntil sends the request in the background WITHOUT blocking the
  // response. A bare un-awaited fetch is killed on Edge/serverless (Vercel,
  // Cloudflare, Netlify): the runtime freezes the function the instant the
  // response is returned, so the event never leaves. waitUntil keeps it alive,
  // on every platform (no @vercel/functions needed).
  event.waitUntil(
    fetch(`https://arrivl.ai/api/v1/intake/pageview?${params}`).catch(() => {})
  );

  return NextResponse.next();
}

// Track AI-discovery paths (robots.txt, llms.txt, sitemap.xml, .well-known)
// even though their extensions look static: these are the highest-signal
// hits Arrivl captures. The catch-all below then excludes other .txt/.xml/.json.
export const config = {
  matcher: [
    '/robots.txt',
    '/llms.txt',
    '/llms-full.txt',
    '/sitemap.xml',
    '/ai.txt',
    '/sitemap-:path*',
    '/.well-known/:path*',
    '/((?!api|_next/static|_next/image|favicon.ico|manifest.webmanifest|.*\\.(?:png|jpg|jpeg|svg|gif|webp|ico|css|js|mjs|map|woff2?|ttf|otf|eot|txt|xml|json)$).*)',
  ],
};
```

The matcher deliberately allowlists /robots.txt, /llms.txt, /llms-full.txt,
/sitemap.xml, /ai.txt, and /.well-known — AI crawlers fetching those paths are
some of the highest-signal hits Arrivl captures, so they're tracked even though
their extensions would otherwise be treated as static.

### Plain HTTP

Any server-side code can call the intake endpoint directly:

```
GET https://arrivl.ai/api/v1/intake/pageview
  ?url=https://yoursite.com/blog/post
  &userAgent=<URL-encoded User-Agent>
  &ref=<URL-encoded Referer>
  &accept=<URL-encoded Accept header, empty if none>
  &ip=<client IP>
  &websiteKey=ak_<your 40-char key>
```

Build the `url` param from the externally-visible host — read
`X-Forwarded-Host` and fall back to `Host`. Behind a reverse proxy (Vercel,
Fly, Railway, Render, an ALB, nginx, any container platform) the raw `Host`
header is an internal hostname, often `localhost`; Arrivl records that host on
every event and can never mark such an install verified.

Returns `{"ok": true}` on success with HTTP 202. The call should run in the
background so it never slows the page render. On serverless/Edge keep it alive
with that platform's background-task primitive (on Next.js, `event.waitUntil`) —
a bare un-awaited fetch is dropped when the function freezes after the response.

### Deploy, then verify

Writing the code is not enough: a project flips to connected only after Arrivl
receives its first real event. Set `ARRIVL_WEBSITE_KEY` in production, deploy, then
load a page on the live site (or send one test request) to fire that first event.

## Integrations

For sites that can't (or shouldn't) drop a snippet into their codebase, Arrivl
ships pre-built integrations that wire the intake endpoint up without touching
the site's source. Public integration pages live under `/integrations`:

- [/integrations](https://arrivl.ai/integrations) — directory of every platform
  we connect to, grouped by setup mechanism.
- [/integrations/shopify](https://arrivl.ai/integrations/shopify) — **available
  today**. DNS-based setup: the merchant adds a couple of records at their
  domain registrar (GoDaddy / Namecheap / Cloudflare / AWS / etc.). No Shopify
  app, no theme edits, no admin permissions. Setup takes about 5 minutes and
  works on every Shopify plan from Basic to Plus. Once active, every AI agent
  hitting any URL on the storefront is logged through the same intake pipeline
  as a JavaScript snippet would use.

For platforms not yet listed, the JavaScript or HTTP intake (above) covers
any stack with a backend — Next.js, Rails, Django, custom Express, etc.

## FAQ

### Why should I care about AI traffic at all?

AI agents are how more people discover products every day. When someone asks
ChatGPT or Claude for a recommendation, those tools visit websites to decide
what to say. If you can't see them, you have no idea how your site shows up
in the answers your future customers are getting.

### How does Arrivl detect AI agents?

By user agent strings, IP ranges, and request patterns. Major AI companies
publish official identifiers for their bots, and Arrivl maintains an updated
list as new ones launch. For agentic browsers like Claude for Chrome and
ChatGPT Atlas, behavioral signals separate them from humans.

### How accurate is the data?

For agents with official identifiers, close to 100%. For newer or less
transparent agents, Arrivl flags traffic as "unidentified AI" rather than
mislabel it — customers always know what's confident and what's still
being classified.

### How long does it take to set up?

A few minutes. One line of code on your site and data starts flowing
immediately.

### How much does Arrivl cost?

Free during beta. Sign up now for full access while the product is being
built with early users.

### Does Arrivl track my human visitors?

No. Arrivl tracks only AI agents. Existing tools (GA, Amplitude) handle
human traffic — the two work side by side.

## Contact

- Email: hello@arrivl.ai
- X: https://x.com/arrivlai