# AGENTS.md — Arrivl integration guide > Working for a person who is evaluating Arrivl (a free AI-readiness audit of > their site, booking a demo)? Read https://arrivl.ai/agents instead. This file > is for installing the tracking snippet. Arrivl is a growth analytics product for the agentic web. It tracks how AI crawlers, AI search indexers, agentic browsers, social-preview bots, and traditional search engines interact with a website — and links those visits to the human referrals that follow. If you're a coding agent helping a developer install Arrivl, this file describes what to do. ## Install One GET request per page view, sent in the background so it never slows the page render. Send it through the middleware/proxy `event.waitUntil` — on Edge/serverless (Vercel, Cloudflare, Netlify) the runtime freezes the function the instant the response is returned, so a bare un-awaited fetch is killed before it leaves and the event silently never arrives. ### Next.js (middleware.ts for 14/15, proxy.ts for 16) ```ts // middleware.ts (Next.js 15) or proxy.ts (Next.js 16) // Next 16 note: rename the function to `proxy` and the file to `proxy.ts`. import { NextRequest, NextResponse, NextFetchEvent } from 'next/server'; export function middleware(request: NextRequest, event: NextFetchEvent) { const { pathname } = request.nextUrl; // Never track your own API or static assets: the matcher below excludes // most of these, this is a second-layer guard. if (pathname.startsWith('/api/')) return NextResponse.next(); // Real client/bot IP. Prefer Cloudflare's CF-Connecting-IP (CF sets it to // the true client) so a CF-fronted site records the real bot IP, not the // Cloudflare edge IP; fall back to the leftmost X-Forwarded-For, then // X-Real-IP. Matches the CF Worker's precedence. const ip = request.headers.get('cf-connecting-ip') || request.headers.get('x-forwarded-for')?.split(',')[0]?.trim() || request.headers.get('x-real-ip') || ''; // intake requires a non-empty ip; if none resolved (rare), skip the send. if (!ip) return NextResponse.next(); // Externally-visible URL. `request.url` carries the raw Host header, and // behind a reverse proxy — Vercel, Fly, Railway, Render, an ALB, nginx, any // container platform — that Host is the INTERNAL address, frequently // localhost. Arrivl stores that hostname on every event and only marks the // install verified once it matches your real domain, so reporting the // internal one means the site never verifies. When the proxy tells us the // public host (leftmost value: proxies append, they don't replace), swap it // into the origin and keep the rest of the URL untouched. No proxy in front // → request.url is already right and is used as-is. const fwdHost = request.headers.get('x-forwarded-host')?.split(',')[0]?.trim(); const fwdProto = request.headers.get('x-forwarded-proto')?.split(',')[0]?.trim(); const trackedUrl = fwdHost ? request.url.replace(/^https?:\/\/[^/]+/, `${fwdProto || 'https'}://${fwdHost}`) : request.url; const params = new URLSearchParams({ url: trackedUrl, userAgent: request.headers.get('user-agent') || '', ref: request.headers.get('referer') || '', // The Accept header, raw. A real browser navigating to a page asks for // text/html; a script wearing a browser's User-Agent usually does not. // Empty when the request has none — Arrivl stores that as unknown. accept: request.headers.get('accept') || '', ip, websiteKey: process.env.ARRIVL_WEBSITE_KEY!, // Which snippet, which revision. Arrivl cannot update this file for you, // so these two are how we can tell you it has gone stale. source: 'nextjs', sv: '3', // WHY THERE IS NO status/bytes HERE, and why that is not an omission. // // Proxy (and middleware before it) runs BEFORE the route renders, and // `NextResponse.next()` means "continue" — it is a signal, not the // response your app went on to produce. So at this point in the request // there is no status code and no byte count in existence to report, and // inventing 200 would put a fabricated number on every soft-404 and every // 5xx an AI crawler hits. // // `shape=pre_response` says exactly that to Arrivl, so a blank status on // this install is reported as NOT APPLICABLE rather than as a missing // field. If you want the real status and size, report from somewhere that // has the response instead — a Node server's `res.on('finish')`, a // framework "after response" hook, or an access-log shipper — and declare // `shape=post_response` / `shape=log` there. shape: 'pre_response', }); // Web Bot Auth signature headers — forward raw when present (rare today). for (const [h, p] of [['signature', 'sig'], ['signature-input', 'sigInput'], ['signature-agent', 'sigAgent']]) { const v = request.headers.get(h); if (v) params.set(p, v); } // event.waitUntil sends the request in the background WITHOUT blocking the // response. A bare un-awaited fetch is killed on Edge/serverless (Vercel, // Cloudflare, Netlify): the runtime freezes the function the instant the // response is returned, so the event never leaves. waitUntil keeps it alive, // on every platform (no @vercel/functions needed). event.waitUntil( fetch(`https://arrivl.ai/api/v1/intake/pageview?${params}`).catch(() => {}) ); return NextResponse.next(); } // Track AI-discovery paths (robots.txt, llms.txt, sitemap.xml, .well-known) // even though their extensions look static: these are the highest-signal // hits Arrivl captures. The catch-all below then excludes other .txt/.xml/.json. export const config = { matcher: [ '/robots.txt', '/llms.txt', '/llms-full.txt', '/sitemap.xml', '/ai.txt', '/sitemap-:path*', '/.well-known/:path*', '/((?!api|_next/static|_next/image|favicon.ico|manifest.webmanifest|.*\\.(?:png|jpg|jpeg|svg|gif|webp|ico|css|js|mjs|map|woff2?|ttf|otf|eot|txt|xml|json)$).*)', ], }; ``` For Next.js 16 the file is `proxy.ts` and the exported function is named `proxy` (not `middleware`). Everything else, including the `(request, event)` signature, is identical. **Important:** do NOT exclude `/robots.txt`, `/llms.txt`, `/llms-full.txt`, `/sitemap.xml`, `/ai.txt`, or `/.well-known/*` — AI crawlers fetch those paths to discover a site, and those hits are the highest-signal data Arrivl captures. They're explicitly allowlisted in the matcher above even though their extensions look static. ### Any other stack (plain HTTP) ``` GET https://arrivl.ai/api/v1/intake/pageview ?url= &userAgent= &ref= &accept= &ip= &websiteKey= ``` **Build `url` from the externally-visible host, not the raw `Host` header.** Behind a reverse proxy — Vercel, Fly, Railway, Render, an ALB, nginx, k8s ingress, any container platform — `Host` (and everything derived from it: Next's `request.url`, Express's `req.get('host')`, a Lambda event's `headers.host`) carries the INTERNAL hostname, often `localhost`. Arrivl records that hostname on every event and only marks the install verified when it matches the site's real domain, so an internal host means the site can never verify. Resolve it as `X-Forwarded-Host` → `Host`, taking the leftmost value when the header is a comma-separated proxy chain, and derive the scheme the same way from `X-Forwarded-Proto`. Returns `{"ok": true}` with HTTP 202. Call it server-side in the background. On serverless/Edge, keep the request alive with that platform's background-task primitive (on Next.js, `event.waitUntil`) — a bare un-awaited fetch is dropped when the function freezes after the response. `.catch(() => {})` swallows transient errors so tracking never breaks a page render. ## Environment - `ARRIVL_WEBSITE_KEY` — 40-char hex API key prefixed with `ak_`. Create one at https://arrivl.ai/dashboard/settings after signup. Store in `.env.local` for dev and in production secrets (Vercel env vars, AWS Secrets Manager, etc.). Never commit it. ## Deploy and verify Writing the code is not enough. Arrivl marks a site connected only after it receives its first real event. So set `ARRIVL_WEBSITE_KEY` in production, DEPLOY, then load a page on the live site (or run the test curl below). That first event is what flips the project to connected in the dashboard. ## What you'll see in the dashboard - **Referred Humans** — visitors whose `Referer` header matches chatgpt.com, claude.ai, perplexity.ai, or google.com/search - **Agent Visits** — total bot pageviews in the selected time window - **AI Traffic Trend** — stacked daily bar chart, top 5 platforms + Other - **Human Referrals by AI** — bot → human conversion funnel per AI platform - **Top Content Pages** — most-crawled paths with sparkline - **Agent Intent** — donut + legend across 7 buckets (research, indexing, training, agentic, preview, search, unknown) - **Live Feed** — last 20 bot hits with 10-second polling ## Bot coverage Classification runs at ingestion time against a vendored registry covering 138 AI bots + 9 browser/runtime control entries. Major platforms included: OpenAI (GPTBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User), Google (Googlebot, Google-Extended, GoogleOther), Perplexity, xAI (Grok), Meta, Apple, Microsoft (Bingbot, Copilot), ByteDance (Bytespider), DeepSeek, Mistral, Cohere, AI2, HuggingFace, Common Crawl, Bright Data, Apify, Firecrawl, Exa, Tavily, Browserbase, Cognition, Perplexity, You.com, Phind, Kagi, DuckDuckGo, Brave, Mozilla, Slack, Discord, LinkedIn, and more. ## Testing your install Hit the intake endpoint with a test bot UA to verify the key works and the request reaches Arrivl: ``` curl "https://arrivl.ai/api/v1/intake/pageview\ ?url=https://yoursite.com/test\ &userAgent=Mozilla/5.0%20(compatible;%20GPTBot/1.0)\ &ref=\ &accept=text/html\ &ip=1.2.3.4\ &websiteKey=ak_YOUR_KEY" ``` Expected response: `{"ok": true}` within ~200ms. The event will appear in the dashboard Live Feed within 10 seconds. ## Support - hello@arrivl.ai - https://x.com/arrivlai