AI traffic and crawlers

People who arrive from AI assistants, and the AI bots that read your pages.

Visits from AI assistants

Visits referred by an assistant get the AI channel, so you can see their visitors, revenue and conversion next to search and social. Nothing to set up.

Links tagged utm_source=chatgpt, perplexity or claude count as AI too.

AI crawlers

Crawlers like GPTBot and ClaudeBot don't run JavaScript, so the browser script never sees them. Report them from your server. Only the bot's user agent and the path are sent, and crawler hits are never billed as pageviews.

import { NextResponse, type NextRequest } from "next/server";

const AI_BOTS = /ChatGPT-User|OAI-SearchBot|GPTBot|Claude-User|Claude-SearchBot|ClaudeBot|Perplexity|Google-Extended|Applebot-Extended|meta-externalagent|Bytespider|CCBot|Amazonbot|DuckAssistBot|MistralAI-User/;

export function middleware(req: NextRequest) {
  const ua = req.headers.get("user-agent") ?? "";
  if (AI_BOTS.test(ua)) {
    fetch("https://app.atrby.site/api/ingest", {
      method: "POST",
      body: JSON.stringify({ batch: [{ type: "crawl", siteId: "st_your_site_key", ua, path: req.nextUrl.pathname }] }),
    }).catch(() => {});
  }
  return NextResponse.next();
}

Any server can do the same: post { batch: [{ type: "crawl", siteId, ua, path }] } to /api/ingest when a request's user agent matches.

AI readiness

AI traffic → Readiness checks your site from our server: it fetches /robots.txt, /llms.txt and the homepage on your domain only (5–8 second timeouts, size caps, at most 3 redirects on the same site) and keeps the result for an hour. It reports:

• What robots.txt lets each AI crawler do on the homepage: allowed, blocked, or allowed with some paths blocked, and whether the bot is named or follows the * rules.
• Whether llms.txt exists, its size, a structure score and whether up to 10 of its links on your domain answer.
• Whether the homepage has a title and meta description, and how long it took to answer once from our server.

We can't change files on your server. The Allow and Block switches build a recommended robots.txt that keeps your other rules; copy it and upload it yourself.

What the intents mean