Measure agent-traffic for any site news update

Mintlify, which hosts documentation for thousands of companies and serves hundreds of millions of requests a month, has open-sourced agent-traffic — a classifier and curated dataset that tells AI coding agents, AI crawlers, search bots, and human browsers apart at the HTTP layer. The library ships as an npm package with zero dependencies and runs on Node, Bun, Deno, Cloudflare Workers, and Vercel Edge.

This article covers what agent-traffic does, how its classification pipeline works, how to integrate it into a Cloudflare Worker or query existing logs, what the data model looks like, and where the library's current limits lie. It is aimed at developers who operate documentation sites, APIs, or any web service that needs to know whether an incoming request is a person, a coding agent like Claude Code, an AI crawler collecting training data, or an impersonator spoofing a known bot.

01Growing agent share on documentation infrastructure

For years, Mintlify's documentation sites saw traffic that was almost entirely developers reading pages in a browser. That picture has changed: Claude Code now pulls pages into coding sessions, ChatGPT answers questions using hosted docs, and Cursor checks API references before generating code. As that agent share grew, Mintlify found existing open-source bot lists insufficient — they matched user-agent substrings, lumped all automated traffic into one bucket, and took every request at face value.

Mintlify studied its own traffic, tested existing lists against it, and tracked which signals reliably separated categories. The result, agent-traffic, is an MIT-licensed package that combines a tested dataset of user agents, headers, and official verification sources with a small classifier that applies them.

02IP verification catches GPTBot impersonators

The classifier runs checks in a fixed order and stops at the first match: known agent or crawler (verified by IP when possible), prefers markdown, generic HTTP client or headless browser, browser (unless headers contradict it), otherwise a bot. The IP-verification step downloads published IP-range files from OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, Common Crawl, and DuckDuckGo via loadIpRanges().

A request claiming to be GPTBot from IP 73.12.4.9 — outside OpenAI's published ranges — returns category: 'bot', kind: 'impersonator', verified: false, and a reasons array stating exactly why. The verified field can be true (IP is in the operator's published ranges), false (provably not who it claims), or null (not checked or impossible to check — coding agents like Claude Code run on the user's own machine and are never IP-verified).

03Markdown Accept header surfaces disguised agents

Agents that borrow a standard browser user agent still betray themselves through the Accept header. When a request arrives with a Chrome user-agent string but Accept: text/markdown, text/html;q=0.9, the classifier returns category: 'agent', kind: 'unidentified', confidence: 'medium', with a reason noting that browsers never prefer markdown over HTML. The helper prefersMarkdown(accept) exposes this check as a standalone boolean for simpler pipelines.

Enabling the headersComplete option activates fake-browser detection, which is safe when browser headers reach the server untouched over HTTPS. Developers should leave it off if plain HTTP or a proxy in front can strip headers. The maxRangeAgeDays option (default 180) governs how stale an operator's IP-range file can be before it stops being used to flag impersonators.

04Five categories, twelve kinds, and a Cloudflare Worker pattern

The library defines five top-level categories: agent (an AI product fetching a page right now for a person, with kinds assistant, coding, browser, and unidentified), ai-crawler (an AI company collecting pages ahead of time, with kinds training, ai-search, and ingest), crawler (a search engine), bot (any other automation, covering seo, preview, ads, monitor, scanner, archiver, feed, http-client, headless, fake-browser, impersonator, and unidentified), and human. A sixth result, unknown, means nothing could be classified.

In a Cloudflare Worker, loadIpRanges() is called once and its promise cached so concurrent requests do not each trigger a download. The Worker can then branch on result.category === 'agent' to serve a markdown response instead of HTML. The underlying dataset — agents.json, http-clients.json, and headers.json — is importable as plain JSON from @mintlify/agent-traffic/data/*.json for use from other languages or SQL queries against ClickHouse, DuckDB, or Postgres, all of which are listed as verified targets.

05Identity sharing and invisible agentic browsers

The library documents two classes of limitation. Some products share an HTTP identity: Codex has been reported to fetch as ChatGPT-User, and Cursor through Exa as ExaUser, so results name only what can be proved from the request itself. Logs that store only the user agent can still identify known agents, crawlers, and HTTP clients, but agents borrowing a browser user agent will appear human unless the Accept header is also logged.

A second category of agents is invisible at the HTTP layer entirely. Agentic browsers that run inside a person's own Chrome — Atlas, Comet, and Claude in Chrome are named explicitly — look exactly like that person's normal browser traffic. The project accepts pull requests against data/agents.json with real example user agents and a source, following CONTRIBUTING.md, to keep the dataset current as agents change their fetchers.