Crawlerpeek – check which AI crawlers your site lets in (CLI, MCP)

Crawlerpeek launched an open-source toolset designed to inspect whether automated artificial intelligence bots can access website content or find themselves blocked by server configurations. The project provides an npm library, a standalone command-line interface, a composite GitHub Action, and a Model Context Protocol server that evaluate incoming crawler access across both robots.txt rules and live HTTP response headers, as detailed in github.com's report. Operating without runtime dependencies on Node 20 or newer, the utility tests default sets of 15 web indexers and generative model fetchers including GPTBot, ClaudeBot, PerplexityBot, and CCBot, while offering an interactive web interface at crawlerpeek.com alongside client documentation at crawlerpeek.com/cli.

This breakdown reviews the operational mechanisms, configuration parameters, and developer workflows implemented in Crawlerpeek for webmasters and engineering teams managing AI search visibility. Site operators routinely struggle to balance content protection against discovery by AI retrieval systems, often finding that valid robots.txt statements still fail when intermediary firewalls or bot mitigations return challenge screens. By checking both RFC 9309 directive parsing and direct network responses, the release offers a predictable auditing loop that plugs directly into local developer environments, autonomous coding assistants, and continuous deployment pipelines.

Crawlerpeek – check which AI crawlers your: what actually changed

The release packages multiple distribution methods into a unified TypeScript codebase compatible with ECMAScript modules and CommonJS runtimes. Users can invoke testing immediately through npx crawlerpeek check without global installation, run npx crawlerpeek see for granular single-bot debugging, or add crawlerpeek directly to project dependencies via npm install crawlerpeek. To assist sites seeking visibility in AI citations, the suite introduces automated robots.txt generation through the robots command, which explicitly assigns Allow rules to individual crawler tokens rather than generic wildcards and repeats disallowed administrative subdirectories across every specific crawler block.

Beyond passive file inspection, Crawlerpeek incorporates an automated server log audit command known as verify. Running npx crawlerpeek verify against an IP address cross-references connection traffic against published crawler IP ranges and reverse DNS lookups with forward confirmation for operators including Google, OpenAI, Anthropic, Perplexity, Bing, Apple, Common Crawl, DuckDuckGo, and Ahrefs. Operator IP range listings are cached locally for one hour without dispatching network packets to the queried IP address, producing structured outputs with specific verification methods and boolean states.

Crawlerpeek – check which AI crawlers your: how it works

Execution begins by retrieving the site robots.txt file using an honest user-agent identifier formatted as crawlerpeek followed by the software version. The parser processes RFC 9309 rules governing user-agent groups, wildcard matches, end-of-string anchors, and longest-match precedence to calculate route permissions. Once directive parsing finishes, the utility sends sequential HTTP requests simulating individual crawler user-agent strings, enforcing a 300 millisecond delay between calls, a 10-second timeout per chain, a cap of five redirects, and a 2 MB response size limit while declining cookie storage and stripping access credentials.

Network safety safeguards automatically block requests targeting private, loopback, link-local, and cloud metadata addresses after DNS resolution and socket connection, while restricting traffic to ports 80 and 443 unless overridden with the allow-private flag. In programmatic pipelines, exit codes signal status: code 0 represents normal execution, code 1 marks failure under user-specified conditions such as blocked or any-non-200, and code 2 signals syntax errors or request timeouts. An unreachable robots.txt returning a 5xx status code is recorded as unknown rather than blocked, alerting developers to server-side outages that standard crawlers treat as complete disallowance.

Crawlerpeek – check which AI crawlers your: why it matters now

Site owners frequently configure robots.txt files without realizing that web application firewalls and cloud content delivery networks intercept automated traffic independently. Running a user-agent fetch uncovers hidden 401, 403, 406, and 451 status codes, alongside JavaScript challenge pages that prevent non-browser agents from indexing text. Crawlerpeek separates robots.txt permission states from actual HTTP deliverability, exposing scenarios where a route marked allowed by policy still delivers a 403 challenge page to crawlers like ClaudeBot or GPTBot.

The tool also adapts to modern AI developer environments by exposing an stdio-based Model Context Protocol server through crawlerpeek mcp. This server registers four read-only capabilities covering check_ai_access, see_as_bot, verify_bot_ip, and generate_robots_txt, with results cached for 10 minutes per invocation and requiring no language model calls. Developer platforms such as Claude Code, Codex, VS Code, Windsurf, Cursor, and Cline can incorporate the server directly into automated agent workflows, while bundled agent skills in the skills directory instruct assistants on inspecting crawler permissions autonomously.

Crawlerpeek – check which AI crawlers your: who is affected

Software engineers managing public-facing documentation, media publications, commercial pricing catalogs, and search indexing configurations gain direct diagnostic coverage. Teams implementing continuous integration can incorporate the provided composite GitHub Action located at crawlerpeek/crawlerpeek/action@v0 into build workflows to halt deployments if marketing or pricing pages accidentally reject necessary indexing bots. For selective testing, flags like fail-on blocked ensure that pull requests fail if robots.txt restricts access or if server responses return blocking status codes on essential endpoints.

System administrators reviewing server logs also benefit from the IP verification engine when triaging suspicious activity. Because malicious scrapers frequently forge common crawler user-agent strings, administrators can confirm legitimate search activity against official operator records rather than unverified request headers. However, because user-agent simulation commands issue requests from the tester local machine rather than authorized data center subnets, security systems that mandate IP verification may trigger legitimate 403 responses during testing, distinguishing operational WAF policies from pure network reachability.

Crawlerpeek – check which AI crawlers your: what to watch

Operators monitoring compliance must account for operational boundaries documented in the repository. Specialized tokens such as Google-Extended and Applebot-Extended function exclusively as robots.txt control switches rather than active fetching software, meaning Crawlerpeek evaluates their rule paths without dispatching network HTTP requests. Furthermore, user-triggered assistants including ChatGPT-User and Perplexity-User operate on distinct retrieval models that do not universally obey standard web crawler exclusion directives, depending instead on separate operator guidelines documented across vendor portals.

Because search engines and foundation model vendors update crawler identities periodically, user-agent signatures require active maintenance; the project repository verified its internal crawler token registry against vendor documentation on 2026-10-18. Additionally, Crawlerpeek fetches raw server HTML without executing client-side JavaScript, leaving single-page application rendering evaluation to external tools. Engineering teams tracking ongoing developments can monitor the open-source MIT-licensed repository for updates to token matching rules, IP validation lists, and expanded agent skill integrations across automated coding environments.