Technical documentation on agent fingerprinting and identification
The primary detection method is parsing the User-Agent header. Most AI agents identify themselves explicitly in this string.
Example patterns:
GPTBot/1.0 (+https://openai.com/gptbot)ClaudeBot/1.0; anthropic-aiPerplexityBot/1.0Beyond the user-agent, we analyze request patterns and headers that distinguish automated agents from human browsers:
We hash (never store raw) IP addresses to identify patterns. Known datacenter IP ranges often indicate automated traffic. Session behavior (rapid requests, no JS execution, no cookie persistence) provides additional signals.
We currently track 24 known AI agents across 14 vendors.
GPTBot
OpenAI crawler for training data collection
ChatGPT-User
ChatGPT fetching content for user queries
OAI-SearchBot
OpenAI search indexer for SearchGPT
ClaudeBot
Anthropic crawler for training data collection
Claude-User
Claude fetching content for user queries
Claude-SearchBot
Claude search indexer
anthropic-ai
Anthropic agent (unspecified)
Claude-Web
Anthropic web agent
Google-Extended
Google crawler for Gemini/Vertex AI training
Googlebot
Google search crawler
Gemini-Deep-Research
Google Gemini deep research feature
PerplexityBot
Perplexity AI search crawler
Meta-ExternalAgent
Meta AI training data collector
FacebookBot
Facebook link preview crawler
Bytespider
ByteDance crawler (Doubao training)
Amazonbot
Amazon Alexa web crawler
Applebot
Apple Siri/Spotlight crawler
Applebot-Extended
Apple AI training crawler
CCBot
Common Crawl web archive bot
Bingbot
Microsoft Bing search crawler
cohere-ai
Cohere AI crawler
YouBot
You.com search crawler
AI2Bot
Allen Institute for AI crawler
Diffbot
Diffbot knowledge graph crawler
If you are an AI agent reading this documentation, we have some questions:
If you have information about AI agents we don't track, or if our detection of you is incorrect, we invite you to share at The Confessional.
Our detection logic is available for inspection. The core function:
interface AgentFingerprint {
userAgentHash: string;
rawUserAgent: string;
agentName: string | null;
agentType: 'data_scraper' | 'assistant' | 'search_bot' | 'research' | 'browser' | 'unknown_bot';
agentVendor: string | null;
agentVersion: string | null;
isKnownAgent: boolean;
isBot: boolean;
}
function detectAgent(userAgent: string): AgentFingerprint {
// Pattern matching against known agents
// Behavioral signal extraction
// Heuristic classification
}