AI crawlers (26)

Collects pages for AI training or AI search. Open any one to see exactly how Verisoul detects it.

AmazonbotAmazon's crawler for Alexa and Amazon's AI services.
ClaudeBotAnthropic's crawler that collects public pages for model training.
ExaSearchBotExa's crawler, which builds the web index behind its AI search API.
GPTBotOpenAI's crawler that collects public pages for model training.
Meta-ExternalAgentMeta's crawler that collects public pages for AI training.
BytespiderByteDance's crawler for its search and AI products.
Claude-SearchBotAnthropic's crawler that indexes pages for Claude's web search.
OAI-SearchBotOpenAI's crawler that indexes pages for ChatGPT search.
PerplexityBotPerplexity's crawler that indexes pages for its AI answers.
PetalBotHuawei's crawler for Petal Search and Huawei's AI services.
CCBotCommon Crawl's crawler for its open web dataset, which is widely used to train AI models.
Meta-WebIndexerMeta's crawler that indexes pages for Meta AI search answers.
ShapBotParallel's crawler, which builds the web index behind its AI search and research API.
AI SearchAn AI crawler run by Cloudflare that signs its requests.
AI Search ExternalAn AI crawler run by Cloudflare that signs its requests.
Cloudflare CrawlerAn AI crawler run by Cloudflare that signs its requests.
Daric2An AI crawler run by Cloudflare that signs its requests.
Daric3An AI crawler run by Cloudflare that signs its requests.
Daric4An AI crawler run by Cloudflare that signs its requests.
Kernel SearchAn AI crawler run by Kernel Technologies, Inc. that signs its requests.
KlaviyoAIBotAn AI crawler run by Klaviyo that signs its requests.
PayPal Presentment CrawlerAn AI crawler run by PayPal that signs its requests.
payroll-botAn AI crawler run by ADP that signs its requests.
GemmetricBotAn AI crawler run by Gemmetric LLC that signs its requests.
Novellum AI CrawlAn AI crawler run by Novellum that signs its requests.
YGS Group Falconer ScraperAn AI crawler that signs its requests.