Operator IP lists
JSON files, text lists and documentation pages where an operator lists the addresses its crawler or scanner uses. Re-fetched every 10–60 minutes; if a list can't be refreshed, its entries expire rather than being trusted forever.
Bot signals — Coverage
Who we recognise, and how we know.
Three flags in every Pro response — risk.is_ai, risk.is_crawler, risk.is_scanner. Each one traces back to something the operator publishes about itself. This is the full list, generated from the catalogues the API runs on.
01 — Evidence
Three ways an address earns a flag.
No third-party blocklists and no behavioural guesses. If an operator publishes nothing we can verify, it isn't on this page.
Forward-confirmed reverse DNS
For operators that document a hostname convention instead of ranges: the address's PTR name must end in the documented suffix and resolve back to the same address.
Scanner feeds & networks
Scanning projects' own IP feeds, published ranges, scanner hostnames and registry assignments for networks dedicated to scanning. New addresses can be verified in the background after a first lookup.
What false means
false means no current match. It does not prove the traffic is human, or that the address is safe.
02 — Coverage
Every operator, by flag.
143 sources in total; source definitions last reviewed .
is_ai AI crawlers & agents 28
Training crawlers, AI search and retrieval fetchers, and agent egress from operators that publish their addresses.
| Operator | What is covered | Evidence | Refresh |
|---|---|---|---|
| Amazon Amzn-SearchBot ↗ | Amazon AI/search product crawling egress. | Published ranges | hourly |
| Amazon Amzn-User ↗ | Amazon user-triggered AI retrieval egress. | Published ranges | hourly |
| Amazonbot ↗ | Amazonbot published crawling infrastructure, documented as supporting AI training among other uses; not proof every request trains AI. | Published ranges | hourly |
| Anthropic API tools ↗ | Only active outbound MCP/web tool ranges; excludes inbound-only and phased-out entries. | Published ranges | hourly |
| Anthropic Claude bots ↗ | Shared ClaudeBot, Claude-SearchBot and Claude-User published egress. | Published ranges | hourly |
| Anthropic Claude-Agent ↗ | Exclusive Claude-Agent browser egress only; receipt-relative SAIL expiry is enforced. | Published ranges | every 15 min |
| Apple Applebot ↗ | Shared Apple search crawler infrastructure documented as supporting AI training and grounding; not proof of individual request purpose. Applebot-Extended is a policy token, not a separate crawler. | Published ranges | hourly |
| DuckDuckGo DuckAssistBot ↗ | DuckAssist AI-answer retrieval infrastructure; shared with DuckDuckBot, not proof of individual request purpose. | Published ranges | hourly |
| Google-Agent ↗ | Google-hosted user-directed agents only; excludes common crawlers and generic/customer-controlled fetcher feeds. | Published ranges | hourly |
| KarlaCrawler ↗ | Client-authorized website ingestion for Karla AI retrieval; shared with known_crawler. | Published ranges | hourly |
| Microsoft Bingbot ↗ | Shared Bing search crawler infrastructure whose indexed content supports AI answers and model training; not a dedicated Copilot fleet or proof of individual request purpose. | Published ranges | hourly |
| MistralAI-Index ↗ | Mistral AI search indexing; no claim about the unlisted training fleet. | Published ranges | hourly |
| MistralAI-User ↗ | Mistral user-triggered retrieval egress. | Published ranges | hourly |
| Moonshot Kimi-SearchBot ↗ | Kimi AI search indexing egress; published addresses can overlap other Kimi bots. | Published ranges | hourly |
| Moonshot Kimi-User ↗ | Kimi user-triggered retrieval egress; published addresses can overlap other Kimi bots. | Published ranges | hourly |
| Moonshot KimiBot ↗ | Kimi foundation-model training crawler egress; published addresses can overlap other Kimi bots. | Published ranges | hourly |
| OpenAI ChatGPT integrations ↗ | ChatGPT plugins, connectors, GPT Actions, agentic commerce and Private Safety Processing egress; not an authentication guarantee. | Published ranges | hourly |
| OpenAI ChatGPT-User ↗ | ChatGPT user-triggered retrieval egress. | Published ranges | hourly |
| OpenAI Codex cloud ↗ | Codex cloud outbound service connections, not arbitrary Azure address space. | Published ranges | hourly |
| OpenAI GPTBot ↗ | Documented GPTBot training crawler egress. | Published ranges | hourly |
| OpenAI OAI-SearchBot ↗ | Documented AI search crawler egress. | Published ranges | hourly |
| Parallel ShapBot ↗ | Parallel AI web search and extraction index crawler egress; does not claim coverage of the separately signed Shap-User fetcher. | Published ranges | hourly |
| Perplexity-User ↗ | Perplexity user-triggered retrieval egress. | Published ranges | hourly |
| PerplexityBot ↗ | Perplexity AI search crawler egress. | Published ranges | hourly |
| ProRata Gist crawler ↗ | Published production and testing crawler egress for Gist RAG retrieval. | Published ranges | hourly |
| Qodo hosted code review ↗ | Multi-tenant Qodo outbound connections to Git environments only. Does not cover customer-specific or self-hosted deployments; not a web crawler. | Published ranges | hourly |
| SEObot ↗ | Published crawler egress for SEObot AI SEO agents; shared with known_crawler. | Published ranges | hourly |
| You.com YouBot ↗ | You.com search/retrieval crawler explicitly designated IP range only. | Published ranges | hourly |
is_crawler Search, SEO & archive crawlers 49
Search engines, SEO and site-audit tools, and archives. 7 are verified by forward-confirmed reverse DNS; the rest by published ranges.
| Operator | What is covered | Evidence | Refresh |
|---|---|---|---|
| 360 Search ↗ | Six explicit HTML wildcard /24 ranges only. Linked PDF is not parsed; this is partial published coverage. | Published ranges | hourly |
| AddSearchBot ↗ | Published website indexing crawler addresses. | Published ranges | hourly |
| Ahrefs crawlers ↗ | Ahrefs crawler infrastructure | Published ranges | hourly |
| Arquivo.pt Web Crawler ↗ | Arquivo.pt published web archiving crawler ranges. | Published ranges | hourly |
| Audisto crawlers ↗ | Operator-published exact IPv4 and IPv6 crawler addresses. | Published ranges | hourly |
| Babbar Barkrowler ↗ | Babbar publishes the reverse DNS suffix babbar.eu for its SEO link-graph crawler. The forward lookup is applied even though the page does not state it. | Verified rDNS | on lookup |
| Baiduspider ↗ | Only the crawler-specific IPv4-encoded hostname convention demonstrated in Baidu's official forward/reverse DNS example. Does not classify all baidu.com or Baidu ERNIE. | Verified rDNS | on lookup |
| Botify ↗ | Shared SiteCrawler, AlertPanel and SpeedWorkers infrastructure; does not identify product or request purpose. | Published ranges | hourly |
| ByteDance Bytespider ↗ | Toutiao Search crawler subnets from the public About Bytespider document. Does not establish Doubao or TikTok AI coverage. | Published ranges | hourly |
| Ceramic TerraCotta ↗ | Operator-published TerraCotta crawler addresses; no surrounding cloud ranges. | Published ranges | hourly |
| Coc Coc coccocbot ↗ | Coc Coc documents reverse DNS ending in .coccoc.com followed by a forward lookup that must return the original address. | Verified rDNS | on lookup |
| Common Crawl CCBot ↗ | CCBot | Published ranges | hourly |
| Conductor Caliperbot ↗ | Caliperbot address range exactly as published (first to last host); the separate Conductor Intelligence page crawler is not included. | Published ranges | hourly |
| Conductor Monitoring ↗ | Website monitoring crawler subnets (formerly ContentKing) from the 'What IP addresses does Conductor use when monitoring websites?' answer only. | Published ranges | hourly |
| Coveo cloud crawlers ↗ | Inbound indexing addresses for US, Canada, Europe, Australia and HIPAA only; excludes outbound AWS/CloudFront ranges. | Published ranges | hourly |
| DataForSEO Bot ↗ | Backlink crawler subnets. Normalize IPv6 host bits only because the operator explicitly publishes subnet identifiers. | Published ranges | hourly |
| DuckDuckBot ↗ | DuckDuckBot and DuckAssistBot | Published ranges | hourly |
| Google common crawlers ↗ | Google-operated common crawlers; multiple crawler identities share these ranges. | Published ranges | hourly |
| Google special crawlers ↗ | Google-operated special-case crawlers, including advertising validation. | Published ranges | hourly |
| Google-operated user-triggered fetchers ↗ | Google-operated user-triggered fetchers only. Excludes the separate customer-controlled Google Cloud fetcher feed. | Published ranges | hourly |
| Huawei PetalBot ↗ | Crawler-specific IPv4-encoded hostname, confirmed forward. The current official site's JavaScript overrides legacy aspiegel.com prose with petalsearch.com. Search crawler only, not dedicated generative AI attribution. | Verified rDNS | on lookup |
| IbouBot ↗ | IbouBot | Published ranges | hourly |
| InfluenceKit fetcher IPv4 ↗ | Customer-triggered link-preview fetcher origins. Exact published IPv4 hosts only. | Published ranges | hourly |
| InfluenceKit fetcher IPv6 ↗ | Customer-triggered link-preview fetcher origins. Operator explicitly documents empty HTTP 200 as no IPv6 egress. | Published ranges | hourly |
| JetOctopus ↗ | Published Crawling Servers only; browser version strings and user agents do not identify IPs. | Published ranges | hourly |
| Kagibot ↗ | Kagi search index crawler addresses. | Published ranges | hourly |
| Linkup AI ↗ | Linkup AI search-index crawler. Unrelated to linkup.com jobs. | Published ranges | hourly |
| Lumar (Deepcrawl) ↗ | The two crawl IPs Lumar asks site owners to allowlist. Lumar defaults to a Googlebot user agent, so the address is the only first-party identifier. | Published ranges | hourly |
| MojeekBot ↗ | MojeekBot | Published ranges | hourly |
| Naver search crawlers ↗ | Published Naver search crawler ranges / Yeti | Published ranges | hourly |
| Oncrawl ↗ | Published EU and US website crawler addresses. | Published ranges | hourly |
| OpenAI OAI-AdsBot ↗ | OAI-AdsBot advertising landing-page validation crawler. | Published ranges | hourly |
| Pinterestbot ↗ | Published US interval only; no inference about other Pinterest or non-US addresses. | Published ranges | hourly |
| Qwantbot ↗ | Qwantbot | Published ranges | hourly |
| SE Ranking Backlinks IPv4 ↗ | SERankingBacklinksBot published IPv4 egress. | Published ranges | hourly |
| SE Ranking Backlinks IPv6 ↗ | SERankingBacklinksBot published IPv6 egress. | Published ranges | hourly |
| SeekportBot ↗ | Operator-published exact crawler origin addresses. | Published ranges | hourly |
| Semrush SiteAuditBot ↗ | SiteAuditBot only; not the full SemrushBot fleet. | Published ranges | hourly |
| Seobility crawlers ↗ | Seobility crawler infrastructure | Published ranges | hourly |
| SerpstatBot ↗ | SerpstatBot | Published ranges | hourly |
| Seznambot ↗ | Seznam crawler infrastructure | Published ranges | hourly |
| SirdataBot ↗ | Contextual page categorization crawler origins. Operator recommends refreshing every ten minutes. | Published ranges | every 10 min |
| Sitebulb Cloud ↗ | Published UK and US shared Cloud pools only; excludes desktop and dedicated customer addresses. | Published ranges | hourly |
| Sitechecker ↗ | Site Audit, Site Monitoring and On Page Checker crawler egress. | Published ranges | hourly |
| Siteimprove Content suite ↗ | Content suite crawling and website checking only; excludes sign-in, analytics, email and platform service IPs. | Published ranges | hourly |
| Sogou spider ↗ | Sogou states spider hostnames are *.sogou.com and anything else is an impostor. Sogou does not spell out the forward lookup; it is applied anyway. Search crawler only, not generative AI attribution. | Verified rDNS | on lookup |
| Yahoo Slurp ↗ | Yahoo Slurp crawler infrastructure | Published ranges | hourly |
| Yahoo Slurp (DNS) ↗ | Yahoo documents hostnames ending in .crawl.yahoo.net that resolve back to the same IP. Complements the published Yahoo range list. | Verified rDNS | on lookup |
| YandexBot ↗ | Yandex documents reverse DNS ending in yandex.ru, yandex.net or yandex.com plus a matching forward lookup as the way to verify all Yandex robots. Any forward-confirmed host under these suffixes is treated as Yandex robot infrastructure, as Yandex instructs; it is not proof of an individual request's purpose. | Verified rDNS | on lookup |
is_scanner Internet scanners 66
Research, measurement and attack-surface scanning projects. 52 publish a feed, 51 are matched by published or registered ranges, 30 by forward-confirmed reverse DNS, 3 by scanner hostnames they publish (some by more than one). Badges: Feed operator feed, Ranges published or registered ranges, rDNS verified reverse DNS, Hosts published scanner hostnames.
- Acunetix / Invicti Online Hosts
- Aikido DAST and Pentest FeedRanges
- Alpha Strike Labs FeedRanges
- Alpha Strike Labs observed census network FeedRanges
- Apcosys rDNSHosts
- BinaryEdge / Coalition FeedRangesrDNS
- BufferOver rDNS
- Censys FeedRangesrDNS
- CERT Polska Artemis FeedRangesrDNS
- CIRCL Scanning FeedRanges
- Criminal IP / AI Spera FeedRanges
- CyberOK / SKIPA FeedRanges
- CyberResilience / UpGuard research rDNS
- Detectify FeedRanges
- Driftnet FeedRangesrDNS
- Escape Security Scanning FeedRanges
- F6 Attack Surface Management rDNSHosts
- Georgia Tech Internet Research FeedRanges
- HackerTarget FeedRanges
- HCL AppScan on Cloud FeedRanges
- HostedScan FeedRangesrDNS
- ImmuniWeb Community Security Tests FeedRanges
- Infrawatch rDNS
- Internet Census Group FeedRangesrDNS
- Internet Cleanup Foundation FeedRanges
- Internet Research Project rDNS
- InternetData FeedrDNS
- Intruder FeedRanges
- Invicti 360 On-Demand FeedRanges
- IPIP security measurement rDNS
- LeakIX FeedRangesrDNS
- Mandiant / Google Threat Intelligence ASM FeedRanges
- Microsoft Internet-Wide Scanning FeedRanges
- Modat FeedRangesrDNS
- NCSC UK Scanning FeedRangesrDNS
- NETSCOUT Internet Safety Initiative FeedRanges
- netSurvey FeedRanges
- Nokia Deepfield measurement rDNS
- OnlyScans FeedRangesrDNS
- ONYPHE FeedRangesrDNS
- Outpost24 OUTSCAN FeedRanges
- Palo Alto Cortex Xpanse FeedRanges
- Pentest-Tools.com FeedRangesrDNS
- PortSwigger Burp Suite DAST Cloud FeedRanges
- Qualys PCI FeedRanges
- Rapid7 InsightAppSec FeedRanges
- Rapid7 Project Sonar FeedRanges
- Reposify / CrowdStrike rDNS
- Research Scanner rDNS
- RWTH Aachen COMSYS Internet Scanning FeedRanges
- SentinelIQ FeedRanges
- Shadowserver rDNS
- Shodan FeedRangesrDNS
- Silent Push FeedRangesrDNS
- Snyk API & Web (Probely) FeedRanges
- Snyk Evo Continuous Offensive Security FeedRanges
- SOCRadar FeedRangesrDNS
- StackHawk Cloud Scanner FeedRanges
- Stanford Internet Research FeedRanges
- Stretchoid rDNS
- Synthient measurement rDNS
- Tenable Attack Surface Management FeedRanges
- Tenable Vulnerability Management and Web App Scanning FeedRanges
- Trustwave External Vulnerability Scanning FeedRanges
- Veracode DAST and Dynamic Analysis FeedRanges
- VisionHeight FeedRangesrDNS
No operator matches “”.
If it publishes its addresses, suggest it.
03 — In the response
Three booleans. Your rules.
The flags arrive with geolocation and network data in the same lookup, recalculated on every request, even when location data comes from cache.
{
// … location and network fields …
"asn": "15169",
"risk": {
"is_ai": false,
"is_crawler": true,
"is_scanner": false
}
}
04 — Questions
About the coverage.
More in the FAQ.
Where does the coverage list come from?
This page is generated from the same source catalogues the API loads. When an operator is added or removed there, this list changes with it.
What does false mean?
No current match. An address that is not in any published range, feed or verified reverse-DNS rule returns false. It does not prove the traffic is human or safe.
Which plan returns the flags?
The risk object with is_ai, is_crawler and is_scanner is returned for Pro API keys. Free keys return geolocation and network data.
Can you add an operator?
If an operator publishes its IP ranges, a feed or a reverse-DNS convention, it can be added. Send the link to its documentation through the contact form.
Pro
Turn on bot signals.
40,000,000 requests a day with is_ai, is_crawler and is_scanner on every response.