A crawler or fetcher run by an AI company: training crawlers, AI search indexers, and agents that fetch pages when a user asks.
The three kinds behave differently. Training crawlers collect content that may be used to train models; search crawlers build an index for AI answers; user-triggered fetchers retrieve a page on demand. OpenAI, Anthropic and Perplexity, for example, give each kind its own user agent and robots.txt token and publish separate IP lists.
In the APIrisk.is_ai (paid plans) is true when the address belongs to a known AI crawler or agent operator.
A number that identifies an autonomous system: a network, run by one organisation, that announces its own IP prefixes to the rest of the internet.
ISPs, cloud providers, universities and large companies each run one or more autonomous systems. Regional internet registries assign the numbers, and BGP routing uses them to decide where traffic goes.
The ASN tells you whose network an address belongs to, which is often more useful than the city: a hosting provider's ASN suggests a server rather than a person.
In the APIasn is a string, for example "15169".
Carrier-grade NAT
also: CGNAT
#
Address sharing by an ISP, where many subscribers reach the internet through one public IPv4 address.
Mobile networks and many home ISPs use it. One public address can then stand for thousands of people, which limits how precise IP geolocation can be and makes blocking a single IP a blunt tool. RFC 6598 reserves 100.64.0.0/10 for the network between the carrier and its subscribers.
A way to write a block of IP addresses as a starting address and a prefix length, such as 198.51.100.0/24.
The number after the slash says how many leading bits are fixed. 198.51.100.0/24 covers the 256 addresses from 198.51.100.0 to 198.51.100.255; a /32 (IPv4) or /128 (IPv6) is a single address.
Crawler operators publish their addresses as lists of CIDR prefixes, and firewalls and key restrictions accept them in the same form.
Forward-confirmed reverse DNS
also: FCrDNS
#
A check that an IP address's reverse-DNS hostname resolves forward to the same IP address.
Look up the PTR record, check that the hostname ends in a domain the operator documents (for example googlebot.com), then resolve that hostname and confirm the original address is among the results. A spoofer can set the PTR on their own block, but cannot make the operator's domain resolve to it.
Search engines such as Google, Bing, Apple and Yandex document this as the way to verify their crawlers.
A time zone name from the IANA tz database, in Area/Location form such as Europe/Athens or America/New_York.
Unlike a fixed offset such as UTC+2, a tz name carries daylight-saving rules, so dates stay right all year. Every major language and database understands these names.
In the APItimezone is an IANA name, for example "Europe/Athens".
A system that probes large parts of the IPv4 address space for open ports, services and vulnerabilities, such as Shodan or Censys.
Research and attack-surface projects scan constantly, so their traffic appears in every server log. Many identify themselves and explain how to opt out of their scans.
In the APIrisk.is_scanner (paid plans) is true when the address matches a known scanning project.
The numeric address a device uses to send and receive traffic on the internet.
Every request your server receives comes from an IP address: the visitor's own, or that of a proxy, VPN, mobile carrier gateway or cloud provider in between. It is the one piece of information you always get, which is why location and bot checks start from it.
In the APIip is the address that was looked up. For a domain lookup, it is the address the domain resolved to.
Estimating where an IP address is used (country, region, city, coordinates) from registry data, routing and measurements.
Country is usually right; city and coordinates are estimates, and can be far off for mobile networks, VPNs, proxies and corporate gateways. Treat it as a default you let people change, not a fact about them.
In the APIcountry, country_code, region, city, zipcode, latitude, longitude and timezone. A field with no known value is the string "No Data".
Version 4 of the Internet Protocol, with 32-bit addresses written as four numbers such as 192.0.2.1.
IPv4 has about 4.3 billion addresses, which ran out years ago. That is why many users now share one public IPv4 address behind carrier-grade NAT.
Version 6 of the Internet Protocol, with 128-bit addresses written in hexadecimal groups such as 2001:db8::1.
IPv6 addresses are plentiful, so devices often get their own global address. Consecutive zero groups are shortened to ::, so the same address can be written in more than one way.
In the APIThe lookup endpoint accepts IPv6 addresses in the path, for example /api/lookup/2001:4860:4860::8888.
ISO 3166-1 alpha-2
also: country code
#
The international standard two-letter country codes, such as GR for Greece, US for the United States and GB for the United Kingdom.
Use the code, not the name, for logic: names vary by language and source, codes don't. Note that the United Kingdom is GB, not UK.
In the APIcountry_code is the alpha-2 code; country is the matching English name.
The ISP is the company that provides the connection; the organization is the entity the address block is registered or assigned to.
For a home connection both are usually the access provider. For a business or cloud customer, the organization can be the customer while the ISP is the network that carries the traffic.
In the APIisp and organization. When only one of them is known, both carry that value.
Published IP ranges
also: egress ranges
#
Lists of CIDR prefixes an operator publishes as the addresses its crawler or service sends requests from.
OpenAI, Anthropic, Perplexity, Google, Bing and others publish such lists, usually as JSON files. They are the most direct way to check that a request claiming to be a given bot really comes from that operator. They change over time, so a copy pasted into a firewall goes stale.
Regional internet registry
also: RIR
#
One of five organisations that allocate IP addresses and ASNs for a region: AFRINIC, APNIC, ARIN, LACNIC and RIPE NCC.
Their registration records say which organisation holds an address block. Registration is not location: a block registered in one country can be used anywhere.
Reverse DNS
also: rDNS, PTR record
#
A DNS lookup that maps an IP address back to a hostname, using a PTR record under in-addr.arpa (IPv4) or ip6.arpa (IPv6).
Whoever controls the address block controls its PTR records, so a hostname like crawl-66-249-66-1.googlebot.com proves nothing on its own. Anyone with their own block can set any name. It only means something when it is forward-confirmed.
A file at the root of a site that tells crawlers which paths they may fetch, standardised as RFC 9309.
It is a request, not access control: well-behaved crawlers follow it, others ignore it. Some tokens, such as Google-Extended, only control how content is used and are never sent as a user agent. To enforce a rule, combine robots.txt with an IP-based check.
Spoofed bot
also: fake crawler
#
A client that sends a well-known crawler's user agent, such as Googlebot or GPTBot, without coming from that crawler's addresses.
Scrapers do this to get past rate limits, paywalls and bot rules that exempt search engines. A user agent that claims a crawler, combined with an IP address that does not belong to that operator, is the signature.
The User-Agent request header, a free-text string in which the client names itself, such as a browser version or "GPTBot/1.1".
Bots identify themselves here, and so do bots pretending to be someone else: the header is set by the client and costs nothing to fake. Use it to decide which verification to run, never as the verification.
Web crawler
also: spider, bot
#
A program that fetches web pages automatically, for example to build a search index, audit SEO or archive the web.
Well-behaved crawlers identify themselves, follow robots.txt and publish how to verify them. Search engines, SEO tools and archives are the common examples.
In the APIrisk.is_crawler (paid plans) is true when the address belongs to a covered search, SEO or archive crawler.
Protocols for querying registration records of IP blocks, ASNs and domains. RDAP is the structured, JSON-based successor to WHOIS.
A registry lookup tells you who holds a network and how to contact them. It is also how you find out who runs a network that is probing you.
A request header in which proxies and load balancers record the client address they received a request from, as a comma-separated list.
Each proxy appends the address that connected to it, so the right-most entries are the ones your own infrastructure added. The left-most entry is whatever the client claimed. Trust only as many entries as you have proxies; the standard Forwarded header (RFC 7239) has the same caveat.