Spoofed user agents
A request that says "Googlebot" but comes from a residential proxy should not get crawler treatment.
Use case — Bot verification
Tell real search crawlers from impostors.
Scrapers routinely pretend to be Googlebot to get past rate limits and paywalls. risk.is_crawler checks the address against the ranges and reverse-DNS rules the search and SEO operators publish, so a copied user agent is not enough.
01 — The problem
Why it's harder than it looks.
DNS checks are slow to build
Forward-confirmed reverse DNS is the documented way to verify many crawlers, but doing it on every request needs resolvers, caching and timeouts.
Every operator is different
Some publish JSON range files, some HTML pages, some only a hostname suffix. You want one answer.
02 — How it works
One lookup, one answer.
- Operators that publish IP ranges are matched against those ranges, refreshed on a schedule.
- Operators that document a reverse-DNS suffix are verified by resolving the PTR record and confirming it resolves back to the same address.
- The result is one boolean, risk.is_crawler, on every Pro lookup.
{
// …
"country_code": "US",
"asn": "15169",
"risk": {
"is_ai": false,
"is_crawler": true,
"is_scanner": false
}
}
03 — Recipe
Drop-in example.
Call the API from your server, never from the browser, so your key stays private. Cache answers per IP on your side to save requests.
import os
import requests
CLAIMS = ("googlebot", "bingbot", "applebot", "duckduckbot", "yandex", "baiduspider")
def crawler_status(ip: str, user_agent: str) -> str:
"""'verified', 'unverified' (claims to be a crawler but isn't) or 'not a crawler'."""
r = requests.get(
f"https://fast.ipapi.gr/api/lookup/{ip}",
params={"key": os.environ["IPAPI_KEY"]}, # Pro key
timeout=2,
)
r.raise_for_status()
verified = r.json().get("risk", {}).get("is_crawler", False)
if verified:
return "verified"
if any(name in user_agent.lower() for name in CLAIMS):
return "unverified"
return "not a crawler"
04 — Honest limits
What it won't tell you.
- is_crawler does not say which crawler matched. Combine it with the user agent: a known crawler user agent with is_crawler false is unverified.
- Only operators on the coverage list are verified. Unlisted crawlers return false.
- Shared infrastructure (for example Bingbot) can make an address both is_ai and is_crawler.
05 — Questions
Good to know.
How do I verify Googlebot with the API?
Look up the client IP. If the user agent claims Googlebot and risk.is_crawler is true, the address is in Google's published crawler ranges. If it is false, treat the request as unverified.
Which crawlers are covered?
The bot signals coverage page lists every search engine, SEO tool and archive crawler the flag recognises, and how each one is verified.
Do I need the Pro plan?
Yes. The risk flags are returned only for Pro API keys.
Other use cases
Add bot signals to every lookup.
Pro: 40,000,000 requests a day with is_ai, is_crawler and is_scanner.