User agents lie
Any scraper can send "GPTBot" or "ClaudeBot". Blocking or allowing on the header alone rewards whoever copies it.
Use case — AI crawlers
Know when an AI is reading your site.
Training crawlers, AI search fetchers and user-triggered agents announce themselves in a user-agent string that anyone can copy. The IP address is harder to fake. risk.is_ai tells you whether a request comes from address space an AI operator publishes as its own.
01 — The problem
Why it's harder than it looks.
Lists go stale
Operators add and retire ranges. A hard-coded list in your firewall drifts out of date without anyone noticing.
You need a decision, not a dataset
Rate limit, serve a licence notice, log for analytics or let it through: you want one boolean at request time.
02 — How it works
One lookup, one answer.
- Each AI operator's published IP list is fetched on a schedule and kept in memory.
- Every lookup checks the address against those lists and sets risk.is_ai, even when the location data comes from cache.
- If an operator's list can't be refreshed, its entries expire instead of being trusted forever.
{
// …
"country_code": "US",
"asn": "15169",
"risk": {
"is_ai": true,
"is_crawler": false,
"is_scanner": false
}
}
03 — Recipe
Drop-in example.
Call the API from your server, never from the browser, so your key stays private. Cache answers per IP on your side to save requests.
// Refuse requests from published AI crawler infrastructure.
// Needs a Pro key. Behind a proxy, set app.set('trust proxy', ...) so req.ip is the client.
const LOOKUP = 'https://fast.ipapi.gr/api/lookup/';
async function blockAiCrawlers(req, res, next) {
try {
const r = await fetch(LOOKUP + encodeURIComponent(req.ip) + '?key=' + process.env.IPAPI_KEY,
{ signal: AbortSignal.timeout(1500) });
if (r.ok) {
const { risk } = await r.json();
if (risk && risk.is_ai) return res.status(403).send('Automated AI access is not permitted.');
}
} catch {
// Fail open: a lookup error should never lock out real visitors.
}
next();
}
app.use(blockAiCrawlers);
04 — Honest limits
What it won't tell you.
- is_ai says the address belongs to published AI infrastructure. It does not name the operator or the product.
- Operators that publish no IP ranges cannot be detected this way. false means "no current match", not "human".
- Some operators share infrastructure between AI and search crawling, so an address can be both is_ai and is_crawler.
05 — Questions
Good to know.
Which AI crawlers does risk.is_ai cover?
Every operator listed on the bot signals coverage page, generated from the same source list the API uses. It includes published ranges for crawlers such as GPTBot, ClaudeBot and PerplexityBot.
Is risk.is_ai available on the free plan?
No. The risk object with is_ai, is_crawler and is_scanner is part of the Pro plan. Free keys return geolocation and network data.
Can I use it to block AI training crawlers?
Yes. Call the lookup from your server or edge for the client IP and refuse, rate limit or tag the request when risk.is_ai is true. Many operators also honour robots.txt, which you can use alongside it.
From the blog
Go deeper.
- How to verify AI crawler IP addresses (GPTBot, ClaudeBot, PerplexityBot)
- OpenAI crawler IP ranges explained: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot
- ClaudeBot and PerplexityBot IP addresses: the official lists and how to use them
- Block or allow AI crawlers in Nginx by verified IP, not just user agent
Other use cases
Add bot signals to every lookup.
Pro: 40,000,000 requests a day with is_ai, is_crawler and is_scanner.