How to verify AI crawler IP addresses (GPTBot, ClaudeBot, PerplexityBot)

Open the access log of any public website and you will find user agents that contain GPTBot, ClaudeBot or PerplexityBot. Some of those requests really come from OpenAI, Anthropic and Perplexity. Others come from scrapers that copied the string, because a famous user agent often gets better treatment than an unknown one.

The user agent is just a header, and the client chooses it. To know who is actually on the other end you need something the client cannot pick freely: the source IP address. This guide shows how to check an IP against the lists the AI operators publish themselves, and how to do it in production without writing a crawler-list maintenance job.

Why the user agent is not enough

A request's User-Agent header is self-reported. Changing it is one flag in curl:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://example.com/

So two things can go wrong:

1. Spoofing. A scraper claims to be GPTBot or ClaudeBot. If your rules wave AI crawlers through (for example because you want to appear in AI search answers), you have just let the scraper in too.
2. Mislabelling. A real AI fetcher sends a generic browser user agent, or a user agent you don't recognise yet. User-agent rules miss it completely.

The IP address solves both. A TCP connection cannot complete with a forged source address, so if a request arrives from an address inside OpenAI's published GPTBot ranges, it came from that infrastructure. Bot traffic sent through a residential proxy will not match any published range, and that is the point: it isn't the operator.

Which AI companies publish their crawler IPs

More and more AI operators publish a machine-readable list of the addresses their bots use. Most of them use the same small JSON format Google introduced for Googlebot: a creationTime and a list of prefixes, each with an ipv4Prefix or ipv6Prefix in CIDR notation.

That is not the full list. Amazon, Moonshot (Kimi), You.com and several smaller AI crawlers publish addresses too. Our bot signals coverage page lists every source the API currently uses.

Two details matter when you use these files:

• One operator, several bots, several purposes. OpenAI separates GPTBot (collects content that may be used for training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (fetches a page because a user asked). Blocking one does not block the others. We cover the differences in OpenAI crawler IP ranges explained.
• User-triggered fetchers may ignore robots.txt. OpenAI says that for ChatGPT-User "robots.txt rules may not apply", and Perplexity says Perplexity-User "generally ignores robots.txt rules" because a person asked for the page. If you need to control those visits, the IP is your best handle.

Check one IP by hand

To check a single address from your logs, download the right file and see whether the address falls inside any prefix. With jq and Python's standard library:

curl -s https://openai.com/gptbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix' > gptbot.txt
python3 -c "
import ipaddress, sys
ip = ipaddress.ip_address(sys.argv[1])
nets = [ipaddress.ip_network(l.strip()) for l in open('gptbot.txt') if l.strip()]
print('match' if any(ip in n for n in nets if n.version == ip.version) else 'no match')
" 203.0.113.50

Replace 203.0.113.50 with the address from your log line. A match means the request came from GPTBot infrastructure. No match means it did not come from the GPTBot ranges. It may still be another OpenAI bot, so check the file for the user agent it claimed.

Check in code (Node.js)

For a server that needs to decide on every request, load the lists once, refresh them on a timer, and match in memory. Node.js has a built-in CIDR matcher, net.BlockList, so you need no dependency:

// ESM (save as .mjs or set "type": "module"), Node.js 18+
import { BlockList, isIPv6 } from 'node:net';

const SOURCES = {
  gptbot: 'https://openai.com/gptbot.json',
  'oai-searchbot': 'https://openai.com/searchbot.json',
  'chatgpt-user': 'https://openai.com/chatgpt-user.json',
  claude: 'https://claude.com/crawling/bots.json',
  perplexitybot: 'https://www.perplexity.ai/perplexitybot.json',
  'perplexity-user': 'https://www.perplexity.ai/perplexity-user.json'
};

let lists = new Map();

async function refresh() {
  const next = new Map();
  for (const [name, url] of Object.entries(SOURCES)) {
    const res = await fetch(url, { signal: AbortSignal.timeout(10000) });
    if (!res.ok) throw new Error(`${name}: HTTP ${res.status}`);
    const { prefixes = [] } = await res.json();
    const list = new BlockList();
    for (const p of prefixes) {
      const cidr = p.ipv4Prefix || p.ipv6Prefix;
      if (!cidr) continue;
      const [addr, bits] = cidr.split('/');
      list.addSubnet(addr, Number(bits), p.ipv6Prefix ? 'ipv6' : 'ipv4');
    }
    next.set(name, list);
  }
  lists = next; // swap only when every file loaded
}

function matchAiBot(ip) {
  const type = isIPv6(ip) ? 'ipv6' : 'ipv4';
  for (const [name, list] of lists) {
    if (list.check(ip, type)) return name;
  }
  return null;
}

await refresh();
setInterval(() => refresh().catch((err) => console.error('AI ranges refresh failed:', err.message)), 6 * 60 * 60 * 1000);

matchAiBot(req.ip) returns the bot family or null. Note the design choices: a failed refresh keeps the last good lists instead of emptying them, and every file has a timeout.

Check in code (Python)

The same idea in Python with the standard library only:

import ipaddress
import json
import urllib.request

SOURCES = {
    "gptbot": "https://openai.com/gptbot.json",
    "claude": "https://claude.com/crawling/bots.json",
    "perplexitybot": "https://www.perplexity.ai/perplexitybot.json",
}

def load_ranges():
    ranges = {}
    for name, url in SOURCES.items():
        with urllib.request.urlopen(url, timeout=10) as resp:
            data = json.load(resp)
        ranges[name] = [
            ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in data.get("prefixes", [])
            if p.get("ipv4Prefix") or p.get("ipv6Prefix")
        ]
    return ranges

def match_ai_bot(ip, ranges):
    addr = ipaddress.ip_address(ip)
    for name, nets in ranges.items():
        if any(addr in net for net in nets if net.version == addr.version):
            return name
    return None

A linear scan is fine for a few hundred prefixes. If you add many operators, build a sorted interval list or use a radix tree.

The maintenance problem

The code above works, but it is the easy part. What makes this hard in practice:

• The lists change. When we checked on 12 November 2026, OpenAI's chatgpt-user.json had been regenerated less than a week earlier. A list you copied into a config file a month ago is already out of date.
• Formats are not uniform. Most operators use the Google-style JSON, but some publish plain text, HTML tables or documentation pages.
• Some files go stale. Bing's bingbot.json still carried a creationTime from January 2024 when we checked. Your refresh job should notice when a source stops changing or starts failing, and should never treat an empty download as "no bots".
• Each operator has several bots. You have to decide, per bot, whether you want it, which means tracking new user agents as they appear.

If you want to keep it in-house, budget for a scheduled job, alerting on failed or empty downloads, and a review whenever an operator announces a new bot.

Or ask an API

This is the problem the risk object in the IPAPI.GR lookup API solves. On the Pro plan every lookup includes three booleans:

{
  "ip": "203.0.113.50",
  "country_code": "US",
  "isp": "Example Hosting",
  "asn": "64500",
  "risk": {
    "is_ai": true,
    "is_crawler": false,
    "is_scanner": false
  }
}

risk.is_ai is true when the address is inside a range an AI operator publishes for its crawlers, search fetchers or agents. We build it only from operator-published sources like the files in the table above. We don't guess from hosting providers, traffic patterns or third-party blocklists. The flags are recalculated on every request, even when the location data comes from cache.

Calling it from Node.js:

const ip = req.ip;
const res = await fetch(`https://fast.ipapi.gr/api/lookup/${encodeURIComponent(ip)}?key=${process.env.IPAPI_KEY}`);
const data = await res.json();

if (!res.ok) throw new Error(data.message);
if (data.risk?.is_ai) {
  // verified AI crawler, fetcher or agent: allow, rate limit, tag or block
}

The same check in PHP, for WordPress plugins or any PHP app:

<?php
function is_verified_ai(string $ip): bool {
    $url = 'https://fast.ipapi.gr/api/lookup/' . rawurlencode($ip) . '?key=' . urlencode(getenv('IPAPI_KEY'));
    $ch = curl_init($url);
    curl_setopt_array($ch, [CURLOPT_RETURNTRANSFER => true, CURLOPT_TIMEOUT => 3]);
    $body = curl_exec($ch);
    $ok = curl_getinfo($ch, CURLINFO_HTTP_CODE) === 200;
    curl_close($ch);
    $data = $ok && $body ? json_decode($body, true) : [];
    return !empty($data['risk']['is_ai']);
}

Two rules for reading the result:

• true is evidence. false is not proof of a human. false only means "no current match in a published list". An AI company that publishes nothing will never match, and neither will a scraper.
• Combine it with the user agent. If the user agent says GPTBot and is_ai is false, treat the request as an impostor. If is_ai is true but the user agent looks like a browser, it is probably a user-triggered fetch such as ChatGPT-User or Perplexity-User.

The risk object is part of Pro. Free keys return the geolocation and network fields, which is enough to try the integration first. Field details are in the API docs.

What to do with a verified AI crawler

Verification tells you who. What you do next is a policy decision:

• Allow it if you want your pages to appear in ChatGPT search, Perplexity answers or Claude's web results. Verify, so that only the real bots get the allowance.
• Rate limit it separately from human traffic, so a crawl burst cannot slow your site down.
• Block training but allow search. OpenAI and Anthropic separate their training crawlers from their search bots. Robots.txt is the documented way to opt out, and IP checks let you enforce it. See Block AI crawlers in Nginx by verified IP.
• Log and measure first. Many sites find that a large share of "GPTBot" requests are not from OpenAI at all.

FAQ

Do all AI companies publish their crawler IPs?

No. Many of the large ones do (OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Mistral and others), but some crawlers publish only a user agent, or nothing. Requests from those can't be verified by IP.

Can an AI crawler use IPs outside its published list?

The lists describe what the operator declares. Traffic from undeclared addresses won't match, which is the point of verification: you can only trust what the operator stands behind.

How often should I refresh the lists?

At least daily, and every few hours for the busiest files. Keep the last good copy when a download fails.

Is IPv6 covered?

Where the operator publishes IPv6 prefixes (ipv6Prefix entries), yes. Several AI operators currently publish IPv4 only.

Does robots.txt still matter if I verify IPs?

Yes. Robots.txt is how the AI operators document their opt-out. IP verification is how you enforce your policy and detect impostors.

Summary

• A user agent is a claim. A source IP inside an operator's published range is evidence.
• OpenAI, Anthropic, Perplexity, Google, Apple, Mistral and others publish JSON lists of their bot addresses. Use the official files, not copies on third-party sites.
• Doing it yourself means downloading, refreshing, monitoring and matching several lists in different formats.
• One lookup with risk.is_ai gives you the same answer from the same official sources, refreshed for you.

Next: how to verify Googlebot and Bingbot with reverse DNS, and the less polite traffic, internet scanners.