How to block AI scrapers: what you can verify, what you can't, and the code for both

"How do I block AI scrapers?" is really two questions, and most guides answer only the first.

1. How do I stop the AI companies' declared crawlers? GPTBot, ClaudeBot, CCBot and the rest identify themselves, publish their rules, and most publish their IP addresses. This part is solvable, and you can solve it exactly.
2. How do I stop scrapers that don't identify themselves? Headless browsers, scraping frameworks, and tools that borrow a famous user agent. Nobody publishes their addresses. This part can't be solved exactly. You can only make it expensive.

Treating them as one problem leads to the two classic mistakes: a user-agent blocklist that only stops honest bots, or a blanket block of cloud networks that also stops Googlebot, Amazonbot and your own monitoring. This guide handles each kind with the tool that fits it.

First, measure: what is actually hitting you?

Before you block anything, look at your logs. The script below reads an access log in the common "combined" format (the default for Nginx and Apache), and for every request that claims to be an AI bot, checks the client IP against that operator's official list. It also counts scripted clients and the busiest networks.

#!/usr/bin/env python3
"""Audit an access log (combined format) for AI bot claims and scraper traffic.
Usage: python3 ai_log_audit.py /var/log/nginx/access.log"""
import ipaddress, json, re, sys, urllib.request
from collections import Counter

# user-agent token -> official IP files of that operator (checked 12 Nov 2026)
CLAIMS = {
    "GPTBot": ["https://openai.com/gptbot.json"],
    "OAI-SearchBot": ["https://openai.com/searchbot.json"],
    "ChatGPT-User": ["https://openai.com/chatgpt-user.json"],
    "ClaudeBot": ["https://claude.com/crawling/bots.json"],
    "Claude-SearchBot": ["https://claude.com/crawling/bots.json"],
    "Claude-User": ["https://claude.com/crawling/bots.json"],
    "PerplexityBot": ["https://www.perplexity.ai/perplexitybot.json"],
    "Perplexity-User": ["https://www.perplexity.ai/perplexity-user.json"],
    "Applebot": ["https://search.developer.apple.com/applebot.json"],
    "CCBot": ["https://index.commoncrawl.org/ccbot.json"],
    "Amazonbot": ["https://developer.amazon.com/amazonbot/ip-addresses/"],
}
TOOLS = re.compile(r"python-requests|python-urllib|aiohttp|httpx|curl/|wget/|Go-http-client|"
                   r"Scrapy|HeadlessChrome|node-fetch|axios/|okhttp|Java/|libwww-perl", re.I)
LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "[^"]*" \d{3} \S+ "[^"]*" "([^"]*)"')

def fetch_networks(url):
    req = urllib.request.Request(url, headers={"User-Agent": "log-audit/1.0"})
    with urllib.request.urlopen(req, timeout=15) as r:
        text = r.read().decode("utf-8", "replace")
    if not text.lstrip().startswith("{"):  # Amazon: JSON inside an HTML page
        m = re.search(r"<code[^>]*>\s*(\{.*?\})\s*</code>", text, re.S)
        text = m.group(1)
    nets = []
    for p in json.loads(text).get("prefixes", []):
        v = p.get("ipv4Prefix") or p.get("ipv6Prefix") or p.get("ip_prefix")
        if v:
            nets.append(ipaddress.ip_network(v, strict=False))
    return nets

def main(path):
    cache = {}
    def networks(token):
        out = []
        for url in CLAIMS[token]:
            if url not in cache:
                cache[url] = fetch_networks(url)
            out += cache[url]
        return out

    verified, spoofed, tools, nets = Counter(), Counter(), Counter(), Counter()
    with open(path, encoding="utf-8", errors="replace") as f:
        for line in f:
            m = LINE.match(line)
            if not m:
                continue
            ip_s, ua = m.groups()
            try:
                ip = ipaddress.ip_address(ip_s)
            except ValueError:
                continue
            token = next((t for t in CLAIMS if t.lower() in ua.lower()), None)
            if token:
                ok = any(ip in n for n in networks(token) if n.version == ip.version)
                (verified if ok else spoofed)[token] += 1
                if not ok:
                    nets[ip_s] += 1
                continue
            if TOOLS.search(ua):
                tools[TOOLS.search(ua).group(0).rstrip("/")] += 1
            prefix = ipaddress.ip_network(f"{ip}/{24 if ip.version == 4 else 48}", strict=False)
            nets[str(prefix)] += 1

    print("Claimed AI bots, verified by IP:")
    for t in sorted(set(verified) | set(spoofed)):
        print(f"  {t:18} verified {verified[t]:>7}   spoofed {spoofed[t]:>7}")
    print("\nScripted clients by user agent:")
    for name, n in tools.most_common(10):
        print(f"  {name:18} {n:>7}")
    print("\nBusiest sources (spoofed bot IPs, and /24 or /48 networks for everything else):")
    for net, n in nets.most_common(15):
        print(f"  {net:24} {n:>7}")

if __name__ == "__main__":
    main(sys.argv[1])

It needs only the Python standard library. Run against a small test log with one real GPTBot address from gptbot.json, one fake GPTBot, a real Amazonbot, a real Applebot and two scripted clients, it prints:

Claimed AI bots, verified by IP:
  Amazonbot          verified       1   spoofed       0
  Applebot           verified       1   spoofed       0
  GPTBot             verified       1   spoofed       1

Scripted clients by user agent:
  python-requests          1
  HeadlessChrome           1

Busiest sources (spoofed bot IPs, and /24 or /48 networks for everything else):
  198.51.100.0/24                2
  203.0.113.9                    1

On a real site, the "spoofed" column is the interesting one. Any non-zero count means someone is borrowing a bot's name, usually because they expect a bot-friendly site to wave it through. If your web server sits behind a CDN or load balancer, make sure the first field of the log is the real client IP (Nginx real_ip or Apache mod_remoteip), or every request will look like it came from your proxy.

Part 1: declared AI crawlers

Decide what you want

"Block AI" isn't one setting. The declared bots split into three groups, and most sites want different answers for each:

Our AI crawler list gives the token, purpose and IP file for each bot, checked against the operators' pages.

Step 1: robots.txt

For training crawlers, robots.txt is the documented opt-out, and it's the one the operators promise to honour. OpenAI, Anthropic, Common Crawl, Amazon, Apple (through Applebot-Extended), Google (through Google-Extended) and Meta all document it. Two of those, Google-Extended and Applebot-Extended, are only robots.txt tokens: no separate crawler exists, so there is nothing to block at the network level.

There's also a reason to keep robots.txt even if you add a firewall: Anthropic notes that blocking its IPs "may not work correctly or persistently guarantee an opt-out", because a bot that can't fetch your robots.txt can't read your preferences either. Kimi says the same. If you block by IP, still serve /robots.txt to everyone.

Step 2: verify every claim by IP

Robots.txt does nothing about an impostor. And the moment you allow a bot that your firewall would otherwise challenge, a user-agent rule becomes a hole: anyone who types PerplexityBot into their scraper gets your allow rule.

The fix is to verify the claim. Most AI operators publish the IP addresses their bots use. A request that says GPTBot and comes from an address in gptbot.json is GPTBot. One that says GPTBot and comes from anywhere else isn't. A TCP connection can't complete from a forged source address, so this check is reliable in a way the header is not.

Here is that logic as Express middleware for Node.js 18 or later, with no dependencies. It comes in two files. The first fetches the operator publications when the server starts and on a timer, never per request. It accepts both the Google-style JSON files and the Amazon pages that wrap JSON in HTML:

// ranges.mjs: download the official AI crawler IP files. Node.js 18+ (built-in fetch).
// Most files use Google's format: { creationTime, prefixes: [{ ipv4Prefix | ipv6Prefix }] }.
// Amazon serves the same JSON inside an HTML page, uses bare addresses,
// and one of its lists uses the key "ip_prefix".
export async function fetchPrefixes(url) {
  const res = await fetch(url, { signal: AbortSignal.timeout(15_000) });
  if (!res.ok) throw new Error(`${url}: HTTP ${res.status}`);
  let text = await res.text();
  if (!text.trimStart().startsWith('{')) {
    const m = text.match(/<code[^>]*>\s*(\{[\s\S]*?\})\s*<\/code>/);
    if (!m) throw new Error(`${url}: no JSON found`);
    text = m[1];
  }
  const data = JSON.parse(text);
  const out = [];
  for (const p of data.prefixes ?? []) {
    const v = p.ipv4Prefix ?? p.ipv6Prefix ?? p.ip_prefix;
    if (typeof v !== 'string') continue;
    out.push(v.includes('/') ? v : `${v}/${v.includes(':') ? 128 : 32}`);
  }
  if (out.length === 0) throw new Error(`${url}: empty list`);
  return out;
}

The second file is the middleware itself. It keeps the downloaded prefixes in BlockList objects in memory and decides per bot whether to block or allow, but only after the IP has confirmed the bot is real:

// ai-bots.mjs: Express middleware. Verifies AI bot claims against the operators' IP files.
import { BlockList, isIPv6 } from 'node:net';
import { fetchPrefixes } from './ranges.mjs';

// token in the user agent -> official IP file -> what you want to happen
const BOTS = [
  { token: 'gptbot', url: 'https://openai.com/gptbot.json', action: 'block' },
  { token: 'oai-searchbot', url: 'https://openai.com/searchbot.json', action: 'allow' },
  { token: 'chatgpt-user', url: 'https://openai.com/chatgpt-user.json', action: 'allow' },
  { token: 'claudebot', url: 'https://claude.com/crawling/bots.json', action: 'block' },
  { token: 'claude-searchbot', url: 'https://claude.com/crawling/bots.json', action: 'allow' },
  { token: 'claude-user', url: 'https://claude.com/crawling/bots.json', action: 'allow' },
  { token: 'perplexitybot', url: 'https://www.perplexity.ai/perplexitybot.json', action: 'allow' },
  { token: 'perplexity-user', url: 'https://www.perplexity.ai/perplexity-user.json', action: 'allow' },
  { token: 'ccbot', url: 'https://index.commoncrawl.org/ccbot.json', action: 'block' },
];

const lists = new Map(); // url -> BlockList (last good copy)

async function refresh() {
  for (const url of new Set(BOTS.map((b) => b.url))) {
    try {
      const list = new BlockList();
      for (const cidr of await fetchPrefixes(url)) {
        const [net, bits] = cidr.split('/');
        list.addSubnet(net, Number(bits), isIPv6(net) ? 'ipv6' : 'ipv4');
      }
      lists.set(url, list);
    } catch (err) {
      console.error('AI bot list refresh failed, keeping the previous copy:', err.message);
    }
  }
}
await refresh();
setInterval(refresh, 60 * 60 * 1000).unref();

const clientIp = (req) => (req.ip || '').replace(/^::ffff:/, '');

export function aiBotPolicy(req, res, next) {
  const ua = (req.get('user-agent') || '').toLowerCase();
  const bot = BOTS.find((b) => ua.includes(b.token));
  if (!bot) return next();                    // not claiming to be an AI bot
  const list = lists.get(bot.url);
  if (!list) return next();                   // list never loaded: fail open
  const ip = clientIp(req);
  const verified = list.check(ip, isIPv6(ip) ? 'ipv6' : 'ipv4');
  if (!verified) return res.status(403).type('text').send('Unverified crawler');
  res.locals.verifiedBot = bot.token;         // for logging, analytics, caching
  if (bot.action === 'block') return res.status(403).type('text').send('Crawling not permitted');
  return next();
}

Wire it in before your routes, and tell Express how many proxies sit in front of it so req.ip is the real client:

import express from 'express';
import { aiBotPolicy } from './ai-bots.mjs';

const app = express();
app.set('trust proxy', 'loopback'); // e.g. Nginx on the same host; adjust to your setup
app.get('/robots.txt', (req, res) => res.sendFile('/srv/site/robots.txt')); // always reachable
app.use(aiBotPolicy);

We tested this with real addresses from the official files. A request with GPTBot's user agent from 132.196.86.5 (inside 132.196.86.0/24 in gptbot.json) gets the "Crawling not permitted" 403. The same user agent from 203.0.113.9 gets "Unverified crawler". OAI-SearchBot from 104.210.140.130 (inside 104.210.140.128/28 in searchbot.json) and PerplexityBot from 107.20.236.150 (listed in perplexitybot.json) pass through.

Three design choices worth copying:

• Fail open on first load, keep the last good copy after that. If an operator's file is briefly unreachable, you don't want to start blocking a search crawler you allowed.
• Don't verify the bots you don't care about. A request that claims nothing passes straight through. This middleware only judges claims.
• The robots.txt route comes before the policy, so blocked bots can still read your opt-out.

Step 3: the bots you can't verify by IP

A few AI crawlers publish no IP list. On 12 Nov 2026 that included Meta's crawlers (Meta-ExternalAgent and friends) and MistralAI-Training. For those you have the robots.txt opt-out and the user agent. Treat the user agent as a label you act on only to block, never to allow: blocking a fake Meta-ExternalAgent costs you nothing, but allowing one opens a hole.

Part 2: scrapers that don't say who they are

This is the harder half, and it's worth being honest about it: no IP list identifies undeclared scrapers. They run on cloud servers, on rented residential connections, or in real browsers. Anyone who tells you a single API flag catches them all is overselling. IPAPI.GR doesn't detect proxies or VPNs either; its flags only cover known AI crawlers, crawlers and scanners.

What works is a set of cheap signals that, combined, make scraping your site slow and expensive.

Signals worth watching

• Scripting libraries in the user agent. python-requests, Go-http-client, Scrapy, curl, HeadlessChrome. Lazy scrapers don't change the default. Legitimate API clients use them too, so apply this to your HTML pages, not your API.
• Request rate per network, not just per IP. Distributed scrapers rotate addresses inside the same provider's ranges. Counting per /24 (IPv4) or /48 (IPv6), or per ASN, shows a pattern that per-IP limits miss. The log audit above already groups by /24.
• Fetching only HTML. Real browsers request your CSS, JavaScript and images. A client that pulls thousands of pages and no assets is not a person reading them.
• Crawling in order. Paginated listings hit page by page, or every product ID in sequence.
• Claiming a bot it isn't. Every spoofed claim the middleware above rejects is a scraper you've already identified. Log those IPs: they often send other traffic too.

Responses that scale

1. Rate limit per network. Generous enough for a human, tight enough to make bulk copying slow. Apply it after the verified-bot check, so a real Googlebot or OAI-SearchBot isn't throttled by mistake.
2. Challenge, don't block, where you can. A JavaScript or CAPTCHA challenge on expensive or valuable paths stops scripts and costs humans a second. Cloudflare, Vercel and AWS WAF all offer challenge actions.
3. Keep the high-value data behind a login or an API key. Then you control who gets it and how much.
4. Don't block whole clouds. It's tempting to deny every AWS, Azure or Google Cloud range. But verified bots live there. When we checked on 12 Nov 2026, Amazonbot's published addresses had EC2 hostnames, GPTBot's ranges were announced by Microsoft (AS8075), and Anthropic's 216.73.216.0/22 and PerplexityBot's addresses by Amazon (AS16509 and AS14618). Many uptime monitors and payment webhooks run in the same clouds. If you restrict cloud traffic, do it after the verified-bot check, and as a challenge rather than a block.

robots.txt vs IP blocking for AI: which one?

Both, for different jobs:

Doing the verification with one API call

The middleware above is fine for a handful of operators. If you want every known AI, search and SEO crawler without maintaining the downloads, the IPAPI.GR lookup returns risk.is_ai and risk.is_crawler on any paid plan. IPAPI.GR's risk flags tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. The same response includes asn, isp and organization, which is what you need to rate limit per network.

async function classify(ip) {
  const url = `https://ipapi.gr/api/lookup/${encodeURIComponent(ip)}?key=${process.env.IPAPI_KEY}`;
  const res = await fetch(url, { signal: AbortSignal.timeout(3000) });
  if (!res.ok) return null; // treat as unknown, don't block on errors
  const data = await res.json();
  return {
    aiBot: data.risk?.is_ai === true,
    searchOrSeoBot: data.risk?.is_crawler === true,
    network: data.asn, // e.g. "15169"; may be "No Data"
  };
}

Remember what the flags mean: true means the address belongs to a known AI crawler, search or SEO crawler, or scanner. false means it isn't one of those, not "human". For a one-off check of an address from your logs, the free bot verification tool gives the same verdict, and the bot signals page lists every bot it covers.

FAQ

Will blocking GPTBot hurt my visibility in ChatGPT?

Not in search. ChatGPT search uses OAI-SearchBot, which you can keep allowing. GPTBot only collects content that may be used for training.

Is it legal for AI companies to ignore robots.txt?

That depends on your jurisdiction and isn't something an IP check can settle. Practically, the training crawlers listed above all document that they honour robots.txt. The exceptions are user-triggered fetchers, which several operators say may not follow it because a person asked for the page.

Can I just block every user agent that contains "bot"?

You'll block Googlebot, Bingbot and Applebot, so your search traffic goes with it. And scrapers don't put "bot" in their user agent. Verify claims instead, and handle undeclared traffic with rate limits.

How often should I refresh the IP lists?

Hourly is enough. Most files change every few weeks. What matters more is keeping the last good copy when a download fails, and never hard-coding addresses from an article.