Block or allow AI crawlers in Nginx by verified IP, not just user agent

Search for "block AI crawlers nginx" and nearly every result gives you the same snippet: a map on $http_user_agent with a list of bot names, then return 403. It's short and it works against bots that announce themselves honestly. It has two problems:

1. The bots that announce themselves honestly also follow robots.txt. GPTBot, ClaudeBot and the other training crawlers document robots.txt as their opt-out, so a user-agent block mostly repeats what robots.txt already says.
2. It does nothing about impostors, and it can't allow a bot safely. If you want OAI-SearchBot or PerplexityBot in so you appear in AI search, a user-agent allow rule lets in anyone who copies the name.

This guide builds a setup where Nginx knows whether an IP really belongs to an AI operator, using the official IP files, and combines that with the user agent.

Step 0: robots.txt still comes first

Robots.txt is the documented opt-out for OpenAI, Anthropic and Google's AI training. Anthropic notes that IP blocks may not work as an opt-out, because a blocked bot can't read your robots.txt. So start with it:

# No model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Yes to AI search
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Nginx then enforces the policy for traffic that ignores robots.txt, and stops impostors.

Step 1: generate an Nginx geo file from the official lists

Nginx's geo module maps a client address to a value using CIDR ranges, and it's fast even with thousands of entries. We generate its input from the operators' JSON files with a small Python script:

#!/usr/bin/env python3
"""Writes /etc/nginx/ai-ranges.conf from operator-published IP lists."""
import ipaddress
import json
import os
import sys
import tempfile
import urllib.request

SOURCES = {
    "openai": [
        "https://openai.com/gptbot.json",
        "https://openai.com/searchbot.json",
        "https://openai.com/chatgpt-user.json",
    ],
    "anthropic": ["https://claude.com/crawling/bots.json"],
    "perplexity": [
        "https://www.perplexity.ai/perplexitybot.json",
        "https://www.perplexity.ai/perplexity-user.json",
    ],
}
OUT = "/etc/nginx/ai-ranges.conf"

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "ai-ranges-sync/1.0"})
    with urllib.request.urlopen(req, timeout=20) as resp:
        data = json.load(resp)
    nets = []
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr, strict=False))
    if not nets:
        raise ValueError(f"{url}: no prefixes")  # never publish an empty list
    return nets

lines = []
for operator, urls in SOURCES.items():
    for url in urls:
        for net in fetch(url):
            lines.append(f"{net} {operator};")

fd, tmp = tempfile.mkstemp(dir=os.path.dirname(OUT))
with os.fdopen(fd, "w") as f:
    f.write("# generated - do not edit\n" + "\n".join(sorted(set(lines))) + "\n")
os.chmod(tmp, 0o644)
os.replace(tmp, OUT)  # atomic swap
print(f"wrote {len(lines)} ranges", file=sys.stderr)

Any failed download or empty file raises an exception, so the old ai-ranges.conf stays in place. A failed refresh never removes your protection or lets everyone in.

Run it from cron and reload Nginx only if the new config is valid:

# /etc/cron.d/ai-ranges : every 6 hours
17 */6 * * * root /usr/local/bin/ai-ranges-sync && nginx -t -q && nginx -s reload

Step 2: wire it into Nginx

In the http block:

# Which AI operator owns the client IP (from the generated file)
geo $ai_operator {
    default "";
    include /etc/nginx/ai-ranges.conf;
}

# Which AI bot the client claims to be
map $http_user_agent $ai_claim {
    default "";
    ~*GPTBot            training;
    ~*ClaudeBot         training;
    ~*OAI-SearchBot     search;
    ~*Claude-SearchBot  search;
    ~*PerplexityBot     search;
    ~*ChatGPT-User      user;
    ~*Claude-User       user;
    ~*Perplexity-User   user;
}

# Decision: claim + verification
map "$ai_claim:$ai_operator" $ai_block {
    default             0;
    "~^training:.+$"    1;   # verified training crawler: block (robots.txt says no)
    "~^[a-z]+:$"        2;   # claims to be an AI bot, IP not in any list: impostor
}

Note the regex order. Nginx checks regex entries in the order they appear, so a verified training crawler matches the first rule and an impostor (empty operator) matches the second.

In the server block:

location / {
    if ($ai_block = 1) { return 403; }
    if ($ai_block = 2) { return 403; }
    # ... your normal config
}

if with only a return inside is one of the safe uses of if in Nginx.

You can separate the two cases in logs by adding $ai_block and $ai_operator to your log_format. It's worth doing for a week before you enforce anything. Many sites find that a noticeable share of "GPTBot" traffic fails verification.

Step 3: decide what to do with user-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked an assistant about it. OpenAI and Perplexity say these fetchers may not follow robots.txt. The config above lets verified ones through (they aren't in the training rule) and blocks impostors. If you'd rather block them, add "~^user:.+$" 1; to the decision map. Think about the trade-off first: these are often real people asking about your page.

Alternative: one lookup instead of a sync job

The generated file covers the three operators above. Adding Google's agents, Apple, Amazon, Mistral, DuckDuckGo and the rest means more URLs, more formats and more ways for a refresh to fail. If you'd rather not run that, the IPAPI.GR lookup API returns risk.is_ai (and is_crawler, is_scanner) from the operators' published sources. A small local service can serve Nginx's auth_request:

// ai-gate.mjs : run next to Nginx on 127.0.0.1:9100
import http from 'node:http';

const cache = new Map(); // ip -> { isAi, at }
const TTL = 60 * 60 * 1000;

async function isAi(ip) {
  const hit = cache.get(ip);
  if (hit && Date.now() - hit.at < TTL) return hit.isAi;
  const res = await fetch(`https://fast.ipapi.gr/api/lookup/${encodeURIComponent(ip)}?key=${process.env.IPAPI_KEY}`,
    { signal: AbortSignal.timeout(1500) });
  const data = await res.json();
  const result = res.ok && Boolean(data.risk?.is_ai);
  cache.set(ip, { isAi: result, at: Date.now() });
  return result;
}

http.createServer(async (req, res) => {
  const ip = req.headers['x-real-ip'];
  const ua = req.headers['x-original-ua'] || '';
  const claimsAi = /GPTBot|ClaudeBot|OAI-SearchBot|PerplexityBot|ChatGPT-User|Claude-User|Perplexity-User/i.test(ua);
  try {
    // Only claimed AI bots are checked; everyone else passes without a lookup.
    const ok = !claimsAi || await isAi(ip);
    res.writeHead(ok ? 204 : 403).end();
  } catch {
    res.writeHead(204).end(); // fail open: never take the site down because a lookup failed
  }
}).listen(9100, '127.0.0.1');

location = /_ai_gate {
    internal;
    proxy_pass http://127.0.0.1:9100;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Original-UA $http_user_agent;
}

location / {
    auth_request /_ai_gate;
    # ...
}

Only requests that claim to be AI bots cost a lookup, and results are cached per IP. With the free plan the response has no risk object, so this gate would reject every claimed AI bot. Use it with a Pro key, or switch it to log-only while you test. The response format is in the docs.

FAQ

Is blocking by user agent useless?

No. It's a cheap first filter and it stops honest bots that ignore robots.txt (rare). It just can't verify anything, so never use it to allow traffic.

Will this block Googlebot?

No. Googlebot isn't in the lists or the user-agent map above. Google's AI training opt-out is the Google-Extended robots.txt token, which is a control token, not a separate crawler, so there's nothing to block at the IP level.

How often do the lists change?

It varies by operator and file. When we checked in November 2026, OpenAI's ChatGPT-User list had been regenerated within the past week. Every few hours is a reasonable refresh rate.

What about AI crawlers that don't publish IPs?

You can't verify them by IP. A user-agent rule plus rate limiting is the practical option.

Summary

• Robots.txt is the documented opt-out. Nginx enforces it and catches impostors.
• Generate a geo include from the official JSON files. Refresh atomically, and never publish an empty list.
• Combine "what it claims" (user agent) with "where it comes from" (IP) in a map.
• Make sure $remote_addr is the real client IP.

Background reading: how to verify AI crawler IP addresses and OpenAI's four bots.