Block AI crawlers in Apache and HAProxy by verified IP (tested configs)

Most Apache advice for AI crawlers is a RewriteCond %{HTTP_USER_AGENT} block in .htaccess. Most HAProxy advice is the same idea with an ACL on the user agent. Both block the crawlers that honestly identify themselves, which also honour robots.txt, and do nothing about scrapers that borrow their names.

This guide builds a policy that most sites actually want:

• Training crawlers (GPTBot, ClaudeBot, CCBot): blocked.
• AI search bots and user fetchers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User): allowed, but only from their operator's published addresses. A request that claims one of these names from anywhere else is an impostor and gets a 403.
• /robots.txt: readable by everyone, including blocked bots.

The lists come from the operators' own files, regenerated by a script. We ran both configurations on Apache 2.4.58 and HAProxy 2.8.30 against real addresses from those files. The Nginx version of the same idea is a separate guide.

Step 0: keep robots.txt

Robots.txt is the opt-out that OpenAI, Anthropic and Common Crawl document for their training crawlers. A server-level block adds enforcement against impostors, but keep the robots.txt rules, and keep the file reachable. Anthropic warns that blocking its IPs "may not work correctly or persistently guarantee an opt-out", because a blocked bot can't read your robots.txt. The list of tokens is in our AI crawler list.

Step 1: generate the IP files

One script feeds both servers. It downloads the official files, writes one plain list per operator for HAProxy and one include file for Apache, and replaces files atomically so a server never reads half a file. It uses only the Python standard library.

#!/usr/bin/env python3
"""Generate Apache and HAProxy files from the official AI crawler IP lists.
Usage: python3 gen_ai_lists.py /etc/ai-crawlers   (writes into that directory)"""
import ipaddress, json, os, re, sys, tempfile, urllib.request

# list name -> official file (checked 12 Nov 2026)
SOURCES = {
    "gptbot": "https://openai.com/gptbot.json",
    "oai_searchbot": "https://openai.com/searchbot.json",
    "chatgpt_user": "https://openai.com/chatgpt-user.json",
    "anthropic": "https://claude.com/crawling/bots.json",
    "perplexitybot": "https://www.perplexity.ai/perplexitybot.json",
    "perplexity_user": "https://www.perplexity.ai/perplexity-user.json",
    "ccbot": "https://index.commoncrawl.org/ccbot.json",
}

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "ai-list-sync/1.0"})
    with urllib.request.urlopen(req, timeout=15) as r:
        text = r.read().decode("utf-8", "replace")
    if not text.lstrip().startswith("{"):  # Amazon: JSON inside an HTML page
        text = re.search(r"<code[^>]*>\s*(\{.*?\})\s*</code>", text, re.S).group(1)
    nets = []
    for p in json.loads(text).get("prefixes", []):
        v = p.get("ipv4Prefix") or p.get("ipv6Prefix") or p.get("ip_prefix")
        if v:
            nets.append(str(ipaddress.ip_network(v, strict=False)))
    if not nets:
        raise ValueError(f"{url}: empty list")
    return nets

def write_atomic(path, text):
    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path))
    with os.fdopen(fd, "w") as f:
        f.write(text)
    os.chmod(tmp, 0o644)
    os.replace(tmp, path)

def main(outdir):
    lists = {name: fetch(url) for name, url in SOURCES.items()}  # all or nothing
    os.makedirs(outdir, exist_ok=True)
    apache = ["# Generated by gen_ai_lists.py. Do not edit."]
    for name, nets in lists.items():
        write_atomic(os.path.join(outdir, f"{name}.lst"), "\n".join(nets) + "\n")  # HAProxy
        apache += [f"SetEnvIfExpr \"-R '{n}'\" ip_{name}" for n in nets]           # Apache
    write_atomic(os.path.join(outdir, "apache-ai-ips.conf"), "\n".join(apache) + "\n")
    print(", ".join(f"{k}: {len(v)}" for k, v in lists.items()))

if __name__ == "__main__":
    main(sys.argv[1])

Run on 12 Nov 2026:

$ sudo python3 gen_ai_lists.py /etc/ai-crawlers
gptbot: 18, oai_searchbot: 39, chatgpt_user: 237, anthropic: 14, perplexitybot: 13, perplexity_user: 10, ccbot: 5

Why one file per operator rather than one combined list? Because the lists overlap. On 12 Nov 2026, six of GPTBot's prefixes (for example 4.227.36.0/25 and 74.7.175.128/25) also appeared in OpenAI's OAI-SearchBot file. A single "which bot is this IP?" map would have to pick one answer. Separate lists let each rule ask its own question: "is this address in this bot's file?"

Apache 2.4

The Apache version uses three standard modules: mod_setenvif to label the request, mod_authz_core for the decision, and mod_authz_host's address matching through the -R expression operator, which the expression docs describe as "Same as %{REMOTE_ADDR} -ipmatch ..., but more efficient". The generated file contains one line per prefix, like this:

# Generated by gen_ai_lists.py. Do not edit.
SetEnvIfExpr "-R '132.196.86.0/24'" ip_gptbot
SetEnvIfExpr "-R '172.182.202.0/25'" ip_gptbot

The policy file, which you write once:

# /etc/apache2/conf-available/ai-crawlers.conf
# Which bot does the request claim to be? (user agent, case-insensitive)
SetEnvIfNoCase User-Agent "GPTBot"          claim_gptbot
SetEnvIfNoCase User-Agent "OAI-SearchBot"   claim_oai_searchbot
SetEnvIfNoCase User-Agent "ChatGPT-User"    claim_chatgpt_user
SetEnvIfNoCase User-Agent "ClaudeBot"       claim_claudebot
SetEnvIfNoCase User-Agent "Claude-SearchBot|Claude-User" claim_claude_other
SetEnvIfNoCase User-Agent "PerplexityBot"   claim_perplexitybot
SetEnvIfNoCase User-Agent "Perplexity-User" claim_perplexity_user
SetEnvIfNoCase User-Agent "CCBot"           claim_ccbot

# Which official list is the client address in? (generated)
Include /etc/ai-crawlers/apache-ai-ips.conf

<Location "/">
    <RequireAll>
        Require all granted
        # Training crawlers: blocked, real or fake
        Require not env claim_gptbot
        Require not env claim_claudebot
        Require not env claim_ccbot
        # Search and user fetchers: allowed only from their operator's addresses
        Require expr "-z reqenv('claim_oai_searchbot') || -n reqenv('ip_oai_searchbot')"
        Require expr "-z reqenv('claim_chatgpt_user') || -n reqenv('ip_chatgpt_user')"
        Require expr "-z reqenv('claim_claude_other') || -n reqenv('ip_anthropic')"
        Require expr "-z reqenv('claim_perplexitybot') || -n reqenv('ip_perplexitybot')"
        Require expr "-z reqenv('claim_perplexity_user') || -n reqenv('ip_perplexity_user')"
    </RequireAll>
</Location>

# Everyone may read robots.txt, including the bots blocked above
<Location "/robots.txt">
    Require all granted
</Location>

On Debian and Ubuntu, enable it with sudo a2enmod setenvif authz_core authz_host && sudo a2enconf ai-crawlers && sudo apachectl configtest && sudo systemctl reload apache2.

How the pieces fit:

• Require not env claim_gptbot fails the request when the variable is set. A negated Require can only deny, never grant: the mod_authz_core docs say it "may never independently authorize a request", which is why it sits inside <RequireAll> next to Require all granted.
• Each Require expr line reads: "either the request doesn't claim to be this bot, or the address is in this bot's list". -z is true for an empty string, -n for a non-empty one, and reqenv() reads the variables set above.
• The later <Location "/robots.txt"> section is merged after <Location "/">, so it overrides the policy for that one path.

Results on Apache 2.4.58 with real addresses from the official files (404 here means "allowed": the test server had no content at that path):

132.196.86.5      compatible; GPTBot/1.4         -> 403   real GPTBot, blocked by policy
132.196.86.5      compatible; GPTBot/1.4         -> 200   same bot, /robots.txt
203.0.113.9       compatible; OAI-SearchBot/1.4  -> 403   impostor
104.210.140.130   compatible; OAI-SearchBot/1.4  -> 404   real OAI-SearchBot, allowed
216.73.216.10     Claude-User/1.0                -> 404   real, allowed
203.0.113.9       Claude-SearchBot/1.0           -> 403   impostor
107.20.236.150    PerplexityBot/1.0              -> 404   real, allowed
198.51.100.7      Mozilla/5.0 Firefox/140.0      -> 404   not a bot, untouched
2600:1f28:365:8000::1  CCBot/2.0                 -> 403   real CCBot (IPv6), blocked

Behind a proxy or CDN

If Apache sits behind a load balancer or CDN, the TCP peer is the proxy, and every check above would see the proxy's address. Use mod_remoteip so Apache replaces the client address with the real one, and trust only your own proxies:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8

Never trust X-Forwarded-For from the internet at large. Anyone can send that header.

.htaccess-only hosting

Include and the server-level SetEnvIfExpr lines don't belong in .htaccess. On shared hosting you can still paste the generated SetEnvIfExpr lines and the <RequireAll> block into .htaccess if your host allows FileInfo and AuthConfig overrides, but you'll have to re-paste after every list change. That's a good moment to consider an API-based check instead (see the end of this guide).

HAProxy

HAProxy can load address patterns straight from a file with -f, and for IPv4 it may load them "into a binary tree, allowing very fast lookups" according to the 3.2 configuration manual. The generated .lst files are one prefix per line, which is exactly that format.

frontend www
    bind :80
    # bind :443 ssl crt /etc/haproxy/certs/   (your TLS setup)

    # Which bot does the request claim to be?
    acl claim_gptbot          req.hdr(user-agent) -m sub -i gptbot
    acl claim_claudebot       req.hdr(user-agent) -m sub -i claudebot
    acl claim_ccbot           req.hdr(user-agent) -m sub -i ccbot
    acl claim_oai_searchbot   req.hdr(user-agent) -m sub -i oai-searchbot
    acl claim_chatgpt_user    req.hdr(user-agent) -m sub -i chatgpt-user
    acl claim_claude_other    req.hdr(user-agent) -m sub -i claude-searchbot claude-user
    acl claim_perplexitybot   req.hdr(user-agent) -m sub -i perplexitybot
    acl claim_perplexity_user req.hdr(user-agent) -m sub -i perplexity-user

    # Is the client address in that operator's official list? (generated files)
    acl ip_oai_searchbot      src -f /etc/ai-crawlers/oai_searchbot.lst
    acl ip_chatgpt_user       src -f /etc/ai-crawlers/chatgpt_user.lst
    acl ip_anthropic          src -f /etc/ai-crawlers/anthropic.lst
    acl ip_perplexitybot      src -f /etc/ai-crawlers/perplexitybot.lst
    acl ip_perplexity_user    src -f /etc/ai-crawlers/perplexity_user.lst

    acl robots_txt path /robots.txt

    # Training crawlers: blocked, real or fake (robots.txt stays readable)
    http-request deny deny_status 403 if !robots_txt claim_gptbot or !robots_txt claim_claudebot or !robots_txt claim_ccbot
    # Search bots and user fetchers: only from their operator's addresses
    http-request deny deny_status 403 if claim_oai_searchbot !ip_oai_searchbot
    http-request deny deny_status 403 if claim_chatgpt_user !ip_chatgpt_user
    http-request deny deny_status 403 if claim_claude_other !ip_anthropic
    http-request deny deny_status 403 if claim_perplexitybot !ip_perplexitybot
    http-request deny deny_status 403 if claim_perplexity_user !ip_perplexity_user

    default_backend app

backend app
    server app1 127.0.0.1:8080

Notes:

• -m sub -i is a case-insensitive substring match, so GPTBot/1.4 and future versions both match. Several patterns on one line are alternatives.
• src is the address of the TCP peer. That's right when HAProxy is your edge. If something sits in front of HAProxy, either use the PROXY protocol (accept-proxy on the bind line) or, for a trusted CDN, set the source from its header with http-request set-src restricted to the CDN's own addresses.
• In HAProxy conditions, or separates alternatives and adjacent ACLs are combined with "and", so the first deny line reads "not robots.txt and claims GPTBot, or not robots.txt and claims ClaudeBot...".
• IPv4 and IPv6 prefixes can live in the same .lst file. CCBot's list includes an IPv6 /56.

Check and apply with haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxy. We ran this frontend on HAProxy 2.8.30 with the same addresses as the Apache test, and every request got the same answer as in the table above. (The only test-only change was a set-src line so one machine could play several client addresses.)

Step 2: keep the lists fresh

The operators update their files every few weeks. A daily or hourly cron job is plenty. The generator replaces nothing unless every download succeeds, and each server only reloads if its configuration still validates:

# /etc/cron.d/ai-crawlers
17 * * * * root python3 /usr/local/bin/gen_ai_lists.py /etc/ai-crawlers && apachectl configtest && apachectl graceful
19 * * * * root haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy

Use only the lines for the server you run. A graceful reload in both servers finishes in-flight requests before switching configuration.

Where this approach stops

• Bots without published IPs. Meta's crawlers publish user agents but no IP list on their crawler page. You can add a claim_ rule that blocks them by name, which is safe because blocking an impostor costs nothing. You can't add an IP-verified allow rule.
• Large lists in Apache. Each SetEnvIfExpr line is checked on every request. A few hundred prefixes is fine. Amazon's Amazonbot list alone was 1.292 single addresses on 12 Nov 2026; for lists that size, HAProxy's file-based ACLs or Nginx's geo module are a better fit.
• Undeclared scrapers. No published list covers scrapers that never say who they are. That needs rate limiting and challenges; see how to block AI scrapers.

Skip the list maintenance

If you don't want a sync job per server, the IPAPI.GR lookup returns risk.is_ai and risk.is_crawler for any address on any paid plan. IPAPI.GR's risk flags tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. They cover every AI, search and SEO crawler on the bot signals page, not just the seven files above. To check a single address from your Apache or HAProxy logs, use the free bot verification tool.

FAQ

Can I do this with mod_rewrite instead?

You can match the user agent with RewriteCond %{HTTP_USER_AGENT}, and RewriteCond expr "-R '132.196.86.0/24'" also works. But the Require approach above keeps access control in the authorization phase, where Apache's documentation puts it, and it's easier to read.

Why block GPTBot at all if it honours robots.txt?

You don't have to. The block rule mainly catches impostors that use GPTBot's name. If you prefer, change Require not env claim_gptbot into an expr rule like the search bots' rules, so only unverified GPTBot claims are refused, and let robots.txt handle the real one.

Does this affect Googlebot or Bingbot?

No. Neither user agent matches any rule here. To reject fake Googlebots too, see verifying Googlebot and Bingbot.