AWS WAF and AI crawlers: IP sets from the official lists vs Bot Control's CategoryAI

If your site runs behind CloudFront, an Application Load Balancer or API Gateway, AWS WAF is where your bot policy lives. For AI crawlers it offers two quite different tools:

• Bot Control, a managed rule group with a CategoryAI rule. Little setup, extra cost, and AWS decides what counts as an AI bot.
• Your own rules on IP sets, filled from the lists the AI operators publish. No extra fees beyond normal WAF usage, exact per-bot control, and you maintain the sync.

This guide covers both, with a script that keeps IP sets in sync with the official files. AWS facts come from the AWS WAF developer guide and the operator facts from each operator's own page, all checked on 12 Nov 2026.

Option 1: Bot Control and CategoryAI

The Bot Control rule group (AWSManagedRulesBotControlRuleSet) labels bot traffic and blocks what it can't verify. AWS notes that "you are charged additional fees when you use this managed rule group".

The rule that matters here is CategoryAI. Unlike the other category rules, which act only on unverified bots, AWS documents that it "applies the action to all matches, regardless of whether the bots are verified or unverified". Its default action is Block, and it adds the label awswaf:managed:aws:bot-control:bot:category:ai.

That makes it a blunt instrument: on its default setting it blocks AI search bots and user-triggered fetchers along with training crawlers. If that's what you want, you're done. If you want to keep AI search, the usual pattern is to override CategoryAI's action to Count, then write your own rules after the rule group that act on its labels (AWS documents that the labels are available to rules that run after the group):

• awswaf:managed:aws:bot-control:bot:category:ai for any AI bot,
• awswaf:managed:aws:bot-control:bot:verified for bots Bot Control could verify,
• awswaf:managed:aws:bot-control:bot:name:<name> for a specific bot. Check your WAF logs or the DescribeManagedRuleGroup API's AvailableLabels for the exact names; we don't reproduce them here because AWS can change them.

One detail from AWS's notes: Bot Control uses the request's origin IP to decide whether a bot is verified, and you can't point it at a forwarded-IP header. If verified bots reach WAF through a proxy, AWS suggests a rule before Bot Control that uses the forwarded address to allow them explicitly. The IP sets below do exactly that.

Option 2: IP sets from the operators' lists

Most AI operators publish their addresses (our AI crawler list says who does and where). AWS WAF can match them with IP sets. The limits that matter, from the quotas page:

Two more rules from the IP set guide: a set holds either IPv4 or IPv6 addresses, chosen at creation, and single addresses must be written as CIDR (192.0.2.44/32). Sets for CloudFront must be created with scope CLOUDFRONT in us-east-1. Sets for regional resources (ALB, API Gateway, App Runner and others) use scope REGIONAL in the resource's Region.

The sync script

This script downloads each official file, splits it into IPv4 and IPv6, and creates or updates one IP set per operator and address family. It refuses to touch anything if any download fails, and skips the update when nothing changed. It needs boto3 and the IAM permissions listed in the docstring.

#!/usr/bin/env python3
"""Keep AWS WAF IP sets in sync with the official AI crawler IP lists.
Usage: python3 sync_waf_ipsets.py REGIONAL eu-west-1     (or: CLOUDFRONT us-east-1)
Needs: boto3, and IAM permissions wafv2:ListIPSets, wafv2:CreateIPSet, wafv2:GetIPSet, wafv2:UpdateIPSet."""
import ipaddress, json, re, sys, urllib.request
import boto3

# IP set base name -> official file (checked 12 Nov 2026)
SOURCES = {
    "ai-oai-searchbot": "https://openai.com/searchbot.json",
    "ai-chatgpt-user": "https://openai.com/chatgpt-user.json",
    "ai-anthropic": "https://claude.com/crawling/bots.json",
    "ai-perplexitybot": "https://www.perplexity.ai/perplexitybot.json",
    "ai-perplexity-user": "https://www.perplexity.ai/perplexity-user.json",
    "ai-ccbot": "https://index.commoncrawl.org/ccbot.json",
}
MAX_ADDRESSES = 10_000  # AWS WAF quota per IP set

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "waf-ipset-sync/1.0"})
    with urllib.request.urlopen(req, timeout=15) as r:
        text = r.read().decode("utf-8", "replace")
    if not text.lstrip().startswith("{"):  # Amazon: JSON inside an HTML page
        text = re.search(r"<code[^>]*>\s*(\{.*?\})\s*</code>", text, re.S).group(1)
    nets = {4: [], 6: []}
    for p in json.loads(text).get("prefixes", []):
        v = p.get("ipv4Prefix") or p.get("ipv6Prefix") or p.get("ip_prefix")
        if v:
            n = ipaddress.ip_network(v, strict=False)  # always CIDR, e.g. 1.2.3.4/32
            nets[n.version].append(str(n))
    if not nets[4] and not nets[6]:
        raise ValueError(f"{url}: empty list")
    return nets

def existing_sets(waf, scope):
    sets, marker = {}, None
    while True:
        kw = {"Scope": scope, "Limit": 100, **({"NextMarker": marker} if marker else {})}
        page = waf.list_ip_sets(**kw)
        for s in page.get("IPSets", []):
            sets[s["Name"]] = s
        marker = page.get("NextMarker")
        if not marker or not page.get("IPSets"):
            return sets

def main(scope, region):
    waf = boto3.client("wafv2", region_name=region)
    wanted = {}
    for base, url in SOURCES.items():  # download everything first: all or nothing
        nets = fetch(url)
        for version in (4, 6):
            if nets[version]:
                if len(nets[version]) > MAX_ADDRESSES:
                    raise ValueError(f"{base}: {len(nets[version])} prefixes exceed the IP set quota")
                wanted[f"{base}-v{version}"] = ("IPV4" if version == 4 else "IPV6", sorted(nets[version]))

    have = existing_sets(waf, scope)
    for name, (ip_version, addresses) in wanted.items():
        if name not in have:
            arn = waf.create_ip_set(Name=name, Scope=scope, IPAddressVersion=ip_version,
                                    Addresses=addresses, Description="Official AI crawler ranges")["Summary"]["ARN"]
            print(f"created {name} ({len(addresses)}) {arn}")
            continue
        current = waf.get_ip_set(Name=name, Scope=scope, Id=have[name]["Id"])
        if sorted(current["IPSet"]["Addresses"]) == addresses:
            print(f"unchanged {name} ({len(addresses)})")
            continue
        waf.update_ip_set(Name=name, Scope=scope, Id=have[name]["Id"], Addresses=addresses,
                          LockToken=current["LockToken"], Description="Official AI crawler ranges")
        print(f"updated {name} ({len(addresses)})")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

UpdateIPSet replaces the whole address list and needs the lock token from a fresh GetIPSet. If someone else changed the set in between, AWS rejects the call instead of overwriting their change, and the next run picks it up.

We ran the script against the live operator files and a stubbed WAFV2 client (botocore's Stubber, which validates every call against the WAFV2 API model). On 12 Nov 2026 it produced seven sets: ai-oai-searchbot-v4 (39 prefixes), ai-chatgpt-user-v4 (237), ai-anthropic-v4 (14), ai-perplexitybot-v4 (13), ai-perplexity-user-v4 (10), ai-ccbot-v4 (4) and ai-ccbot-v6 (1).

Run it on a schedule, for example from cron or an EventBridge-scheduled job:

# hourly, regional web ACL in eu-west-1
23 * * * * python3 /opt/waf/sync_waf_ipsets.py REGIONAL eu-west-1 >> /var/log/waf-ipset-sync.log 2>&1

The rules

With the sets in place, two kinds of rule cover most policies. Both were validated with an UpdateWebACL call against the same API model. Replace the example ARNs with the ones the script prints.

Block impostors. A request whose user agent says OAI-SearchBot but whose source address isn't in OpenAI's searchbot.json. Repeat for each bot you allow, or combine them with an OrStatement.

{
  "Name": "block-unverified-oai-searchbot",
  "Priority": 10,
  "Action": { "Block": {} },
  "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "unverified-oai-searchbot" },
  "Statement": {
    "AndStatement": {
      "Statements": [
        {
          "ByteMatchStatement": {
            "FieldToMatch": { "SingleHeader": { "Name": "user-agent" } },
            "PositionalConstraint": "CONTAINS",
            "SearchString": "oai-searchbot",
            "TextTransformations": [{ "Priority": 0, "Type": "LOWERCASE" }]
          }
        },
        {
          "NotStatement": {
            "Statement": {
              "IPSetReferenceStatement": { "ARN": "arn:aws:wafv2:eu-west-1:111122223333:regional/ipset/ai-oai-searchbot-v4/EXAMPLE-ID" }
            }
          }
        }
      ]
    }
  }
}

Block a training crawler by address, whatever user agent it sends. CCBot publishes both IPv4 and IPv6, so the rule checks both sets:

{
  "Name": "block-ccbot-by-ip",
  "Priority": 20,
  "Action": { "Block": {} },
  "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "ccbot-by-ip" },
  "Statement": {
    "OrStatement": {
      "Statements": [
        { "IPSetReferenceStatement": { "ARN": "arn:aws:wafv2:eu-west-1:111122223333:regional/ipset/ai-ccbot-v4/EXAMPLE-ID" } },
        { "IPSetReferenceStatement": { "ARN": "arn:aws:wafv2:eu-west-1:111122223333:regional/ipset/ai-ccbot-v6/EXAMPLE-ID" } }
      ]
    }
  }
}

A few practical notes:

• Lowercase before matching. The LOWERCASE text transformation lets one search string match OAI-SearchBot/1.4 and any future casing. Match the bot's token, not the full user agent: operators change version numbers.
• Anthropic's file is shared. ClaudeBot, Claude-SearchBot and Claude-User all use the addresses in bots.json. If you allow Claude-SearchBot by IP set, also require its user agent, or the training crawler gets the same pass.
• Behind another proxy? If WAF sees a proxy's address rather than the client's, add "IPSetForwardedIPConfig": { "HeaderName": "X-Forwarded-For", "FallbackBehavior": "NO_MATCH", "Position": "FIRST" } to the IP set reference. Only do this when you control the proxy that sets the header; otherwise any client can choose its own "address".
• Keep robots.txt reachable. Add a higher-priority rule that allows /robots.txt (a ByteMatchStatement on UriPath with EXACTLY), so blocked crawlers can still read your opt-out. Anthropic warns that IP blocks alone "may not work correctly or persistently guarantee an opt-out".

Combining both

If you already pay for Bot Control, the two work well together:

1. Your IP-set rules for the bots you care about, at a lower priority number than the rule group, so they run first: an Allow rule for each verified AI search bot (user agent matches and source address is in its IP set, the opposite of the impostor rule above), plus the impostor Block rules.
2. Bot Control with CategoryAI left on Block, to catch the AI bots you didn't list.

Requests your earlier rules allow never reach CategoryAI, so the verified search bots you chose stay in, and everything else in the AI category is blocked.

Which should you use?

Checking addresses outside WAF

WAF logs and sampled requests show the client IP. To find out who an address really belongs to, paste it into the free bot verification tool. For application code, background jobs or log pipelines, the IPAPI.GR lookup API returns risk.is_ai and risk.is_crawler on any paid plan, without a sync job. IPAPI.GR's risk flags tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. The bot signals page lists every covered bot, for example Amazonbot, whose own published addresses are EC2 instances.

FAQ

Can I put Amazonbot in an IP set?

Yes. The script's fetch handles Amazon's HTML-wrapped list. On 12 Nov 2026 Amazonbot's list had 1.292 single addresses, well under the 10.000 per set. Amazon says Amazonbot respects robots.txt, so decide whether you need more than that.

Does blocking AWS IP ranges block AI crawlers?

It blocks some, and a lot else besides. Anthropic's published range 216.73.216.0/22 and PerplexityBot's addresses were announced by Amazon on 12 Nov 2026, and Amazonbot's addresses had EC2 hostnames. So do many legitimate services. Verify by operator list instead of blocking a whole cloud.

How quickly do updates apply?

UpdateIPSet changes take effect without touching the web ACL, so the next sync run is the only delay. The operators themselves change their files every few weeks.

Will this block Googlebot?

No. None of these rules match Googlebot. Google publishes its crawler ranges too, if you want to add a fake-Googlebot rule; see how to verify Googlebot.