Cloudflare WAF rules for AI crawlers: AI Crawl Control, verified bots and the operators' own IP lists

Many Cloudflare guides for AI crawlers still say "flip the Block AI bots switch". That switch is gone. On 15 September 2026 Cloudflare deprecated "Block AI Bots" in favour of three separate controls, and changed how its block settings treat crawlers like Googlebot and Applebot.

This guide covers three layers, from least to most work:

1. Cloudflare's built-in AI controls, and what they now do.
2. WAF custom rules on Cloudflare's own verified-bot fields.
3. WAF custom rules that check the operators' own published IP lists, kept up to date with a script.

Everything here was checked against Cloudflare's documentation and the operators' pages on 12 Nov 2026.

Layer 1: the built-in controls

Search, Training and Agent

Cloudflare now classifies AI bots by what they do. From its announcement:

• Search: "crawling to build a search index".
• Training: "crawling to train or fine-tune a model".
• Agent: "user-directed agents visiting a page on behalf of a human".

Each has its own setting in the dashboard's Security settings, and Cloudflare says the controls "are available to all customers, on all plans". Two details matter if you set these up before September:

• Your old setting was migrated. If you had the legacy block enabled, Cloudflare mapped it to Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads.
• "Block" now reaches mixed-use crawlers. Block and "Block on pages with ads" apply to crawlers that serve both search and AI, including Googlebot, Bingbot and Applebot. If what you want is "no training, keep search", pick Disallow AI Training, not Block. That option publishes the preference in your robots.txt (for Google through Google-Extended, for Apple through Applebot-Extended) and blocks training-only crawlers from Amazon, Anthropic, Meta and OpenAI.

AI Crawl Control

AI Crawl Control (formerly AI Audit) is available on all plans. It shows which AI services crawl you, lets you allow or block individual crawlers, and tracks robots.txt compliance. For most sites, this plus the Search/Training/Agent settings is the whole job.

When the built-in controls aren't enough

You'll want your own rules when:

• You want a bot Cloudflare doesn't classify the way you do. Cloudflare decides which bots count as Search, Training or Agent. Your view of, say, Amazonbot or ChatGPT-User may differ.
• You need an exact, per-operator decision. For example "allow OAI-SearchBot, block GPTBot, and block anything that claims to be either from the wrong network".
• You want the decision to rest on the operator's own published list, and to be able to show which file and prefix it came from.
• You run a bot Cloudflare doesn't know yet. New AI crawlers appear every few months.

Layer 2: rules on Cloudflare's verified-bot fields

Cloudflare exposes its own bot verification to WAF expressions through two fields:

• cf.client.bot, a boolean: the request "originated from a known good bot or crawler".
• cf.verified_bot_category, a string with the bot's type. The category list includes AI Crawler (crawls content used to train models), AI Assistant (driven by user action) and AI Search, which Cloudflare keeps for backward compatibility while new search crawlers move to the Search behaviour.

Two short rules cover a lot. Block every user agent that claims a well-known AI bot but isn't one Cloudflare has verified:

(lower(http.user_agent) contains "gptbot"
  or lower(http.user_agent) contains "claudebot"
  or lower(http.user_agent) contains "perplexitybot"
  or lower(http.user_agent) contains "oai-searchbot")
and not cf.client.bot

Block verified training crawlers by category:

cf.verified_bot_category eq "AI Crawler"

Plan notes: Cloudflare allows 5 custom rules on Free, 20 on Pro, 100 on Business and 1.000 on Enterprise (custom rules). Cloudflare's values reference limits the regex operator matches to Business and Enterprise, which is why the rules here chain contains with or instead of one regex. Check the fields reference for which bot fields your plan exposes.

The trade-off: with these fields, Cloudflare decides what is verified and which category it belongs to. That's a reasonable choice for many sites. If you want the operator's own list to be the source of truth, use layer 3.

Layer 3: rules on the operators' official IP lists

The AI operators that matter most publish their IP addresses (see our AI crawler list for who publishes what). Cloudflare WAF can match ip.src against those prefixes in two ways: inline in the expression, or through an IP list.

Option A: inline sets (small lists)

The expression language accepts an inline set of addresses and CIDR ranges, separated by spaces: ip.src in {192.0.2.0/24 2001:db8::/32} (values reference). The catch is the 4.096-character limit per expression. GPTBot (18 prefixes), OAI-SearchBot (39), Anthropic's shared file (14) and PerplexityBot (13 addresses) fit easily. ChatGPT-User's 237 prefixes don't.

This Node.js script (version 18 or later) builds a "spoofed AI bot" expression straight from the official files and refuses to print one that's too long. It uses the ranges.mjs loader shown below.

// cf-expression.mjs: print a Cloudflare WAF expression that blocks spoofed AI bot user agents.
// Usage: node cf-expression.mjs > expression.txt
import { fetchPrefixes } from './ranges.mjs';

const BOTS = {
  gptbot: 'https://openai.com/gptbot.json',
  'oai-searchbot': 'https://openai.com/searchbot.json',
  claudebot: 'https://claude.com/crawling/bots.json',
  'claude-searchbot': 'https://claude.com/crawling/bots.json',
  perplexitybot: 'https://www.perplexity.ai/perplexitybot.json',
};

const parts = [];
for (const [token, url] of Object.entries(BOTS)) {
  const cidrs = await fetchPrefixes(url);
  parts.push(`(lower(http.user_agent) contains "${token}" and not ip.src in {${cidrs.join(' ')}})`);
}
const expression = parts.join(' or ');
if (expression.length > 4096) {
  console.error(`Expression is ${expression.length} characters; the limit is 4,096. Use an IP list instead.`);
  process.exit(1);
}
console.log(expression);
console.error(`${expression.length} characters`);

Run on 12 Nov 2026, it produced a 1.947-character expression. The start of it:

(lower(http.user_agent) contains "gptbot" and not ip.src in {132.196.86.0/24 172.182.202.0/25 172.182.207.0/25 20.125.66.80/28 20.171.206.0/24 20.171.207.0/24 4.227.36.0/25 52.230.152.0/24 74.7.175.128/25 74.7.227.0/25 ...}) or (lower(http.user_agent) contains "oai-searchbot" and not ip.src in {...})

Paste it into a custom rule with the Block action. The downside is obvious: when OpenAI adds a range, your rule is stale until you regenerate it. Option B fixes that.

Option B: an IP list kept in sync by the API

Cloudflare IP lists are available on every plan. Free gets 1 custom list, Pro and Business 10, and the item limit is 10.000 across all lists (500.000 on Enterprise). Items can be IPv4 CIDRs from /8 to /32 and IPv6 from /12 to /128, including single addresses. You reference a list in an expression as ip.src in $list_name.

Because Free has one list, put the bots you want to allow in it: AI search and user-triggered fetchers. Create it once in the dashboard's account-level Lists section, name it ai_search_bots (list names can use only lowercase letters, numbers and underscores), and note its ID. Then this script replaces its contents from the official files. It uses the "Update all list items" endpoint, PUT /accounts/{account_id}/rules/lists/{list_id}/items, which removes all items and adds the new ones as an asynchronous bulk operation, then polls GET /accounts/{account_id}/rules/lists/bulk_operations/{operation_id} until it reports completed or failed.

Both Cloudflare scripts in this article import the same helper, ranges.mjs. Its only job is to turn an operator's publication into a flat array of CIDR strings that Cloudflare's list API accepts. Single addresses get an explicit /32 or /128, because list items are CIDRs. It throws on an empty result, so a failed download can never wipe your Cloudflare list:

// ranges.mjs: download the official AI crawler IP files. Node.js 18+ (built-in fetch).
// Most files use Google's format: { creationTime, prefixes: [{ ipv4Prefix | ipv6Prefix }] }.
// Amazon serves the same JSON inside an HTML page, uses bare addresses,
// and one of its lists uses the key "ip_prefix".
export async function fetchPrefixes(url) {
  const res = await fetch(url, { signal: AbortSignal.timeout(15_000) });
  if (!res.ok) throw new Error(`${url}: HTTP ${res.status}`);
  let text = await res.text();
  if (!text.trimStart().startsWith('{')) {
    const m = text.match(/<code[^>]*>\s*(\{[\s\S]*?\})\s*<\/code>/);
    if (!m) throw new Error(`${url}: no JSON found`);
    text = m[1];
  }
  const data = JSON.parse(text);
  const out = [];
  for (const p of data.prefixes ?? []) {
    const v = p.ipv4Prefix ?? p.ipv6Prefix ?? p.ip_prefix;
    if (typeof v !== 'string') continue;
    out.push(v.includes('/') ? v : `${v}/${v.includes(':') ? 128 : 32}`);
  }
  if (out.length === 0) throw new Error(`${url}: empty list`);
  return out;
}

With the helper in place, the sync script below is the part you schedule. It never edits the list item by item: one PUT replaces the whole list, so a stale prefix can't linger after the operator drops it.

// cf-sync-list.mjs: replace a Cloudflare IP list with the official AI crawler ranges.
// Env: CF_API_TOKEN (permission: Account Filter Lists Edit), CF_ACCOUNT_ID, CF_LIST_ID
import { fetchPrefixes } from './ranges.mjs';

const API = 'https://api.cloudflare.com/client/v4';
const { CF_API_TOKEN, CF_ACCOUNT_ID, CF_LIST_ID } = process.env;
const SOURCES = {
  'OAI-SearchBot': 'https://openai.com/searchbot.json',
  'ChatGPT-User': 'https://openai.com/chatgpt-user.json',
  'Anthropic bots': 'https://claude.com/crawling/bots.json',
  PerplexityBot: 'https://www.perplexity.ai/perplexitybot.json',
  'Perplexity-User': 'https://www.perplexity.ai/perplexity-user.json',
};

async function cf(method, path, body) {
  const res = await fetch(`${API}/accounts/${CF_ACCOUNT_ID}${path}`, {
    method,
    headers: { Authorization: `Bearer ${CF_API_TOKEN}`, 'Content-Type': 'application/json' },
    body: body && JSON.stringify(body),
  });
  const json = await res.json();
  if (!json.success) throw new Error(`${method} ${path}: ${JSON.stringify(json.errors)}`);
  return json.result;
}

// 1. Build the new contents. Abort on any failed download: never upload a partial list.
const items = [];
const seen = new Set();
for (const [name, url] of Object.entries(SOURCES)) {
  for (const ip of await fetchPrefixes(url)) {
    if (seen.has(ip)) continue;
    seen.add(ip);
    items.push({ ip, comment: name });
  }
}
console.log(`uploading ${items.length} prefixes`);

// 2. Replace all items (asynchronous bulk operation), then wait for it to finish.
const { operation_id } = await cf('PUT', `/rules/lists/${CF_LIST_ID}/items`, items);
for (let i = 0; i < 30; i++) {
  const op = await cf('GET', `/rules/lists/bulk_operations/${operation_id}`);
  if (op.status === 'completed') { console.log('list updated'); process.exit(0); }
  if (op.status === 'failed') throw new Error(`bulk operation failed: ${op.error}`);
  await new Promise((r) => setTimeout(r, 2000));
}
throw new Error('bulk operation still pending after 60 s');

On 12 Nov 2026 the five files added up to 313 prefixes, far inside the 10.000-item budget. Cloudflare allows one pending bulk operation per account at a time, so run the sync from a single scheduler, for example hourly from cron or a scheduled CI job:

# crontab -e
17 * * * * cd /opt/cf-ai-sync && CF_API_TOKEN=... CF_ACCOUNT_ID=... CF_LIST_ID=... node cf-sync-list.mjs >> sync.log 2>&1

The script throws before uploading if any download fails, so a temporary outage at one operator leaves the previous list in place instead of emptying it.

The rules

With the list in place, two custom rules give you a verified policy. Order matters: Cloudflare evaluates custom rules top to bottom, and documents that custom rules run before Super Bot Fight Mode, and that a terminating action stops the request before it reaches bot settings.

Rule 1: block impostors. Action: Block.

(lower(http.user_agent) contains "oai-searchbot"
  or lower(http.user_agent) contains "chatgpt-user"
  or lower(http.user_agent) contains "claude-searchbot"
  or lower(http.user_agent) contains "claude-user"
  or lower(http.user_agent) contains "perplexitybot"
  or lower(http.user_agent) contains "perplexity-user")
and not ip.src in $ai_search_bots

Rule 2: let the real ones through your other defences. Action: Skip, with "All remaining custom rules" and "All Super Bot Fight Mode rules" selected (skip options). Note that Cloudflare says the separate Bot Fight Mode can't be skipped.

ip.src in $ai_search_bots
and (lower(http.user_agent) contains "oai-searchbot"
  or lower(http.user_agent) contains "chatgpt-user"
  or lower(http.user_agent) contains "claude-searchbot"
  or lower(http.user_agent) contains "claude-user"
  or lower(http.user_agent) contains "perplexitybot"
  or lower(http.user_agent) contains "perplexity-user")

Requiring the user agent in rule 2 matters. Anthropic's single file covers ClaudeBot as well as Claude-SearchBot and Claude-User, so an IP match alone would also let the training crawler skip your defences.

For training crawlers, you don't need a list at all: leave Training on Disallow AI Training (or block them in robots.txt), and add a rule like lower(http.user_agent) contains "gptbot" with Block if you want a hard stop. A real GPTBot that's blocked costs you nothing; a fake one is blocked too.

Test before you enforce

Start each rule with a non-blocking action (Log, if your plan offers it, or Managed Challenge on a test path) and watch Security Events for a day. Then check:

• Security Events shows rule 1 matching only requests whose IPs aren't in the operators' files. You can confirm any single address with the free bot verification tool.
• Real OAI-SearchBot and PerplexityBot requests show up as allowed. If you see none, the bots may simply not have visited yet.
• /robots.txt stays reachable for everything, so blocked crawlers can still read your preferences.

Verified bots vs operator lists: which to trust?

Many sites end up using both: Cloudflare's controls for the broad policy, and operator lists for the few bots where an exact, auditable decision matters.

Outside Cloudflare: the same answer from an API

If part of your stack isn't behind Cloudflare (an origin API, a log pipeline, a job that scores past traffic), the IPAPI.GR lookup gives you the same answer as risk.is_ai and risk.is_crawler on any paid plan. IPAPI.GR's risk flags tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. The bot signals page lists every bot those flags cover, with a page per bot such as OAI-SearchBot or PerplexityBot.

FAQ

Is "Block AI bots" still available?

Cloudflare deprecated it on 15 September 2026 and migrated existing settings to the Search, Training and Agent controls. Check your zone's Security settings to see what yours became.

Will blocking "AI Crawler" block Googlebot?

The legacy AI Crawler category is for crawlers whose content is used to train models. Googlebot is a search crawler. But the new Block setting for the Training control does reach mixed-use crawlers such as Googlebot, which is why Cloudflare points to Disallow AI Training if you want to stop training and keep search.

Can I put Amazonbot's list in a Cloudflare list?

Yes. The fetchPrefixes loader above handles Amazon's HTML page. Amazonbot alone was 1.292 single addresses on 12 Nov 2026, which fits in the 10.000-item budget but uses a good share of it. Since Amazon documents that Amazonbot respects robots.txt, a robots.txt rule may be all you need.

Why not match the full user-agent string?

Operators change version numbers (GPTBot is at GPTBot/1.4 today). Match the token, lowercased, with contains.