Block or allow AI bots in Next.js: proxy.ts with verified IPs, and Vercel's AI Bots ruleset

If you search for how to block GPTBot in Next.js, you'll find the same snippet many times: read the user-agent header in middleware.ts, return a 403 if it contains a bot name. It works for bots that tell the truth. It has two gaps:

• It can't safely allow anything. If you want OAI-SearchBot or PerplexityBot in, so you're cited in AI answers, a user-agent allow rule admits anyone who types that name into a scraper.
• It's out of date. Next.js 16 renamed middleware to proxy. The old file still works for now but is deprecated.

This guide builds a proxy.ts that verifies AI bots against the IP lists their operators publish, with no network calls on the request path. Then it covers Vercel's managed rulesets, which may be all you need. Facts were checked against the Next.js, Vercel and operator documentation on 12 Nov 2026.

Before you start: robots.txt

For training crawlers, robots.txt is the opt-out the operators document. In the App Router you can generate it from app/robots.ts:

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: ['GPTBot', 'ClaudeBot', 'CCBot', 'Google-Extended', 'Applebot-Extended', 'Meta-ExternalAgent'], disallow: '/' },
      { userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'], allow: '/' },
      { userAgent: '*', allow: '/' },
    ],
  };
}

Google-Extended and Applebot-Extended aren't crawlers. They're tokens that tell Google and Apple not to use your content for model training, so robots.txt is the only place they can go. Our AI crawler list explains each token.

Keep /robots.txt reachable for every client, including bots you block. Anthropic notes that blocking its IPs "may not work correctly or persistently guarantee an opt-out", because a blocked bot can't read your robots.txt.

proxy.ts in Next.js 16

The essentials from the proxy reference:

• The file is proxy.ts (or .js) at the project root, or in src, next to app or pages.
• It exports one function, named proxy or as the default export.
• It defaults to the Node.js runtime in Next.js 16, and setting the runtime option in a proxy file throws an error. That matters here: we can use Node's built-in net.BlockList for CIDR matching.
• A codemod migrates old files: npx @next/codemod@canary middleware-to-proxy .

On Next.js 15.5, the same code works as middleware.ts with a function named middleware, opting into the Node.js runtime (stable since 15.5).

Also note that NextRequest has no ip property any more. Next.js 15 removed geo and ip "as these values are provided by your hosting provider". We'll read the IP from a header your platform sets.

Step 1: snapshot the official IP files at build time

The Next.js docs warn that proxy is "invoked separately of your render code" and that "you should not attempt relying on shared modules or globals". So instead of downloading lists while serving requests, we fetch them at build time and compile them into the bundle. Every deploy carries a fresh snapshot, and the proxy never waits on a network call.

The build script needs one helper: something that downloads an operator file and hands back CIDR strings. It runs in Node.js 18 or later during prebuild, never inside the proxy, so the 15-second timeout and the HTML parsing for Amazon's pages cost nothing at request time. If a file comes back empty, the helper throws and the build fails loudly instead of shipping a bundle with no ranges:

// scripts/ranges.mjs: download one official AI crawler IP file.
// Most use Google's format: { creationTime, prefixes: [{ ipv4Prefix | ipv6Prefix }] }.
// Amazon serves the same JSON inside an HTML page, uses bare addresses,
// and one of its lists uses the key "ip_prefix".
export async function fetchPrefixes(url) {
  const res = await fetch(url, { signal: AbortSignal.timeout(15_000) });
  if (!res.ok) throw new Error(`${url}: HTTP ${res.status}`);
  let text = await res.text();
  if (!text.trimStart().startsWith('{')) {
    const m = text.match(/<code[^>]*>\s*(\{[\s\S]*?\})\s*<\/code>/);
    if (!m) throw new Error(`${url}: no JSON found`);
    text = m[1];
  }
  const data = JSON.parse(text);
  const out = [];
  for (const p of data.prefixes ?? []) {
    const v = p.ipv4Prefix ?? p.ipv6Prefix ?? p.ip_prefix;
    if (typeof v !== 'string') continue;
    out.push(v.includes('/') ? v : `${v}/${v.includes(':') ? 128 : 32}`);
  }
  if (out.length === 0) throw new Error(`${url}: empty list`);
  return out;
}

Next, the build step. It calls the helper once per source and writes the results into a TypeScript module that the proxy imports like any other code:

// scripts/build-ai-ranges.mjs: snapshot the official AI crawler IP files before `next build`.
// package.json: "prebuild": "node scripts/build-ai-ranges.mjs"
import { existsSync } from 'node:fs';
import { writeFile } from 'node:fs/promises';
import { fetchPrefixes } from './ranges.mjs';

const OUT = new URL('../lib/ai-ranges.generated.ts', import.meta.url);
const SOURCES = {
  gptbot: 'https://openai.com/gptbot.json',
  'oai-searchbot': 'https://openai.com/searchbot.json',
  'chatgpt-user': 'https://openai.com/chatgpt-user.json',
  anthropic: 'https://claude.com/crawling/bots.json',
  perplexitybot: 'https://www.perplexity.ai/perplexitybot.json',
  'perplexity-user': 'https://www.perplexity.ai/perplexity-user.json',
  ccbot: 'https://index.commoncrawl.org/ccbot.json',
};

try {
  const ranges = {};
  for (const [name, url] of Object.entries(SOURCES)) ranges[name] = await fetchPrefixes(url);
  await writeFile(OUT, `// Generated by scripts/build-ai-ranges.mjs on ${new Date().toISOString()}. Do not edit.
export const AI_RANGES: Record<string, string[]> = ${JSON.stringify(ranges, null, 2)};
`);
  console.log(Object.entries(ranges).map(([k, v]) => `${k}: ${v.length}`).join(', '));
} catch (err) {
  if (!existsSync(OUT)) throw err; // first build: no snapshot to fall back to
  console.warn(`AI ranges not refreshed (${err.message}); building with the previous snapshot.`);
}

Add it to package.json so it runs before every build:

{
  "scripts": {
    "prebuild": "node scripts/build-ai-ranges.mjs",
    "build": "next build"
  }
}

On 12 Nov 2026 it printed:

gptbot: 18, oai-searchbot: 39, chatgpt-user: 237, anthropic: 14, perplexitybot: 13, perplexity-user: 10, ccbot: 5

Commit lib/ai-ranges.generated.ts. If an operator's file is unreachable during a build, the script keeps the previous snapshot instead of failing your deploy.

Step 2: the proxy

// proxy.ts (Next.js 16+; in Next.js 15.5 name the file middleware.ts and the function middleware)
import { NextResponse, type NextRequest } from 'next/server';
import { BlockList, isIPv6 } from 'node:net';
import { AI_RANGES } from './lib/ai-ranges.generated';

type Action = 'allow' | 'block';
// user-agent token -> which official list proves it -> your policy
const RULES: { token: string; list: string; action: Action }[] = [
  { token: 'gptbot', list: 'gptbot', action: 'block' },
  { token: 'oai-searchbot', list: 'oai-searchbot', action: 'allow' },
  { token: 'chatgpt-user', list: 'chatgpt-user', action: 'allow' },
  { token: 'claudebot', list: 'anthropic', action: 'block' },
  { token: 'claude-searchbot', list: 'anthropic', action: 'allow' },
  { token: 'claude-user', list: 'anthropic', action: 'allow' },
  { token: 'perplexitybot', list: 'perplexitybot', action: 'allow' },
  { token: 'perplexity-user', list: 'perplexity-user', action: 'allow' },
  { token: 'ccbot', list: 'ccbot', action: 'block' },
];

const LISTS = new Map<string, BlockList>();
for (const [name, cidrs] of Object.entries(AI_RANGES)) {
  const list = new BlockList();
  for (const cidr of cidrs) {
    const [net, bits] = cidr.split('/');
    list.addSubnet(net, Number(bits), isIPv6(net) ? 'ipv6' : 'ipv4');
  }
  LISTS.set(name, list);
}

// On Vercel, x-real-ip is set by the platform. Self-hosted, make your reverse proxy set it
// (Nginx: proxy_set_header X-Real-IP $remote_addr;) and never trust a client-supplied value.
function clientIp(request: NextRequest): string | null {
  const ip = request.headers.get('x-real-ip')?.trim();
  return ip ? ip.replace(/^::ffff:/, '') : null;
}

export function proxy(request: NextRequest) {
  const ua = (request.headers.get('user-agent') ?? '').toLowerCase();
  const rule = RULES.find((r) => ua.includes(r.token));
  if (!rule) return NextResponse.next();

  const ip = clientIp(request);
  const list = LISTS.get(rule.list);
  const verified = !!ip && !!list && list.check(ip, isIPv6(ip) ? 'ipv6' : 'ipv4');

  if (!verified) {
    return new NextResponse('Unverified crawler', { status: 403 });
  }
  if (rule.action === 'block') {
    return new NextResponse('Crawling not permitted', { status: 403 });
  }
  const res = NextResponse.next();
  res.headers.set('x-verified-bot', rule.token); // visible in logs; remove if you prefer
  return res;
}

export const config = {
  // Skip static assets and robots.txt, so blocked crawlers can still read your opt-out.
  matcher: ['/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)'],
};

What it does, request by request:

We ran exactly these cases with real addresses from the official files: 132.196.86.5 (in 132.196.86.0/24, gptbot.json), 104.210.140.130 (in 104.210.140.128/28, searchbot.json) and 216.73.216.10 (in 216.73.216.0/22, Anthropic's bots.json), plus 203.0.113.9, a documentation address that no operator lists.

Some details worth knowing:

• Order of RULES matters only for overlapping tokens. None of the tokens above contain each other, so the first match is the only match.
• Anthropic's file is shared by ClaudeBot, Claude-SearchBot and Claude-User. The IP proves "Anthropic"; the user agent then picks the bot. A ClaudeBot request from an Anthropic address is blocked by your policy, as intended.
• The matcher excludes robots.txt, so every crawler can still read your preferences. It also skips _next/static and _next/image, which the Next.js docs recommend excluding.
• x-real-ip must come from your platform. Vercel documents that it overwrites X-Forwarded-For "to prevent IP spoofing" and that x-real-ip is identical to it. On a self-hosted Node server behind Nginx, set the header in Nginx. Without a trusted proxy in front, any client could send its own x-real-ip.

On Vercel you can also use ipAddress(request) from the @vercel/functions package, which "returns the IP address of the request from the headers".

Keeping the snapshot fresh

The lists change every few weeks, not every hour. A scheduled redeploy once a day is plenty: for example a scheduled CI job that calls a Vercel deploy hook, or your normal deploy cadence if you ship daily. The generated file's header comment tells you when the snapshot was taken.

Vercel's built-in options

If you deploy on Vercel, look at its bot management before you write code. As of 12 Nov 2026:

• AI Bots managed ruleset, available on all plans. It "identifies and filters requests from known AI crawlers and bots", can log or deny them, and "the list of known AI bots is automatically maintained and updated by Vercel". It's inactive by default (labelled Allow in the dashboard).
• Bot Protection managed ruleset, also on all plans. It challenges non-browser traffic, catches requests "that falsely claim to be from a browser such as a curl request identifying as Chrome", and automatically excludes verified bots. Vercel notes it doesn't work well behind another reverse proxy such as Cloudflare.
• Verified bots. Vercel keeps a directory of known bots and verifies them by IP address, reverse DNS, and cryptographic Web Bot Auth signatures. You can write WAF custom rules that block some of them by user agent.

When to use which:

Vercel's directory makes its own classifications. For example it lists Applebot under ai_crawler and PerplexityBot under ai_assistant. If those match your intent, the managed ruleset is simpler. If you want "allow PerplexityBot, block GPTBot", you need per-bot rules.

Checking an address from your logs

When the proxy rejects something as "Unverified crawler", it's useful to know who it really was. Paste the IP into the free bot verification tool. It tells you whether the address belongs to a known bot, and to which one, so you can see when it isn't the bot the user agent claimed.

For server-side enrichment beyond the bots you configured, the IPAPI.GR lookup returns risk.is_ai and risk.is_crawler on any paid plan, along with the ASN and organization. IPAPI.GR's risk flags tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. Call it from a Route Handler or a background job, not from the proxy, so a lookup never delays a page.

FAQ

Do I need the Node.js runtime?

For this code, yes: it uses node:net. Next.js 16's proxy runs on Node.js by default. If you're on an older Next.js with Edge middleware, you'd need a pure-JavaScript CIDR matcher instead of BlockList.

Why not fetch the lists inside the proxy with a cache?

You can, but it puts a network call on the first request of every cold instance and relies on module state the Next.js docs tell you not to rely on. A build-time snapshot is deterministic and costs nothing at request time.

Will this block Googlebot?

No. Googlebot isn't in RULES, so it passes through. Use Google-Extended in robots.txt for Gemini training, and see how to verify Googlebot if you want to reject fake Googlebots too.

What about AI bots that don't publish IPs?

Meta's crawlers, for example, publish user agents but no IP list. For those, you can block by user agent (a fake Meta crawler blocked is no loss) but never allow by user agent. More in how to block AI scrapers.