4 articles tagged “AI crawlers”.
Anyone can send a request that calls itself GPTBot. The IP address is much harder to fake. Here is how to check a request against the IP ranges the AI companies publish themselves, by hand, in code, or with one API call.
OpenAI does not have one crawler. It has four, each with its own purpose, its own robots.txt behaviour and its own published IP file. Treating them as one "GPTBot" is how sites end up blocking the wrong thing.
Anthropic and Perplexity both publish the IP addresses of their bots, but the first search result for "ClaudeBot IP addresses" is a different list entirely. Here are the right files, what each bot does, and how to check a request against them.
The usual Nginx recipe for AI crawlers is a regex on the user agent. It blocks the honest bots and lets every impostor through. Here is a setup that checks the IP against the operators' own lists, generated automatically.
Try it
Free keys include 1,000 lookups a day. No card needed.