5 articles tagged “Bot verification”.
Anyone can send a request that calls itself GPTBot. The IP address is much harder to fake. Here is how to check a request against the IP ranges the AI companies publish themselves, by hand, in code, or with one API call.
OpenAI does not have one crawler. It has four, each with its own purpose, its own robots.txt behaviour and its own published IP file. Treating them as one "GPTBot" is how sites end up blocking the wrong thing.
Anthropic and Perplexity both publish the IP addresses of their bots, but the first search result for "ClaudeBot IP addresses" is a different list entirely. Here are the right files, what each bot does, and how to check a request against them.
Fake Googlebots are one of the oldest tricks on the web. Google and Bing both document how to catch them, but many guides still point at old URLs. Here are both methods, the current files, and code you can run.
Every server on the internet is probed within minutes of going online, and much of that traffic comes from a few dozen known scanning projects. Here is how to tell research scanners apart from attackers, where the operators document their addresses, and what to do about them.
Try it
Free keys include 1,000 lookups a day. No card needed.