Blog
How to check whether a request really comes from GPTBot, Googlebot or a research scanner, and how to put IP geolocation to work in your code.
Anyone can send a request that calls itself GPTBot. The IP address is much harder to fake. Here is how to check a request against the IP ranges the AI companies publish themselves, by hand, in code, or with one API call.
Read articleOpenAI does not have one crawler. It has four, each with its own purpose, its own robots.txt behaviour and its own published IP file. Treating them as one "GPTBot" is how sites end up blocking the wrong thing.
Anthropic and Perplexity both publish the IP addresses of their bots, but the first search result for "ClaudeBot IP addresses" is a different list entirely. Here are the right files, what each bot does, and how to check a request against them.
Fake Googlebots are one of the oldest tricks on the web. Google and Bing both document how to catch them, but many guides still point at old URLs. Here are both methods, the current files, and code you can run.
Every server on the internet is probed within minutes of going online, and much of that traffic comes from a few dozen known scanning projects. Here is how to tell research scanners apart from attackers, where the operators document their addresses, and what to do about them.
The usual Nginx recipe for AI crawlers is a regex on the user agent. It blocks the honest bots and lets every impostor through. Here is a setup that checks the IP against the operators' own lists, generated automatically.
Looking up a country from an IP takes one HTTP request. Getting the right IP, handling IPv6 and private addresses, caching results and failing gracefully takes a little more. Working examples in four languages.
Most "best free IP geolocation API" lists repeat numbers that are years out of date. We read each provider's own pricing and documentation pages and quote what they say about the free tier, including the conditions that decide whether you can actually use it.
The country tells you where an address is. The autonomous system number tells you whose network it is, and that is often the more useful question. How to look up ASNs by hand and in code, and what to do with them.
Try it
Free keys include 1,000 lookups a day. No card needed.