How to verify Googlebot and Bingbot: reverse DNS, IP range files and fake bot detection

Scrapers have pretended to be Googlebot for as long as Googlebot has existed. Sites give search engines generous treatment: no rate limits, no paywall, no CAPTCHA. So "Googlebot" is the most valuable user agent to fake. The defence is old and well documented: check that the address really belongs to the search engine.

There are two ways to do that, and both search engines document them. This guide covers both, with the current official URLs. Several popular tutorials still link pages Google has since moved.

Method 1: forward-confirmed reverse DNS

Forward-confirmed reverse DNS (FCrDNS) uses two lookups:

1. Reverse lookup. Ask DNS for the hostname of the IP (the PTR record).
2. Check the domain. The hostname must end in a domain the operator documents.
3. Forward lookup. Resolve that hostname back to addresses. The original IP must be among them.

Step 3 is the one people skip, and it is the important one. Whoever controls an IP block controls its reverse DNS, so an attacker can make their PTR record say crawl-1-2-3-4.googlebot.com. They cannot make Google's DNS resolve that name back to their address.

Googlebot

Google's guide Verify requests from Google crawlers and fetchers says the hostname must be in googlebot.com, google.com or googleusercontent.com, and gives this example:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Google also documents the hostname pattern for each type of crawler:

So a request that says "Googlebot" in its user agent should resolve to a googlebot.com name. A googleusercontent.com name is a user-triggered fetcher, not the search crawler.

Bingbot

Bing's How to verify Bingbot describes the same two-step check: the reverse lookup must end in search.msn.com, and the forward lookup must return the original IP. Bing also offers a web form, the Verify Bingbot tool, for one-off checks. It advises against running the DNS check on every request. Verify once and cache the result.

Other search crawlers that document reverse DNS

Method 2: the published IP range files

DNS lookups are slow and can fail. Both Google and Bing now also publish their crawler addresses as JSON. You download the file periodically and match locally:

A file can also stop being updated. When we checked on 12 November 2026, Google's common-crawlers.json was dated that same day, while Bing's bingbot.json still had a creationTime of 3 January 2024. That doesn't mean the file is wrong, but it is a good reason to use the DNS check as a fallback for Bingbot rather than relying on the file alone.

Code: FCrDNS in Node.js

This function does both lookups and compares addresses correctly for IPv4 and IPv6. It uses net.BlockList for the comparison, so 2001:db8::1 and 2001:0db8:0:0:0:0:0:1 count as the same address:

import { promises as dns } from 'node:dns';
import { BlockList, isIPv6 } from 'node:net';

const RULES = {
  Googlebot: ['googlebot.com', 'google.com', 'googleusercontent.com'],
  bingbot: ['search.msn.com'],
  Applebot: ['applebot.apple.com'],
  YandexBot: ['yandex.ru', 'yandex.net', 'yandex.com']
};

const cache = new Map(); // ip -> { host, at }
const TTL = 24 * 60 * 60 * 1000;

function sameAddress(a, b) {
  const type = isIPv6(a) ? 'ipv6' : 'ipv4';
  const list = new BlockList();
  list.addAddress(a, type);
  return list.check(b, type);
}

export async function verifyCrawler(ip, bot) {
  const key = `${bot}|${ip}`;
  const hit = cache.get(key);
  if (hit && Date.now() - hit.at < TTL) return hit.host;

  let verified = null;
  try {
    for (const name of await dns.reverse(ip)) {
      const host = name.toLowerCase().replace(/\.$/, '');
      if (!RULES[bot].some((d) => host === d || host.endsWith('.' + d))) continue;
      const addrs = await dns.lookup(host, { all: true });
      if (addrs.some((a) => sameAddress(ip, a.address))) { verified = host; break; }
    }
  } catch {
    verified = null; // NXDOMAIN, timeout, no PTR: not verified
  }
  cache.set(key, { host: verified, at: Date.now() });
  return verified;
}

Call it only for requests whose user agent claims one of those bots, and cache the result per IP, as Bing recommends. Don't run DNS lookups for every visitor.

Code: FCrDNS in Python

import ipaddress
import socket

def verify_crawler(ip, allowed_suffixes):
    try:
        host, _, _ = socket.gethostbyaddr(ip)
    except (socket.herror, socket.gaierror):
        return None
    host = host.lower().rstrip(".")
    if not any(host == s or host.endswith("." + s) for s in allowed_suffixes):
        return None
    try:
        infos = socket.getaddrinfo(host, None)
    except socket.gaierror:
        return None
    original = ipaddress.ip_address(ip)
    if any(ipaddress.ip_address(info[4][0]) == original for info in infos):
        return host
    return None

print(verify_crawler("66.249.66.1", ["googlebot.com", "google.com", "googleusercontent.com"]))

gethostbyaddr returns only the primary PTR name. That is enough for the search engines above, which publish one name per address.

Which method to use

Large sites often combine them: match against the files first, and fall back to DNS for a claimed crawler that isn't in any file.

Skip the plumbing

The risk.is_crawler flag in the IPAPI.GR lookup API does this for you. It covers search engines, SEO and site-audit tools and web archives, using both the published range files and forward-confirmed reverse DNS for operators that document hostnames (including Yandex, Baidu and Yahoo). The full source list is on the bot signals page.

curl "https://fast.ipapi.gr/api/lookup/66.249.66.1?key=YOUR_API_KEY"

On a Pro key the response contains:

"risk": { "is_ai": false, "is_crawler": true, "is_scanner": false }

Combine it with the user agent: user agent says Googlebot + is_crawler: true means verified. User agent says Googlebot + is_crawler: false means an impostor. The flag doesn't say which crawler matched, so keep the user agent for that. Details are in the docs, and the flags come with Pro.

FAQ

Is checking the hostname without the forward lookup enough?

No. Whoever controls an IP block controls its PTR records, and can make them say googlebot.com. Only the forward lookup proves the name belongs to Google.

Googlebot's IP failed verification. Should I panic?

First check that you're testing the real client IP and not your proxy's. Then check whether the user agent was actually Googlebot or another Google fetcher with a different hostname pattern (see the table above).

Does Google-Extended have its own IPs?

No. Google-Extended is a robots.txt control token for AI training use, not a separate crawler, so there's nothing to verify by IP.

What about Bing's AI features?

Microsoft's 2023 webmaster announcement explains how content crawled by Bingbot is used in Bing's chat features and how to control that with meta tags. There's no separate AI crawler to verify, so the Bingbot checks above apply.

Summary

• Never trust a crawler user agent on its own.
• FCrDNS needs both lookups: reverse, then forward back to the same IP.
• Google's current files live under developers.google.com/static/crawling/ipranges/. Bing publishes bingbot.json and documents search.msn.com hostnames.
• Cache verification results per IP. DNS on every request is slow and fragile.

The same approach works for AI crawlers, where most operators publish IP files instead of hostnames. See How to verify AI crawler IP addresses.