Amazonbot, Applebot, Bytespider and Meta-ExternalAgent: who publishes IPs and how to verify each

OpenAI, Anthropic and Perplexity get most of the attention, and their crawler IP lists are well documented (OpenAI, Anthropic and Perplexity). Four other large crawlers cause more confusion: Amazonbot, Applebot, Bytespider and Meta's crawlers. Bot directories contradict each other about them. Some say Amazon publishes no ranges; some say Bytespider can't be verified; some give reverse DNS rules that the operators' own pages don't mention.

So we went to the source. For each operator we read its documentation, downloaded what it publishes, and ran the DNS checks ourselves on 12 Nov 2026. Here is what we found.

Summary

Amazon: three lists, wrapped in web pages

Amazon's Amazonbot documentation (last updated 26 Oct 2026) describes three crawlers:

• Amazonbot improves Amazon products and services, and "may be used to train Amazon AI models". It respects robots.txt.
• Amzn-SearchBot powers search experiences such as Alexa. Amazon says it "does not crawl content for generative AI model training". It also respects robots.txt.
• Amzn-User fetches live information for user requests, and "may not follow all robots.txt directives".

Amazon publishes an IP list for each: Amazonbot, Amzn-SearchBot and Amzn-User. So the directories that say "Amazon publishes no ranges" are wrong. The confusion is understandable, though, because the lists are unusual:

• They're HTML pages, not JSON files. The JSON (the same creationTime + prefixes format Google uses) sits inside a <code> element on the page. A tool that expects application/json will fail.
• The formats differ between lists. The Amazonbot and Amzn-User lists use ipv4Prefix with bare addresses ("3.81.245.78", no /32). The Amzn-SearchBot list uses a different key, ip_prefix, with /32 suffixes.
• They're large. On 12 Nov 2026: 1.292 addresses for Amazonbot, 816 for Amzn-SearchBot, 1.023 for Amzn-User, all single IPv4 addresses, with no overlap between the three. The Amazonbot and Amzn-SearchBot lists carried a creation time of 8 Sep 2026; the Amzn-User list, 4 Nov 2025.

Does reverse DNS work for Amazonbot?

Several bot directories tell you to check for a hostname ending in crawl.amazonbot.amazon. Amazon's current page doesn't mention reverse DNS at all. So we checked every address on the three lists:

• Amazonbot: 524 of 1.292 addresses had a PTR record like 3-91-138-117.crawl.amazonbot.amazon, and the 40 we tested resolved back to the same address. The other 768 had generic EC2 names like ec2-3-81-245-78.compute-1.amazonaws.com.
• Amzn-SearchBot and Amzn-User: every address had a generic EC2 name.

So a reverse DNS check would reject most of Amazon's own published crawler addresses. Use the lists. And don't treat "this address is on EC2" as evidence either way: the published addresses we checked were announced by Amazon's AS14618, a network that hosts countless other customers.

Apple: a file and reverse DNS, both reliable

Apple's Applebot page documents two ways to verify Applebot: reverse DNS in *.applebot.apple.com, and a JSON file of CIDR prefixes at applebot.json. On 12 Nov 2026 the file had 24 IPv4 /27 prefixes. Every address we sampled from it had a forward-confirmed name, for example 17.241.208.161 → 17-241-208-161.applebot.apple.com → 17.241.208.161.

The user agent contains Applebot/0.1; +http://www.apple.com/go/applebot. Apple says Applebot's data powers Spotlight, Siri and Safari, and "may also help train Apple foundation models".

Applebot-Extended is not a crawler. Apple is explicit: "Applebot-Extended does not crawl webpages." It's a robots.txt token that controls whether data Applebot collected may be used to train Apple's foundation models. You'll never see it in a log and there's nothing to verify:

User-agent: Applebot-Extended
Disallow: /

That rule opts out of training without affecting Applebot's search crawling. Blocking Applebot itself would remove you from Spotlight, Siri and Safari suggestions.

ByteDance Bytespider: subnets in a help article, and a DNS surprise

Toutiao (ByteDance's search engine) documents Bytespider on its webmaster help page, About Bytespider, in Chinese. The page gives:

• The user agent, which contains compatible; Bytespider; https://zhanzhang.toutiao.com/, in PC, Android and iOS variants.
• Ten /24 subnets the crawler uses: 110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24 and 60.8.151.0/24.
• A reverse DNS rule: Bytespider hostnames are named *.bytedance.com, and anything else is an impostor.

So "Bytespider publishes no IPs", which several directories say, isn't true. It's just not in English or in a machine-readable file.

Two things we found when we checked:

• The reverse names don't resolve forward. Addresses in every one of the ten subnets had PTR records such as bytespider-110-249-201-77.crawl.bytedance.com. But looking those names up returned NXDOMAIN from bytedance.com's authoritative servers. A strict forward-confirmed reverse DNS check, the standard way to verify Googlebot, therefore fails for Bytespider. A reverse-only check isn't safe either: whoever controls an IP block can set its PTR records to any name. Use the ten subnets.
• ASN checks won't help. The subnets we looked up were announced by China Unicom networks (AS4837 and AS4808), not by a ByteDance network.

What is Bytespider for? The page describes it as the crawler for Toutiao Search. Many articles call it ByteDance's AI training crawler; ByteDance's page says nothing about AI, and we don't claim more than it says. If you don't want it, robots.txt (User-agent: Bytespider) is the documented control, and the page also mentions a crawl-rate setting in Toutiao's webmaster tools.

Meta: user agents, no list

Meta's web crawler documentation lists five crawlers:

The page tells site owners to allow-list "either the user agent strings or the IP addresses (more secure)". But it doesn't publish the IP addresses, and it doesn't describe a reverse DNS rule. Some guides suggest checking whether the address belongs to Meta's network, AS32934 (registered to Facebook, Inc.). That's a heuristic Meta doesn't document, and it can't tell Meta-ExternalAgent from any other Meta service.

What you can do:

• Opt out of training with robots.txt. User-agent: Meta-ExternalAgent then Disallow: /. Meta says its crawlers may cache robots.txt for up to 24 hours.
• Block by user agent if you like. Blocking a fake Meta-ExternalAgent costs you nothing.
• Never allow by user agent. Anyone can send meta-externalagent/1.1.

Verifying all four in code

This script checks addresses against Amazon's three lists, Apple's file (with reverse DNS as a fallback), and Bytespider's ten subnets. It uses only the Python standard library. The general method, for any operator that publishes a JSON file, is in how to verify AI crawler IP addresses; this section covers the formats that don't fit it. Every other AI bot is in the AI crawler list.

#!/usr/bin/env python3
"""Verify Amazonbot, Amzn-SearchBot, Amzn-User, Applebot and Bytespider by IP.
Usage: python3 verify_bots.py 3.91.138.117 17.241.208.161 110.249.201.77 203.0.113.9"""
import ipaddress, json, re, socket, sys, urllib.request

LISTS = {  # official sources, checked 12 Nov 2026
    "Amazonbot": "https://developer.amazon.com/amazonbot/ip-addresses/",
    "Amzn-SearchBot": "https://developer.amazon.com/amazonbot/searchbot-ip-addresses/",
    "Amzn-User": "https://developer.amazon.com/amazonbot/live-ip-addresses/",
    "Applebot": "https://search.developer.apple.com/applebot.json",
}
# Toutiao publishes Bytespider's subnets as text on https://zhanzhang.toutiao.com/docs/intro/26899
BYTESPIDER = ["110.249.201.0/24", "110.249.202.0/24", "111.225.148.0/24", "111.225.149.0/24",
              "220.243.135.0/24", "220.243.136.0/24", "220.243.188.0/24", "220.243.189.0/24",
              "60.8.123.0/24", "60.8.151.0/24"]

def load(url):
    req = urllib.request.Request(url, headers={"User-Agent": "bot-verify/1.0"})
    with urllib.request.urlopen(req, timeout=15) as r:
        text = r.read().decode("utf-8", "replace")
    if not text.lstrip().startswith("{"):  # Amazon wraps the JSON in an HTML page
        text = re.search(r"<code[^>]*>\s*(\{.*?\})\s*</code>", text, re.S).group(1)
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix") or p.get("ip_prefix"), strict=False)
            for p in json.loads(text)["prefixes"]]

def fcrdns(ip, suffixes):
    """Forward-confirmed reverse DNS: PTR must end in a suffix and resolve back to ip."""
    try:
        host = socket.gethostbyaddr(ip)[0].rstrip(".")
        if not host.endswith(suffixes):
            return None
        addrs = {ai[4][0] for ai in socket.getaddrinfo(host, None)}
        return host if ip in addrs else None
    except OSError:
        return None

def main(ips):
    nets = {name: load(url) for name, url in LISTS.items()}
    nets["Bytespider"] = [ipaddress.ip_network(n) for n in BYTESPIDER]
    for ip in ips:
        addr = ipaddress.ip_address(ip)
        hits = [name for name, ns in nets.items() if any(addr in n for n in ns if n.version == addr.version)]
        if not hits:
            host = fcrdns(ip, (".applebot.apple.com",))  # Apple documents reverse DNS too
            hits = [f"Applebot (reverse DNS {host})"] if host else []
        print(f"{ip:18} {', '.join(hits) or 'not in any of these lists'}")

if __name__ == "__main__":
    main(sys.argv[1:])

Output on 12 Nov 2026, with one address from each source and a documentation address that nobody publishes:

$ python3 verify_bots.py 3.91.138.117 3.82.67.224 100.24.86.21 17.241.208.161 110.249.201.77 203.0.113.9
3.91.138.117       Amazonbot
3.82.67.224        Amzn-SearchBot
100.24.86.21       Amzn-User
17.241.208.161     Applebot
110.249.201.77     Bytespider
203.0.113.9        not in any of these lists

The Bytespider subnets are hard-coded because they're published as prose, not as a file. Re-check the page now and then; if ByteDance adds a subnet, the script won't notice on its own.

How IPAPI.GR classifies these four

IPAPI.GR's risk flags, included with every paid plan, tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. For these four operators:

• Amazonbot, Amzn-SearchBot and Amzn-User: is_ai and is_crawler. See Amazonbot, Amzn-SearchBot and Amzn-User.
• Applebot: is_ai and is_crawler. Apple says Applebot's data may help train its models, and it's also a search crawler. See Applebot.
• Bytespider: is_crawler, not is_ai, because ByteDance's page describes a search crawler and says nothing about AI. See Bytespider.
• Meta's crawlers: no flag. A user agent alone never sets one, so a request that only claims to be Meta's crawler stays false.

To check one address from your logs without writing code, use the free bot verification tool.

FAQ

Is Amazonbot an AI crawler?

Partly. Amazon says it improves Amazon's products and services and "may be used to train Amazon AI models". If you want to stay in Alexa answers without contributing training data, allow Amzn-SearchBot and disallow Amazonbot in robots.txt. Amazon also honours a noarchive robots meta tag as "no use for model training".

Can I block Bytespider by IP?

Yes, with the ten /24 subnets. Keep /robots.txt reachable so the crawler can still read your rules, and remember that a request claiming to be Bytespider from outside those subnets isn't Bytespider.

Why not use reverse DNS for everything?

Because operators don't all maintain it. On 12 Nov 2026 it worked perfectly for Applebot, for about four in ten Amazonbot addresses, for no Amzn-SearchBot or Amzn-User addresses, and never for Bytespider, whose reverse names don't resolve forward. Published lists were the reliable method for all three operators that publish anything.

Does Applebot-Extended appear in my logs?

No. It's only a robots.txt token. The crawling is done by Applebot.