Amazonbot, Applebot, Bytespider and Meta-ExternalAgent: who publishes IPs and how to verify each
Popular bot directories disagree about whether Amazonbot, Bytespider and Meta's crawlers can be verified by IP. We read each operator's page and checked thousands of addresses in DNS. The answers are more interesting than any of the directories say.