Block or allow AI crawlers in Nginx by verified IP, not just user agent
The usual Nginx recipe for AI crawlers is a regex on the user agent. It blocks the honest bots and lets every impostor through. Here is a setup that checks the IP against the operators' own lists, generated automatically.