If you are using apache you can use your .htaccess file to do so after identifying the crawler.

This may help: http://sitebeginner.com/apache/blockcrawlerbots/

For NGINX this may help: https://www.knthost.com/nginx/blocking-bots-nginx

Thanks for the answer Jason … I had already looked into this technique. I believe that these crawlers change IP and User Agent very often, making it a cat and mouse game to catch them each and every time … I’m still looking for a ‘catch most’ solution that limits the amount of work that I have to put in …

Haha user agent. They spoof browsers with the user-agent header. If you want them blocked you can start by blocking any of the 50k residential IP proxies they’re using. This means signing up to all the residential IP services ($500/mo minimum) and harvesting their IPs without them knowing you’re sniffing. Google literally has a whole department dedicated to scanning them. Another option is a service like Udger which tries (tries) to keep track. Adplexity isn’t stupid. What’s funny is the number of trackers like Voluum, Binom, ThriveTracker, etc that say they detect bots. They don’t. Sign up for Kintura and use their Google reCAPTCHA v3 integration and you’ll see just how many bots are making it to your site. Welcome to the wild west, sir.