I would like to block crawling tools such as adplexity.com and whatrunswhere.com . These crawlers seem to be pretty smart and keep indexing media placements (ads). As a ‘services’ they allow people to view the indexed ads and let people view the web pages they link to. The result is that people steal idea’s, concepts and even complete landing page code.
There are several services which claim to ‘cloak’ pages and redirect these bots to dummy pages. They charge a very high fee (200 dollars and up / month) and they refuse to say anything about they techniques or ways they work. Not a single thing.
I was wondering if anyone has any idea of how I could be able to identify those spy tools and then redirect them to a dummy web page, while letting regular users go through to a correct page. Are there any applications, techniques or other ways that I can accomplish this on my droplets ? Any tips would be much appreciated!
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
Click below to sign up and get $100 of credit to try our products over 60 days!