Question
How to block website from crawling my server?
I have Ubuntu 18 on my vps with ufw enabled, i tried to block the ip adress from the netstat log but it is still accessing my site like the following :
root@Waseely:~# netstat -c | grep 3000
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 TIME_WAIT
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:46276 ESTABLISHED
I am trying to block the ip adress of the “crawl-66-249-64-3 but no luck
Any idea?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.