How to block website from crawling my server?

June 9, 2019 156 views
Linux Commands Ubuntu 18.04

I have Ubuntu 18 on my vps with ufw enabled, i tried to block the ip adress from the netstat log but it is still accessing my site like the following :
root@Waseely:~# netstat -c | grep 3000
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 TIME_WAIT
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:46276 ESTABLISHED

I am trying to block the ip adress of the "crawl-66-249-64-3 but no luck

Any idea?

2 Answers

This looks like a googlebot, have you tried disallowing google via your robots.txt file?

I tried that , also tried to block the ip it self using
sudo ufw deny from 66.249.64.1/24

Have another answer? Share your knowledge.