How to block website from crawling my server?

June 9, 2019 156 views
Linux Commands Ubuntu 18.04
saadsawash
By:
saadsawash

I have Ubuntu 18 on my vps with ufw enabled, i tried to block the ip adress from the netstat log but it is still accessing my site like the following :
root@Waseely:~# netstat -c | grep 3000
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 TIME_WAIT
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:46276 ESTABLISHED

I am trying to block the ip adress of the "crawl-66-249-64-3 but no luck

Any idea?

Log In to Comment
2 Answers
bobbyiliev June 10, 2019

This looks like a googlebot, have you tried disallowing google via your robots.txt file?

Reply
saadsawash June 10, 2019

I tried that , also tried to block the ip it self using
sudo ufw deny from 66.249.64.1/24

Reply
Have another answer? Share your knowledge.

Related Questions