Related

How do i set my display to localhost using Ubantu VM ? Question Question
need recovery iso Question Question

Question

How to block website from crawling my server?

Posted June 9, 2019 1.1k views
Linux CommandsUbuntu 18.04

I have Ubuntu 18 on my vps with ufw enabled, i tried to block the ip adress from the netstat log but it is still accessing my site like the following :
root@Waseely:~# netstat -c | grep 3000
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:36797 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:57123 ESTABLISHED
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:38822 TIME_WAIT
tcp6 0 0 142.93.23.138:3000 crawl-66-249-64-3:46276 ESTABLISHED

I am trying to block the ip adress of the “crawl-66-249-64-3 but no luck

Any idea?

Add a comment
saadsawash
By:
saadsawash

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
2 answers
bobbyiliev June 10, 2019

This looks like a googlebot, have you tried disallowing google via your robots.txt file?

Reply Report
saadsawash June 10, 2019

I tried that , also tried to block the ip it self using
sudo ufw deny from 66.249.64.1/24

Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help