Report this

What is the reason for this report?

Spiders/Bots problem slow down my site

Posted on February 4, 2015

Hi

I have allot of Spiders/Bots that is hitting my wordpress blog most msn. How can i slow this down so they dont crawl my site that match?



This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

You can use .htaccess file to block bots with some UserAgent. Add following lines after “RewriteEngine on” :

RewriteCond %{HTTP_USER_AGENT} Purebot [NC,OR] RewriteCond %{HTTP_USER_AGENT} DotBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} Ezooms [NC,OR] RewriteCond %{HTTP_USER_AGENT} Robot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ichiro [NC,OR] RewriteCond %{HTTP_USER_AGENT} VoilaBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ReGet [NC,OR] RewriteCond %{HTTP_USER_AGENT} Rankur [NC,OR] RewriteCond %{HTTP_USER_AGENT} cmsworldmap [NC,OR] RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR] RewriteCond %{HTTP_USER_AGENT} SurveyBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} RadioClicker [NC,OR] RewriteCond %{HTTP_USER_AGENT} Embedly [NC,OR] RewriteCond %{HTTP_USER_AGENT} TweetmemeBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} discobot [NC,OR] RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} SearchBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} LinkpadBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} bingbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} SISTRIX [NC,OR] RewriteCond %{HTTP_USER_AGENT} Flexum [NC,OR] RewriteCond %{HTTP_USER_AGENT} BLEXbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} siteexplorer [NC,OR] RewriteCond %{HTTP_USER_AGENT} kmbot RewriteRule ^.* - [F,L]

I recommend changing your wordpress login page from wp-login.php to something different.

Then, in your .htaccess set your wp-login.php page to be blocked, like this:


#prevents access to wp-login
    <Files wp-login.php>
    order allow,deny
    deny from all
    </Files>

you can google to see different ways to change your login page. It is not too difficult, and will reduce the bots to almost zero.

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.