Question

Spiders/Bots problem slow down my site

  • Posted February 4, 2015

Hi

I have allot of Spiders/Bots that is hitting my wordpress blog most msn. How can i slow this down so they dont crawl my site that match?

Subscribe
Share

Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

I recommend changing your wordpress login page from wp-login.php to something different.

Then, in your .htaccess set your wp-login.php page to be blocked, like this:


#prevents access to wp-login
    <Files wp-login.php>
    order allow,deny
    deny from all
    </Files>

you can google to see different ways to change your login page. It is not too difficult, and will reduce the bots to almost zero.

You can use .htaccess file to block bots with some UserAgent. Add following lines after “RewriteEngine on” :

RewriteCond %{HTTP_USER_AGENT} Purebot [NC,OR] RewriteCond %{HTTP_USER_AGENT} DotBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} Ezooms [NC,OR] RewriteCond %{HTTP_USER_AGENT} Robot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ichiro [NC,OR] RewriteCond %{HTTP_USER_AGENT} VoilaBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ReGet [NC,OR] RewriteCond %{HTTP_USER_AGENT} Rankur [NC,OR] RewriteCond %{HTTP_USER_AGENT} cmsworldmap [NC,OR] RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR] RewriteCond %{HTTP_USER_AGENT} SurveyBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} RadioClicker [NC,OR] RewriteCond %{HTTP_USER_AGENT} Embedly [NC,OR] RewriteCond %{HTTP_USER_AGENT} TweetmemeBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} discobot [NC,OR] RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} SearchBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} LinkpadBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} bingbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} SISTRIX [NC,OR] RewriteCond %{HTTP_USER_AGENT} Flexum [NC,OR] RewriteCond %{HTTP_USER_AGENT} BLEXbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} siteexplorer [NC,OR] RewriteCond %{HTTP_USER_AGENT} kmbot RewriteRule ^.* - [F,L]