Spiders/Bots problem slow down my site

Posted February 4, 2015 4.7k views


I have allot of Spiders/Bots that is hitting my wordpress blog most msn. How can i slow this down so they dont crawl my site that match?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Submit an Answer
2 answers

You can use .htaccess file to block bots with some UserAgent.
Add following lines after “RewriteEngine on” :

RewriteCond %{HTTPUSERAGENT} Purebot [NC,OR]
RewriteCond %{HTTPUSERAGENT} DotBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} Ezooms [NC,OR]
RewriteCond %{HTTPUSERAGENT} Robot [NC,OR]
RewriteCond %{HTTPUSERAGENT} ichiro [NC,OR]
RewriteCond %{HTTPUSERAGENT} VoilaBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} ReGet [NC,OR]
RewriteCond %{HTTPUSERAGENT} Rankur [NC,OR]
RewriteCond %{HTTPUSERAGENT} cmsworldmap [NC,OR]
RewriteCond %{HTTPUSERAGENT} libwww-perl [NC,OR]
RewriteCond %{HTTPUSERAGENT} SurveyBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} RadioClicker [NC,OR]
RewriteCond %{HTTPUSERAGENT} Embedly [NC,OR]
RewriteCond %{HTTPUSERAGENT} TweetmemeBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} discobot [NC,OR]
RewriteCond %{HTTPUSERAGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} SearchBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} LinkpadBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} bingbot [NC,OR]
RewriteCond %{HTTPUSERAGENT} Flexum [NC,OR]
RewriteCond %{HTTPUSERAGENT} siteexplorer [NC,OR]
RewriteCond %{HTTPUSERAGENT} kmbot
RewriteRule ^.* - [F,L]

I recommend changing your wordpress login page from wp-login.php to something different.

Then, in your .htaccess set your wp-login.php page to be blocked, like this:

#prevents access to wp-login
    <Files wp-login.php>
    order allow,deny
    deny from all

you can google to see different ways to change your login page. It is not too difficult, and will reduce the bots to almost zero.