Spiders/Bots problem slow down my site

February 4, 2015 1.6k views


I have allot of Spiders/Bots that is hitting my wordpress blog most msn. How can i slow this down so they dont crawl my site that match?

2 Answers

You can use .htaccess file to block bots with some UserAgent.
Add following lines after "RewriteEngine on" :

RewriteCond %{HTTPUSERAGENT} Purebot [NC,OR]
RewriteCond %{HTTPUSERAGENT} DotBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} Ezooms [NC,OR]
RewriteCond %{HTTPUSERAGENT} Robot [NC,OR]
RewriteCond %{HTTPUSERAGENT} ichiro [NC,OR]
RewriteCond %{HTTPUSERAGENT} VoilaBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} ReGet [NC,OR]
RewriteCond %{HTTPUSERAGENT} Rankur [NC,OR]
RewriteCond %{HTTPUSERAGENT} cmsworldmap [NC,OR]
RewriteCond %{HTTPUSERAGENT} libwww-perl [NC,OR]
RewriteCond %{HTTPUSERAGENT} SurveyBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} RadioClicker [NC,OR]
RewriteCond %{HTTPUSERAGENT} Embedly [NC,OR]
RewriteCond %{HTTPUSERAGENT} TweetmemeBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} discobot [NC,OR]
RewriteCond %{HTTPUSERAGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} SearchBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} LinkpadBot [NC,OR]
RewriteCond %{HTTPUSERAGENT} bingbot [NC,OR]
RewriteCond %{HTTPUSERAGENT} Flexum [NC,OR]
RewriteCond %{HTTPUSERAGENT} siteexplorer [NC,OR]
RewriteCond %{HTTPUSERAGENT} kmbot
RewriteRule ^.* - [F,L]

  • I dont wont to block them i wont them to visit my blog but not that frequently.

  • Some search engines understand "Crawl-delay" directive in file robots.txt


    User-agent: MSNBot
    Crawl-delay: 15

    15 seconds can be good enough.

I recommend changing your wordpress login page from wp-login.php to something different.

Then, in your .htaccess set your wp-login.php page to be blocked, like this:

#prevents access to wp-login
    <Files wp-login.php>
    order allow,deny
    deny from all

you can google to see different ways to change your login page. It is not too difficult, and will reduce the bots to almost zero.

Have another answer? Share your knowledge.