Hi
I have allot of Spiders/Bots that is hitting my wordpress blog most msn. How can i slow this down so they dont crawl my site that match?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
You can use .htaccess file to block bots with some UserAgent. Add following lines after “RewriteEngine on” :
RewriteCond %{HTTP_USER_AGENT} Purebot [NC,OR] RewriteCond %{HTTP_USER_AGENT} DotBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} Ezooms [NC,OR] RewriteCond %{HTTP_USER_AGENT} Robot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ichiro [NC,OR] RewriteCond %{HTTP_USER_AGENT} VoilaBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} ReGet [NC,OR] RewriteCond %{HTTP_USER_AGENT} Rankur [NC,OR] RewriteCond %{HTTP_USER_AGENT} cmsworldmap [NC,OR] RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR] RewriteCond %{HTTP_USER_AGENT} SurveyBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} RadioClicker [NC,OR] RewriteCond %{HTTP_USER_AGENT} Embedly [NC,OR] RewriteCond %{HTTP_USER_AGENT} TweetmemeBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} discobot [NC,OR] RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} SearchBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} LinkpadBot [NC,OR] RewriteCond %{HTTP_USER_AGENT} bingbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} SISTRIX [NC,OR] RewriteCond %{HTTP_USER_AGENT} Flexum [NC,OR] RewriteCond %{HTTP_USER_AGENT} BLEXbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} siteexplorer [NC,OR] RewriteCond %{HTTP_USER_AGENT} kmbot RewriteRule ^.* - [F,L]
I recommend changing your wordpress login page from wp-login.php to something different.
Then, in your .htaccess set your wp-login.php page to be blocked, like this:
#prevents access to wp-login
<Files wp-login.php>
order allow,deny
deny from all
</Files>
you can google to see different ways to change your login page. It is not too difficult, and will reduce the bots to almost zero.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.