google bot hammering bandwidth

June 5, 2013 4.3k views
I host files on my website with my VPS.... Have about 11gb worth of files hosted right now and only getting bigger... I'm getting hammered on the bandwidth by a Google bot.... The Google bot downloads EVERY file on the server a couple times a week at least.... Ive verified it is a google bot by ip in logs.....Is there a way to calm this Google bot down?
4 Answers
There are couple of ways to do it.

1. Create a Robots.txt file with following to completely ban google (You can remove it and Google can come again)

User-Agent: Googlebot
Disallow: /

2. Create a Google Webmaster account. (You can limit the google bot)

My recommendation is the first one.
You can rate limit the google bot in the robots.txt as well.

User-Agent: Googlebot
Crawl-delay: 10

will stop googlebot from crawling any quicker than every 10 seconds.
That first option that completely bans google, do you know what affect that might have on my website showing up on google search results? And is there a way to allow google but tell it to only crawl index.php or something?
Here's a resource that will help you: http://www.robotstxt.org/
Have another answer? Share your knowledge.