suggesitons for speed ,performance,memory , server optimization to run crawl web sites

January 28, 2015 1.4k views


i have a webcrawler (which i programmed it) it crawls
it only crawls website and find link of including comments. after this process i am going to fetch comments.
i want to run it on cloud server. i am beginner and What would you recommend for speed, memory of server ?

1 comment
  • Are you running the crawl from java? if so, you should check how much memory and CPU is currently consuming with the desire number of concurrent treats. also where are you storing the crawled data? a DB like MySQL? is so you need to check MySQL usage too to do some math.

Be the first one to answer this question.