Question

10 separate droplets w 10 different IP adds or one droplet w 10 parallel tasks?

Posted January 10, 2015 2.6k views

Hi,

I need to scrape many hundreds of thousands of URLs. Should I go for 10 separate droplets for 10 parallel url scrapers, or should I go with 10 parallel scarpers within the same droplet?
I suspect that IP addresses need to be different for the first case (10 different droplets). Is that correct?

Thanks.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

If you can afford the cost, and the system won’t fall under “abuse”, I’d say go with the 10 droplets plan. This will give you more CPU than a single droplet.

As for a single IP, you can set up a SSH tunnel from 9 of the servers to the 10th. This will give you the same IP across all requests, even though they are coming from different servers.

Submit an Answer