How to pip install large packages?

Posted December 21, 2020 642 views
PythonUbuntu 20.04DigitalOcean Droplets

Is there a way to install large pip packages on basic droplet instances.
There seems like some process which anneals the network bandwith after 700mbs and evantually kills pip - preventing the package from installing. Note process id 17918 was the pip process.

For example:

python -m pip install torch

Here is the shell log of the above command failing:

(tinygrad) root@transformer-cpu ~/tinygrad # ❯❯❯ python -m pip install torch                                                               ✘ 1 master
Collecting torch
  Downloading torch-1.7.1-cp38-cp38-manylinux1_x86_64.whl (776.8 MB)
     |███████████████████████████████▉| 773.3 MB 73 kB/s eta 0:00:48

     |████████████████████████████████| 776.0 MB 7.3 kB/s eta 0:01:47[1]    17918 killed     python -m pip install torch

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Submit an Answer
1 answer

Hi there @iainondigitalocean,

It sounds like that you are running out of RAM, what you could do is to add a SWAP file so that you could have some extra buffer:

Another thing that I’ve seen people reporting as working is to add the --no-cache-dir to your pip install command.

Let me know how it goes.

by Brian Boucheron
One way to guard against out-of-memory errors in applications is to add some swap space to your server. In this guide, we will cover how to add a swap file to an Ubuntu 20.04 server.