Common Python Tools: Using virtualenv, Installing with Pip, and Managing Packages
When it comes to working with Python, especially in the domain of application development, there are certain tools that you will see being mentioned often in various places or open source code. Despite being extremely commonly used, unfortunately sometimes it is hard to get a hold of a good manual to walk you through each step, which is absolutely vital when it comes to getting familiar with such important and needed tools.
In this DigitalOcean article, we aim to fill you in on not only the basics, but also the logic behind popular Python tools and items as we dive into using them in real life scenarios. We will begin with downloading and installing some common libraries, setting and working with virtual environments (using virtualenv), and managing packages for development and production of your own applications.
This article is aimed at beginners as well as those seeking to obtain more in-depth knowledge. If you would like to see and learn more, please feel free to make a suggestion in the comments section below.
Python on CentOS
Please remember that if you are using a CentOS/RHEL system, you should abstain from working with the default Python interpreter that is shipped with the operating system. Instead, you should opt for installing Python yourself.
If you have not done this already and would like to learn how, please see our tutorial: How To Set Up Python 2.7.6 and 3.3.3 on CentOS 6.4 before we begin.
Likewise, in order to install pip and virtualenv on CentOS with a custom Python installation, you can follow the instructions on that article.
Python and Packages
Although Python applications can be made of a single file, usually they consist of a series of functions, objects (classes), handy tools and of course, variables spread across multiple file(s), placed inside modules. These modules together make up what is referred as a package.
The traditional way of installing a package involves first spotting it and then downloading. It sounds soft and simple because it actually is like many things in Python - but it is not perfect.
When the files are ready and unpacked, using the distutils module, you can install it by calling
Installation example of a package:
# Example: cd [package name] cd a_package python setup.py install
distutils (distribution utilities) is a toolset used for the packaging and distributing of code. It is shipped by default with Python (i.e. included in the standard library).
In spite of the simplicity of the procedure explained above, it is no use if the challenge abstracted from installing exists elsewhere in the process: finding and managing them. This is where package management via tools comes in - bringing along several benefits such as:
- Uninstalling (e.g.
pip uninstall package_x),
- Versioning (e.g.
pip install django==x),
- And automatic dependency management (as packages can depend on others).
Packages in Python can be tools, libraries, frameworks and applications. Given the popularity and the beauty of the language, there are tens of thousands of packages available which you can make use of for your own projects.
Package Management Tools
The two most common Python package managers are pip and easyinstall_. Both of them aim to help users with the tasks of:
- Downloading, Installing and Uninstalling
- Managing Python packages and much more
Both of them might appear to do the same thing from the outside and their joint dependence on the common library setuptools increases this notion.
However, in this case, it is what's hidden from the eye that makes the difference — and a lot of it as well.
pip vs easy_install
The first tool created for the task was easy_install. Although it was a relief and a pleasure to use compared to doing everything manually, over time it has proven to be problematic in certain aspects. That created the grounds for development of pip, another package manager.
pip (as defined by the project itself) is a replacement for easy_install, which brings many benefits over its predecessor, including, but not limited to:
- Downloading everything before installing
- Providing useful feedback during the process
- Keeping history of actions being taken
- Providing useful error messages following Python tradition
- Complements virtualenv and works with it very nicely
To learn more about pip, consider reading its introduction located at PyPI package index by clicking here.
A Thorough pip How-To
In this section, we will talk about getting the necessary dependencies for pip, installing its latest built followed by a walk-through of the core functionality offered such as installing, uninstalling, freezing and managing requirements.
When would I use pip?
As promised at the introduction, we aim to give you examples of real life scenarios.
Imagine that you are undertaking the development of a small application. You have set yourself a roadmap, and everything is going well. Then you discover a library (or a module) that can be of great help to you if you included in your application. You can download it the traditional way as we have explained. However, once you have not just one but 3, 4 or even 20 packages you need to deal with, this process becomes cumbersome. Include managing them (e.g. updating, uninstalling, replacing, using a different version), you can see the problems you will need to deal with, which are made redundant using pip, the package manager.
In order to install pip, we first need to take care of its dependencies. Do not worry though, it is very easy.
As explained above, one of the dependencies of pip is the setuptools library. It builds on the (standard) functionality of Python's distribution utilities toolset called distutils. Given that distils is provided by default, all we need left is setuptools.
We are going to securely download the setup files for setuptools using
curl. cURL is a system library which allows data transfer over various protocols (i.e. a common language for data exchange between applications, such as HTTP). It will verify the SSL certificates from the source and pass the data to the Python interpreter.
These setup files, which Python interpreter is going to execute, automate the installation process as they set up the latest stable version on our system.
Execute the following command:
curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | python -
This installation gives us the ability to use pip globally across the system. However, this is not the preferred way to install any other package. What is recommended is to always use self-contained Python environments, virtualenv. We will talk about it in the next section.
Note: You might need to explicitly gain super user privileges in order to continue with the download. In that case, consider using:
curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | sudo python -
After having its single dependency installed, we can now continue with downloading and setting up pip.
We will be again using
curl to have the setup file securely downloaded and installed.
Execute the following command:
curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python -
Default installation folder for pip is located at:
In order to use it without stating the full path, it must be appended to PATH.
After completing this step, we are ready to work with pip.
Using pip is really fun and can be considered headache free. If you have dealt with extremely unnecessary problems in the past and did not even understand why, pip will ensure that they are kept to a minimum for you hereon.
Installing packages using pip
pip can do many things but it would not be a mistake to state that the most often used function of it is installing packages. There are several ways it can handle this job for you.
Finding and installing packages:
# Example: pip install [package name] # Let's install the excellent *requests* library pip install requests
Finding and installing a specific version:
# Example: pip install [package name]==[version] # Let's install version 2.0.0. of requests pip install requests==2.0.0
Installing from a URL or a VCS repository:
# Example: pip install [url] # Let's install virtualenv pip install https://github.com/pypa/virtualenv/tarball/1.9.X
Installing inside a virtualenv:
# Example: pip install [env name] [package name] # This will either install inside an environment, or create one # Let's install requests inside a virtualenv called *venv* pip install -E venv requests
Uninstalling packages with pip:
The second most common function of pip is probably uninstalling packages.
Uninstalling a package:
# Example: pip uninstall [package name] # Let's remove requests library pip uninstall requests
Upgrading packages with pip
If you are thinking of uninstalling to install a newer version of an application, you can try upgrading.
Upgrading a package:
# Example: pip install --upgrade [package name] # Let's upgrade requests library pip install --upgrade requests
Searching for packages with pip
Before deciding to remove or upgrade a package, you might feel the need to first search for one.
Searching for a package:
# Example: pip search [package name] # Let's find all django packages # This might take a while (there's tonnes of them!) pip search django
Creating a list of installed packages with pip
One of the most truly exceptional and life saving abilities of pip is being able to create - with ease - lists ("freeze") of packages installed. This is also often called requirements. Depending on your Python environment (whether it be a virtual or a global one), pip will create a file listing all the packages installed with one single command.
Creating a fresh list ("freeze"):
Note: This command will output a file in the current working directory.
# Example: pip freeze > [file name.ext] # Let's list all the packages currently installed pip freeze > package_list.txt
Creating a list ("freeze") on top of a template:
Note: This command will output a file in the current working directory.
# Example: pip freeze -r [existing file.ext] > [filename.ext] # Let's append new packages installed after the last freeze pip freeze package_list.txt > package_list_new.txt
Installing all the packages from a list with *pip
When you are working on an application - preferably inside a virtual environment - you will have all of its dependencies (required packages) installed there. After having extracted a list of them using freeze, you can get them installed again with install.
Installing all packages from a list ("freeze"):
# Example: pip install -r [file name.ext] # Let's install back all the packages from the previous example pip install -r package_list_new.txt
A Thorough virtualenv How-To
Let's begin with defining what exactly virtualenv is and the situation where it comes in handy.
In the world of Python, an environment is a folder (directory) which contains everything that a Python project (application) needs in order to run in an organised, isolated fashion. When it is initiated, it automatically comes with its own Python interpreter - a copy of the one used to create it - alongside its very own pip.
There are a number of problems that virtualenv solves:
- Creating a fresh, isolated environment for a Python project
- Being able to download packages without requiring admin/sudo privileges
- Easily packaging an application
- Creating a list of dependencies which belongs to a single project created with pip
- Easily recovering the dependencies using a requirements file created with pip
- Giving way to portability across systems
Using virtualenv is the recommended way for working with Python projects, regardless of how many you might be busy with. It is very easy to use and an excellent tool to have at your disposable. It truly does wonders when coupled with pip.
We will start with installing virtualenv the system.
In order to install virtualenv, we are going to call in pip for help. We will install it as a globally available package for the Python interpreter to run.
There are two ways to obtain the application. The version you will be able to get depends on which one you choose.
The simplest method is using pip to search, download and install. This might not provide you the latest stable version.
Downloading virtualenv using pip:
# Example: [sudo] pip install virtualenv sudo pip install virtualenv
Downloading the latest available one using curl:
The latest release of virtualenv is
# Example: [sudo] pip install [github repo]/[version] sudo pip install https://github.com/pypa/virtualenv/tarball/1.1.X
Using this tool consists of getting it to create a folder, containing the Python interpreter and a copy of pip. Afterwards, in order to work with it, we need to either specify the location of that interpreter or
All the applications you install using the interpreter inside the virtual environment will be places within that location.
When you use pip to create a list of them, only the ones inside the folder will be compiled into a file.
Remember: When you are done working with one environment, before switching to another - or working with the globally installed one - you need to make sure to
Creating / Initiating a virtual environment (virtualenv)
Creating an environment using the same interpreter used to run it:
# Example: virtualenv [folder (env.) name] # Let's create an environment called *my_app* virtualenv my_app
Creating an environment with a custom Python interpreter:
# Example: virtualenv --python=[loc/to/python/] [env. name] virtualenv --python=/opt/python-3.3/bin/python my_app
Activating a virtual environment
# Example: source [env. name]/bin/activate # Let's activate the Python environment we just created source my_app/bin/activate
Working with a virtual environment without activating
For various reasons, you might choose not to activate the environment before using it. This brings more flexibility to commands you run, however, you need to make sure to target the correct interpreter each and every time.
# Example: [env. name]/bin/python [arguments] my_app/bin/python python_script.py
Using the pip installation inside the environment without activation
# Example: [env. name]/bin/pip [command] [arguements] # Let's install requests library without activating the env. my_app/bin/pip install requests
Deactivating a virtual environment:
# Example: deactivate # Let's deactivate the environment from earlier deactivate