How To Clean Up Your Docker Environment Using CloudSlang on a CoreOS Cluster
CoreOS is a Linux distribution focused on quickly spinning up clustered environments by utilizing Docker containers and service discovery. If you're new to CoreOS, check out this Getting Started with CoreOS tutorial series.
However, Docker images can occupy quite a lot of disk space on the Docker host. A base image can be hundreds of MB in size and custom images can easily reach 1 GB. If you have many releases of new Docker images for your app, they can easily stockpile on the server storage; the server can run out of disk space if you don't clear out old or unused images from time to time.
CloudSlang is an open source orchestration solution that makes it easy to automate processes using workflows, or flows for short. A flow contains a list of tasks and navigation logic. A task can call an operation, which contains an action that runs a Python script or Java method, or another flow. The CloudSlang language allows you to define flows in a textual, reusable manner, and you can either use the existing content (Docker, OpenStack, and utilities) to manage your deployed applications or create your own custom flows.
In this tutorial, we will clean up the Docker environment for each machine deployed in a CoreOS cluster using CloudSlang. We will use the already existing content, so you will not need to edit any CloudSlang files.
Before you begin, you will need:
A Ubuntu 14.04 Droplet with a sudo non-root user, which will be your CloudSlang server.
Java (version 7 or later) installed on the CloudSlang server. Note that you don't need to install the JDK, only the JRE.
A cluster of three CoreOS machines. If you don't already have one, you can set one up by following this tutorial.
Step 1 — Installing unzip
In this step, we will install
unzip on the CloudSlang server.
First, make sure the package list is up to date.
- sudo apt-get update
Then, install unzip.
- sudo apt-get install unzip
Step 2 — Downloading CloudSlang
In this section, we will download the CloudSlang CLI tool and the available content (predefined operations and flows). The CloudSlang CLI is a command line interface tool that can be used to run flows.
First, download the CloudSlang CLI archive.
- wget https://github.com/CloudSlang/cloud-slang/releases/download/cloudslang-0.7.29/cslang-cli-with-content.zip
Unzip the archive.
- unzip cslang-cli-with-content.zip
This will create a
cslang directory. If you list the contents of that directory,
- ls ~/cslang
You will notice three directories in it:
python-lib, which is used for external Python libraries.
cslang, which contains the CloudSlang CLI files. The
cslang/binfolder contains a file named
cslangwhich is used to start up the CLI. The
cslang/libcontains the necessary dependencies for the application.
content, which contains the ready-made CloudSlang content. The flow we are going to run is located at
content/io/cloudslang/coreosand it is called
cluster_docker_images_maintenance.sl. This flow iterates over all the machines in the cluster and deletes unused Docker images.
Step 3 — Adding the Private Key
CloudSlang needs SSH key access to your CoreOS cluster. In this step, we will add this by creating a new key pair on the CloudSlang server and adding the public key to the CoreOS cluster.
First, create a key pair without a passphrase by following steps 1 and 2 of this tutorial. After you have a key pair, you'll need to add your public key to each of the machines in your CoreOS cluster.
First, get the public key on your CloudSlang server.
You'll see a long output which begins with
ssh-rsa and ends with
username@hostname. Copy this to use in the next command.
SSH into one of your CoreOS servers (the default username is
core), then run the following command to add your public key.
echo "your_public_key" >> ~/.ssh/authorized_keys
You'll need to do this for each server in your CoreOS cluster.
Step 4 — Running the Flow
In this section we will run the flow and verify its behavior.
In order to run the flow, on the CloudSlang server, first change to the
- cd ~/cslang/cslang/bin/
Run the executable called
cslang in order to start up the CLI.
After a moment, you'll see the CloudSlang welcome screen.
0.7.26-SNAPSHOT Welcome to CloudSlang. For assistance type help.
Enter the following command in the CLI, replacing
your_coreos_server_ip with the IP address of one of the CoreOS servers in your cluster.
custom_prefix(cslang>) run --f ../../content/io/cloudslang/coreos/cluster_docker_images_maintenance.sl --i coreos_host=your_coreos_server_ip,coreos_username=core,private_key_file=~/.ssh/id_rsa --cp ../../content/
run command triggers the flow.
--f specifies the path to the flow.
--i specifies the flow inputs: a CoreOS host and username, and the associated private SSH key.
--cp specifies the classpath when the flow depends upon other operations and flows. Because this flow has many different dependencies, we can specify the parent
content folder; the scanning is recursive so subdirectories are also scanned.
The flow logic first retrieves the IP addresses of the machines from the cluster, then iterates over the machines and clear unused images. First, it gets all the images, leaving only unused ones by checking running/stopped containers. Next, it deletes the unused images. Finally, it does the same for dangling images.
While the flow is running, the CLI displays the task names that are executed. Once the flow finishes, the CLI outputs some useful information like the flow outputs and the flow result.
In our case, the flow result will either be SUCCESS (which means unused Docker images were cleared in the cluster) or FAILURE (which means a problem occurred). This flow has one output:
number_of_deleted_images_per_host, which is how many images were deleted on every host in the cluster.
If everything went well you should see an output similar to this:
... Flow : cluster_docker_images_maintenance finished with result : SUCCESS Execution id: 101600001, duration: 0:02:06.180
If you want more information on the execution, look in the
execution.log file which is created by the CLI in the
Now all the unused Docker images are deleted in your CoreOS cluster!
In this tutorial, you have seen how to run CloudSlang on your Ubuntu machine and how to use the CloudSlang CLI to trigger flows. You also used a ready-made workflow to clean up your Docker environment.
Copyright June 9, 2015, Hewlett-Packard Development Company, L.P. Reproduced with Permission.