How to Configure Consul KV Using Docker

Introduction

Consul by HashiCorp is a versatile tool that serves multiple functions in a modern DevOps environment. It’s widely used for service discovery, health checks, load balancing, and, notably, as a distributed key-value (KV) store. The KV store in Consul is perfect for storing dynamic configuration data, feature flags, secrets, and metadata in a highly available, consistent manner across your infrastructure such that it can be dynamically accessed by services in a distributed system. Using Docker to configure Consul’s KV store allows for quick setup and isolated environments, making it ideal for testing and development.

This tutorial will walk you through the process of setting up and configuring Consul’s KV store using Docker. By the end, you will have a fully functional Consul instance running in Docker, with KV pairs configured and accessible. This setup is essential for dynamic service configuration and state management in distributed systems.

Prerequisites

Before you begin, ensure you have the following:

• A DigitalOcean Cloud account.
• A server running Ubuntu and a non-root user with sudo privileges and an active firewall. For guidance on how to set this up, please choose your distribution from this list and follow our initial server setup guide. Please ensure to work with a supported version of Ubuntu.
• Docker installed on your Ubuntu Droplet. Please follow this tutorial on How to install and use docker on Ubuntu.
• Familiarity with the Linux command line. For an introduction or refresher to the command line, you can visit this guide on Linux command line primer.
• Basic knowledge of Docker commands and container management.

Step 1 — Pulling the Consul Docker Image

Let’s pull the official Consul image from Docker Hub. This image is maintained by HashiCorp and includes everything you need to run Consul.

Log in to your Ubuntu Droplet’s console and run:

docker pull hashicorp/consul:latest

Output
latest: Pulling from hashicorp/consul
c8bcd218a73d: Pull complete 
5f1ac8227c2a: Pull complete 
c51fd79d429a: Pull complete 
91eff479bde6: Pull complete 
4dfcc18e51db: Pull complete 
3e2a8bf39bf9: Pull complete 
bd9ddc54bea9: Pull complete 
2054d291fb84: Pull complete 
Digest: sha256:e244c64df77ab3586f177f1692e98575086eb40343dc82a6320f5e79543490eb
Status: Downloaded newer image for hashicorp/consul:latest
docker.io/hashicorp/consul:latest

Step 2 — Running the Consul Container

Now that the Consul image is downloaded, you can start a new Consul container. This container will serve as your Consul server and will allow you to interact with the KV store.

To start the container, run:

docker run -d --name=consul-server -e 

Output
CONSUL_BIND_INTERFACE=eth0 -p 8500:8500 -p 8600:8600/udp hashicorp/consul
c893b6707686bce8434213975a75c936b834cf25fc84d10b407a11c4fa8ca8ba

Here’s what this command does:

• -d runs the container in detached mode (in the background).
• --name=consul-server assigns a name to the container.
• -e CONSUL_BIND_INTERFACE=eth0 sets the network interface that Consul should bind to. This is necessary for proper network communication.
• -p 8500:8500 maps the Consul web UI and API port to the host.
• -p 8600:8600/udp maps the DNS service port for service discovery.

This step is crucial as it sets up the core Consul service, which you will use to configure the KV store.

Step 3 — Verifying the Consul Installation

To ensure that Consul is running correctly, you need to verify the container status and access the Consul UI.

First, run docker ps to list all running containers and verify that the Consul container is running.

❯ docker ps                                                                                                      
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS                                                                                    NAMES
c893b6707686   hashicorp/consul   "docker-entrypoint.s…"   51 seconds ago   Up 50 seconds   8300-8302/tcp, 8600/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/udp   consul-server

Now, check if the Consul is accessible, open a web browser, and navigate to http://localhost:8500. You should see the Consul UI.

This verification step is important to confirm that your Consul instance is running without any issues before storing data in the KV store (Step 5).

Step 4 — Configuring the Firewall (Optional)

If your Consul instance needs to be accessed externally (e.g., from other nodes in a cluster), you must adjust your firewall settings to allow traffic on the necessary ports.

For example, if you’re running Consul on a cloud instance, you may need to allow inbound traffic on ports 8500 (HTTP API) and 8600 (DNS). The specific commands will vary based on your firewall solution (UFW, iptables, etc.).

This step ensures that your Consul instance is accessible from other machines, which is essential for distributed configurations.

Step 5 — Storing Key-Value Pairs

With Consul running, you can now use the KV store to store configuration data. You can add key-value pairs using the Consul CLI or the web UI.

To store a key-value pair via the CLI, run:

docker exec -it consul-server consul kv put config/db_host 
192.168.1.100
Success! Data written to: config/db_host

docker exec -it consul-server consul kv put config/db_port 3306         
Success! Data written to: config/db_port

Here’s what this command does:

• -it - Launches the interactive terminal from the local system to the container.
• consul kv put - kv put command writes the data to the given path KV store.
• config/db_host - path to store the value.
• 192.168.1.100 - Value.

Using the Web UI,

1. Navigate to the Consul UI (http://localhost:8500).
2. Click on the “Key/Value” tab.
3. Create a new key by clicking “Create”.
4. Enter the key (e.g., config/db_host) and value (e.g., 192.168.1.100).

These commands and actions store critical configuration data that your services can access dynamically at runtime.

Step 6 — Retrieving Key-Value Pairs

Once you’ve stored some KV pairs, you’ll want to retrieve them to ensure they’ve been stored correctly.

Using the CLI, retrieve a value using the following command:

docker exec -it consul-server consul kv get config/db_host     
192.168.1.100

Using the Web UI,

1. Go to the “Key/Value” tab in the Consul UI.
2. Find the key you created and click on it to see the stored value.

Retrieving the KV pairs is a necessary step to verify that your data is correctly stored and accessible.

Step 7 — Persisting Data Using Docker Volumes

By default, Docker containers are ephemeral, meaning that any data stored inside them will be lost if the container is removed. To persist your Consul KV data, you should use Docker volumes.

1. Stop and remove the current Consul container:

docker stop consul-server 
docker rm consul-server 

Now, check the containers and you should notice Consul container not running anymore.

docker ps 
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

2.Run a new Consul container with a Docker volume attached:

docker run -d --name=consul-server -e 

Output
CONSUL_BIND_INTERFACE=eth0 -p 8500:8500 -p 8600:8600/udp -v consul_data:/consul/data hashicorp/consul
2d2a7d3ff1911c2283e70506d68391a5cbf9c935a2ae447bfb8fa21481989ef1

docker ps 

Output
CONTAINER ID   IMAGE              COMMAND                  CREATED         STATUS         PORTS                                                                                    NAMES
2d2a7d3ff191   hashicorp/consul   "docker-entrypoint.s…"   5 seconds ago   Up 4 seconds   8300-8302/tcp, 8600/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/udp   consul-server

The -v consul_data:/consul/data option mounts a Docker volume to the container, ensuring that your KV store persists across container restarts.

Step 8 — Automating Consul Startup (Optional)

For production deployments, you might want to automate the startup of your Consul container using Docker Compose. Docker Compose simplifies multi-container Docker applications and makes it easy to manage services.

Create a docker-compose.yml file with the following content:

services:
  consul:
    image: hashicorp/consul:latest
    environment:
      - CONSUL_BIND_INTERFACE=eth0
    volumes:
      - consul_data:/consul/data 
    ports:
      - "8500:8500"
      - "8600:8600/udp"
    restart: always
volumes:
  consul_data:

Then, run:

docker-compose up -d

Output
[+] Running 2/2
 ✔ Network work_default     Created                                                                                                           0.0s 
 ✔ Container consul-server  Started                                                                                                           0.1s 

docker ps

Output
WARN[0000] /Users/anandhkumar/work/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
NAME            IMAGE                     COMMAND                  SERVICE   CREATED          STATUS          PORTS
work-consul-1   hashicorp/consul:latest   "docker-entrypoint.s…"   consul    40 seconds ago   Up 11 seconds   8300-8302/tcp, 8600/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/udp

This command starts Consul automatically and ensures it restarts if it fails, making it more robust for production use.

Step 9 — Cleaning Up

Once you’ve finished working with your Consul instance, you should clean up your Docker environment to free up resources.

Let’s stop and remove the Consul container:

docker stop consul-server   
docker rm consul-server
docker ps

output
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

If you’re done with Consul, you can also remove the Docker image:

docker rmi hashicorp/consul  

output
hashicorp/consul@sha256:e244c64df77ab3586f177f1692e98575086eb40343dc82a6320f5e79543490eb
Deleted: sha256:eff8ccb509560987755a70df8d6c0b9410538d503d99498ae1ea9f48066b0439
Deleted: sha256:b5e6402bbb78eb061d538505a30300ef7f612104eaf0f11b17839a9b29bc5603
Deleted: sha256:1c61ada2ad8074615120d13bd805260d766ae8424cafbda4bded529d6a204d6f
Deleted: sha256:9b36da670e2a59f1d81c6e3c9d55906c576b384df51272977e5a9caea7131e74
Deleted: sha256:8c6e52c441c246f60ca146b71204b7d6511df75fa87a0dc0a0f91141964e8fd9
Deleted: sha256:1fce18208235de2be3c419764ec1d469229af5387447d21649c841632c653cef
Deleted: sha256:68e0a114c9c35b9aa8cac31fa32b27f886361bc85fcc63f34e882e9128f33a14
Deleted: sha256:3da5b888208a9b19694bfeaf8c74a432b50f44542d717c9e1f3ab273e505855a
Deleted: sha256:dea73e9287e6e2f3b7f9fcac4f20767d7badeefa24e52f990f1674e98abfa1a3
Deleted: sha256:201fa22d1f4c7d6e7ec43135c63b2260f303f4864f5eb43569faaa1731628799

Cleaning up helps maintain a tidy development environment and ensures that Docker resources are not unnecessarily consumed.

Conclusion

In this tutorial, you learned how to set up and configure Consul’s KV store using Docker. You’ve covered the installation of Docker, running the Consul container, configuring the KV store, persisting data with Docker volumes, and cleaning up your environment. With these steps, you can now use Consul to dynamically manage configuration data in your distributed systems, leveraging the power of Docker for easy deployment and management.