// Tutorial //

How to Manage Multistage Environments with Ansible

Published on December 1, 2016
Default avatar
By Justin Ellingwood
Developer and author at DigitalOcean.
How to Manage Multistage Environments with Ansible

Introduction

Ansible is a powerful configuration management system used to set up and manage infrastructure and applications in varied environments. While Ansible provides easy-to-read syntax, flexible workflows, and powerful tooling, it can be challenging to manage large numbers of hosts when they vary by deployment environment and functionality.

In this guide, we will discuss some strategies for using Ansible to work with multistage deployment environments. Typically, the requirements for different stages will lead to different numbers and configurations of components. For example, the memory requirements for a development server might be different than those for staging and production and it’s important to have explicit control over how the variables that represent those requirements are prioritized. In this article, we will discuss some ways that these differences can be abstracted and some constructs that Ansible provides to encourage configuration reuse.

Incomplete Strategies for Managing Multistage Environment with Ansible

While there are a number of ways that you can manage environments within Ansible, Ansible itself does not offer an opinionated solution. Rather, it provides many constructs that can be used for managing environments and allows the user to choose.

The approach we will demonstrate in this guide relies on Ansible group variables and multiple inventories. However, there are several other strategies that are worth considering. We will explore some of these ideas below and why they might present problems when implemented in complex environments.

If you want to get started with Ansible’s recommended strategy, skip ahead to the section on using Ansible groups and multiple inventories.

Relying Solely on Group Variables

At first glance, it may appear that group variables provide all of the separation between environments that Ansible requires. You can designate certain servers as belonging to your development environment, and others can be assigned to the staging and production areas. Ansible makes it easy to create groups and assign variables to them.

Group intersection, however, brings serious problems for this system. Groups are often used to categorize more than one dimension. For example:

  • deployment environments (local, dev, stage, prod, etc.)
  • host functionality (web servers, database servers, etc.)
  • datacenter region (NYC, SFO, etc.)

In these cases, hosts will usually be in one group per category. For example, a host may be a web server (functional) on stage (deployment environment) in NYC (datacenter region).

If the same variable is set by more than one group for a host, Ansible has no way of explicitly specifying precedence. You may prefer the variables associated with the deployment environments to override other values, but Ansible doesn’t provide a way to define this.

Instead, Ansible uses the last loaded value. Since Ansible evaluates groups alphabetically, the variable associated with whichever group name happens to be last in dictionary ordering will win. This is predictable behavior, but explicitly managing group name alphabetization is less than ideal from an administrative perspective.

Using Group Children to Establish a Hierarchy

Ansible allows you to assign groups to other groups using the [groupname:children] syntax in the inventory. This gives you the ability to name certain groups members of other groups. Children groups have the ability to override variables set by the parent groups.

Typically, this is used for natural classification. For example, we could have a group called environments that includes the groups dev, stage, prod. This means we could set variables in the environment group and override them in the dev group. You could similarly have a parent group called functions that contains the groups web, database, and loadbalancer.

This usage does not solve the problem of group intersection since child groups only override their parents. Child groups can override variables within the parent, but the above organization has not established any relationship between group categories, like environments and functions. The variable precedence between the two categories is still undefined.

It is possible to exploit this system by setting non-natural group membership. For instance, if you want to establish the following precedence, from highest priority to lowest:

  • development environment
  • region
  • function

You could assign group membership that looks like this:

Example inventory
. . .
[function:children]
web
database
loadbalancer
region

[region:children]
nyc
sfo
environments

[environments:children]
dev
stage
prod

We’ve established a hierarchy here that allows regional variables to override the functional variables since the region group is a child of the function group. Likewise, variables set in the environments groups can override any of the others. This means that if we set the same variable to a different value in the dev, nyc, and web groups, a host belonging to each of these would use the variable from dev.

This achieves the desired outcome and is also predictable. However, it is unintuitive, and it muddles the distinction between true children and children needed to establish the hierarchy. Ansible is designed so that its configuration is clear and easy to follow even for new users. This type of work around compromises that goal.

Using Ansible Constructs that Allow Explicit Loading Order

There are a few constructs within Ansible that allow explicit variable load ordering, namely vars_files and include_vars. These can be used within Ansible plays to explicitly load additional variables in the order defined within the file. The vars_files directive is valid within the context of a play, while the include_vars module can be used in tasks.

The general idea is to set only basic identifying variables in group_vars and then leverage these to load the correct variable files with the rest of the desired variables.

For instance, a few of the group_vars files might look like this:

group_vars/dev
---
env: dev
group_vars/stage
---
env: stage
group_vars/web
---
function: web
group_vars/database
---
function: database

We would then have a separate vars file that defines the important variables for each group. These are typically kept in a separate vars directory for clarity. Unlike group_vars files, when dealing with include_vars, files must include a .yml file extension.

Let’s pretend that we need to set the server_memory_size variable to a different value in each vars file. Your development servers will likely be smaller than your production servers. Furthermore, your web servers and database servers might have different memory requirements:

vars/dev.yml
---
server_memory_size: 512mb
vars/prod.yml
---
server_memory_size: 4gb
vars/web.yml
---
server_memory_size: 1gb
vars/database.yml
---
server_memory_size: 2gb

We could then create a playbook that explicitly loads the correct vars file based on the values assigned to the host from the group_vars files. The order of the files loaded will determine the precedence, with the last value winning.

With vars_files, an example play would look like this:

example_play.yml
---
- name: variable precedence test
  hosts: all
  vars_files:
    - "vars/{{ env }}.yml"
    - "vars/{{ function }}.yml"
  tasks:
    - debug: var=server_memory_size

Since the functional groups are loaded last, the server_memory_size value would be taken from the var/web.yml and var/database.yml files:

  1. ansible-playbook -i inventory example_play.yml
Output
. . . TASK [debug] ******************************************************************* ok: [host1] => { "server_memory_size": "1gb" # value from vars/web.yml } ok: [host2] => { "server_memory_size": "1gb" # value from vars/web.yml } ok: [host3] => { "server_memory_size": "2gb" # value from vars/database.yml } ok: [host4] => { "server_memory_size": "2gb" # value from vars/database.yml } . . .

If we switch the ordering of the files to be loaded, we can make the deployment environment variables higher priority:

example_play.yml
---
- name: variable precedence test
  hosts: all
  vars_files:
    - "vars/{{ function }}.yml"
    - "vars/{{ env }}.yml"
  tasks:
    - debug: var=server_memory_size

Running the playbook again shows values being applied from the deployment environment files:

  1. ansible-playbook -i inventory example_play.yml
Output
. . . TASK [debug] ******************************************************************* ok: [host1] => { "server_memory_size": "512mb" # value from vars/dev.yml } ok: [host2] => { "server_memory_size": "4gb" # value from vars/prod.yml } ok: [host3] => { "server_memory_size": "512mb" # value from vars/dev.yml } ok: [host4] => { "server_memory_size": "4gb" # value from vars/prod.yml } . . .

The equivalent playbook using include_vars, which operates as a task, would look like:

---
- name: variable precedence test
  hosts: localhost
  tasks:
    - include_vars:
        file: "{{ item }}"
      with_items:
        - "vars/{{ function }}.yml"
        - "vars/{{ env }}.yml"
    - debug: var=server_memory_size

This is one area where Ansible allows explicit ordering, which can be very useful. However, as with the previous examples, there are some significant drawbacks.

First of all, using vars_files and include_vars requires you to place variables that are tightly tied to groups in a different location. The group_vars location becomes a stub for the actual variables located in the vars directory. This once again adds complexity and decreases clarity. The user must match the correct variable files to the host, which is something that Ansible does automatically when using group_vars.

More importantly, relying on these techniques makes them mandatory. Every playbook will require a section that explicitly loads the correct variable files in the correct order. Playbooks without this will be unable to use the associated variables. Furthermore, running the ansible command for ad-hoc tasks will be almost entirely impossible for anything relying on variables.

So far, we’ve looked at some strategies for managing multistage environments and discussed reasons why they may not be a complete solution. However, the Ansible project does offer some suggestions on how best to abstract your infrastructure across environments.

The recommended approach is to work with multistage environments by completely separating each operating environment. Instead of maintaining all of your hosts within a single inventory file, an inventory is maintained for each of your individual environments. Separate group_vars directories are also maintained.

The basic directory structure will look something like this:

.
├── ansible.cfg
├── environments/         # Parent directory for our environment-specific directories
│   │
│   ├── dev/              # Contains all files specific to the dev environment
│   │   ├── group_vars/   # dev specific group_vars files
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts         # Contains only the hosts in the dev environment
│   │
│   ├── prod/             # Contains all files specific to the prod environment
│   │   ├── group_vars/   # prod specific group_vars files
│   │   │   ├── all
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts         # Contains only the hosts in the prod environment
│   │
│   └── stage/            # Contains all files specific to the stage environment
│       ├── group_vars/   # stage specific group_vars files
│       │   ├── all
│       │   ├── db
│       │   └── web
│       └── hosts         # Contains only the hosts in the stage environment
│
├── playbook.yml
│
└── . . .

As you can see, each environment is distinct and compartmentalized. The environment directories contain an inventory file (arbitrarily named hosts) and a separate group_vars directory.

There is some obvious duplication in the directory tree. There are web and db files for each individual environment. In this case, the duplication is desirable. Variable changes can be rolled out across environments by first modifying variables in one environment and moving them to the next after testing, just as you would with code or configuration changes. The group_vars variables track the current defaults for each environment.

One limitation is the inability to select all hosts by function across environments. Fortunately, this falls into the same category as the variable duplication problem above. While it is occasionally useful to select all of your web servers for a task, you almost always want to roll out changes across your environments one at a time. This helps prevent mistakes from affecting your production environment.

Setting Cross-Environment Variables

One thing that is not possible in the recommended setup is variable sharing across environments. There are a number of ways we could implement cross-environment variable sharing. One of the simplest is to leverage Ansible’s ability to use directories in place of files. We can replace the all file within each group_vars directory with an all directory.

Inside the directory, we can set all environment-specific variables in a file again. We can then create a symbolic link to a file location that contains cross-environment variables. Both of these will be applied to all hosts within the environment.

Begin by creating a cross-environment variables file somewhere in the hierarchy. In this example, we’ll place it in the environments directory. Place all cross-environment variables in that file:

  1. cd environments
  2. touch 000_cross_env_vars

Next, move into one of the group_vars directory, rename the all file, and create the all directory. Move the renamed file into the new directory:

  1. cd dev/group_vars
  2. mv all env_specific
  3. mkdir all
  4. mv env_specific all/

Next, you can create a symbolic link to the cross-environmental variable file:

  1. cd all/
  2. ln -s ../../../000_cross_env_vars .

When you have completed the above steps for each of your environments, your directory structure will look something like this:

.
├── ansible.cfg
├── environments/
│   │
│   ├── 000_cross_env_vars
│   │
│   ├── dev/
│   │   ├── group_vars/
│   │   │   ├── all/
│       │   │   ├── 000_cross_env_vars -> ../../../000_cross_env_vars
│   │   │   │   └── env_specific
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts
│   │
│   ├── prod/
│   │   ├── group_vars/
│   │   │   ├── all/
│   │   │   │   ├── 000_cross_env_vars -> ../../../000_cross_env_vars
│   │   │   │   └── env_specific
│   │   │   ├── db
│   │   │   └── web
│   │   └── hosts
│   │
│   └── stage/
│       ├── group_vars/
│       │   ├── all/
│       │   │   ├── 000_cross_env_vars -> ../../../000_cross_env_vars
│       │   │   └── env_specific
│       │   ├── db
│       │   └── web
│       └── hosts
│
├── playbook.yml
│
└── . . .

The variables set within 000_cross_env_vars file will be available to each of the environments with a low priority.

Setting a Default Environment Inventory

It is possible to set a default inventory file in the ansible.cfg file. This is a good idea for a few reasons.

First, it allows you to leave off explicit inventory flags to ansible and ansible-playbook. So instead of typing:

  1. ansible -i environments/dev -m ping

You can access the default inventory by typing:

  1. ansible -m ping

Secondly, setting a default inventory helps prevent unwanted changes from accidentally affecting staging or production environments. By defaulting to your development environment, the least important infrastructure is affected by changes. Promoting changes to new environments then is an explicit action that requires the -i flag.

To set a default inventory, open your ansible.cfg file. This may be in your project’s root directory or at /etc/ansible/ansible.cfg depending on your configuration.

Note: The example below demonstrates editing an ansible.cfg file in a project directory. If you are using the /etc/ansibile/ansible.cfg file for your changes, modify the editing path below. When using /etc/ansible/ansible.cfg, if your inventories are maintained outside of the /etc/ansible directory, be sure to use an absolute path instead of a relative path when setting the inventory value.

  1. nano ansible.cfg

As mentioned above, it is recommended to set your development environment as the default inventory. Notice how we can select the entire environment directory instead of the hosts file it contains:

[defaults]
inventory = ./environments/dev

You should now be able to use your default inventory without the -i option. The non-default inventories will still require the use of -i, which helps protect them from accidental changes.

Conclusion

In this article, we’ve explored the flexibility that Ansible provides for managing your hosts across multiple environments. This allows users to adopt many different strategies for handling variable precedence when a host is a member of multiple groups, but the ambiguity and lack of official direction can be challenging. As with any technology, the best fit for your organization will depend on your use-cases and the complexity of your requirements. The best way to find a strategy that fits your needs is to experiment. Share your use case and approach in the comments below.


Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in our Questions & Answers section, find tutorials and tools that will help you grow as a developer and scale your project or business, and subscribe to topics of interest.

Sign up
About the authors
Default avatar
Developer and author at DigitalOcean.

Still looking for an answer?

Was this helpful?
10 Comments

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Great article! Having used this approach for the past year, I’ve been working on plans to address some issues this structure doesn’t easily solve: isolating playbook and role changes to specific environments.

The problem especially in environments when multiple individuals are collaborating is merging playbook and/or role changes impacts all environments simultaneously. Most shops want some measure of controlled roll out of changes, so something else is needed. In the Puppet world, this is done through tools like r10k that leverage git branches for each environment and externalize the management of puppet modules (or ansible roles in our case) to separate git repositories with something like ansible galaxy to pull environment-specific versions in.

Each environmental branch looks something like:

ansible.cfg
group_vars/
    all/
        vars
        vault
    db/...
    web/...
host_vars/
inventory
playbooks/
    db.yml
    web.yml
requirements.yml
roles/

where the requirements.yml file contains URLs to relevant versioned roles to pull in.

Would love to hear thoughts and ideas on addressing this class of problem!

How would you organize playbooks? I manage several enterprise software and hardware platforms. For example, I might have 30 playbooks for installing and managing and application. Then I have another 40 playbooks for deploying and managing a hardware platform.

Some of these playbooks do the same thing, so I want those in the same directory, bu the ones that are platform-specific, I want those in their own folder so that they are easier to manage. Say , in source control.

I’d like to hear your ideas on how to accomplish this!

Great article! Do you have a public repo with an example? I’m still new to ansible and would be useful. Thanks!

Great post! After the following directory setup, I am planning to use the roles/ to store my tasks per ansible project.

Questions:

  1. Will I be able to reference the group_vars on the roles/tasks?
  2. Will I be able to call in my role’s tasks on the playbooks?

├── ansible.cfg ├── environments │   ├── prod │   │   ├── group_vars │   │   └── hosts │   ├── uat │   ├── group_vars │   └── hosts ├── roles │   └── push_ssh_config | └── install_nginx ├── play1.xml ├── play2.xml ├── play3.xml

PS: Please excuse me for the alignment above, I hope you get the idea of the directory structure.

lets say we have different environment and we want to manage large domain name in inventory can we use as variable like we have different environment. dev.example.com qa.example.com prod.customer.com

vars dev=dev.example.com qa=qa.example.com prod=prod.customer.com

so in inventory can we just define host

like host.{dev} host.{qa} host.{prod}

I’ve been using --limit make my playbooks run on a specific environment instead. Issue I ran into with the other suggestions is that I can’t use it to automate things which need to be aware of the entire network. For example, I was trying to generate my haproxy configuration.

My directory looks like this:

group_vars/
  prod
  stage
roles/
  ...
deploy.yml
load-balancer.yml
inventory

Inside my inventory file, I have something like the following:

[stage_db]
db1.stage.com
[stage_web]
web1.stage.com
[stage]
stage_web
stage_db

[prod_db]
db.prod.com
[prod_web]
web1.stage.com
[prod]
prod_db
prod_web

[db]
prod_db
stage_db
[web]
prod_web
prod_stage

This way, I can still reference, in my roles/playbooks, groups as I would normally:

- name: open up firewall for db
  become: yes
  with_items: "{{groups['web']}}"
  ufw:
    rule: allow
    from_ip: "{{hostvars[item].ansible_eth1.ipv4.address}}"
    port: 3306

Where I need to I can also reference the prod and stage environments explicitly.

The only downside to this approach is that I can’t have one big site.yml with everything in it, but I personally never needed one.

Justin n all, great write-up and discussion! The twist in my case is that the production server farm is completely separate - (fire)walled off - from the staging one, where we have multiple environments (dev, QA, functional, non-functional). It follows that each (of the two) server farm has its own ansible (installation and) controller. Do you think it’s worth “pretending” (in configuration terms) that production is part of the whole? Shouldn’t we “abstract” production and instantiate the staging ones from it?

In Ansible >1.2, you can put a group_vars folder in the playbook directory. Variables in that group_vars folder will overwrite variables in inventories/environment/group_vars

See Ansible documentation:

Tip: In Ansible 1.2 or later the group_vars/ and host_vars/ directories can exist in the playbook directory [AND] the inventory directory. If both paths exist, variables in the playbook directory will override variables set in the inventory directory.

Hi Justin! Quick question. Is there a Github repository I could tinker with? I’m trying to replicate the structure at the end of the article, but I’m still trying to understand how to properly arrange each hosts inventory file in order to leverage the variable definitions workaround you’re proposing.

Thanks again for an awesome article!

Hi Justin, thanks for the fantastic article! It gives me some idea on how to accomplish my current work to migrate and consolidate a huge bash based deployment scripting (with 103 different configurations) into a idempotent and more versatile approach.

Say, we have a set of n Customers, each one requiring a DEV, TEST, PROD stage. With each stage there should be setups of webservers and appservers. It can be possible, that you have different amounts of webservers and appservers (e.g. for HA or scaling), so maybe DEV has 1 WS and 1 APP while TEST and PROD consist of 2 WS and 3 APP each.

Formalized file system structure:

customer_n/environments/{DEV, TEST, PROD}/group_vars/{APP, WS}

I guess it would be right to model this and the further configuration (e.g. ports and so on) of each WS/APP instance in a (nested) dictionary approach in group_vars.

I’m elaborating on this and would be interested in any suggestion.