In the modern world of cloud computing, configuration management is a crucial step. Configuration management tools allow you to reliably deploy configurations to your servers. One of the more mature configuration management tools in this space is Puppet.
In a typical Puppet environment, a user writes Puppet modules on their workstation, pushes the modules to a version control server (e.g. Git), then pulls those modules down to a Puppet master. A server running the Puppet client periodically connects to the Puppet master to see if anything has changed, and applies the changes if so.
This scenario works just fine until you have to start scaling up how many servers are checking in or the modules become fairly complex. At that point you have two options: cluster your Puppet Master to handle the load (which will likely require you to buy the commercial version of Puppet), or just drop the Puppet master altogether. This article will look into the second option.
A masterless Puppet setup requires a copy of all Puppet modules to be copied to each node via Git and then have Puppet apply the changes locally. The disadvantage with this method is that each server downloads all of the modules, then applies what is relevant, so it’s not the best choice for e.g. setups with sensitive information. However, running without a Puppet master gives you a lot of flexibility and works great without having to scale your infrastructure.
If you are new to Puppet, then you may want to pause here to read this article on Puppet first, as this tutorial assumes a working knowledge of the tool. If you’re new to Git, you can check out this introduction to Git series, too.
In this tutorial, we’ll be working with two Droplets: one running as a Git server, and the other that we’ll be applying changes to via Puppet. We’ll refer to the IP addresses of these Droplets with your_git_server_ip
and your_puppet_server_ip
respectively.
So, to follow this tutorial, you will need:
The easiest way to set up Git Labs is to use the one click image: on the Droplet creation page under Select Image, click the Applications tab, then click GitLab 7.10.0 CE on 14.04. You can also follow this tutorial to set up Git Labs manually.
The first step is to create a repository where all of our Puppet modules and manifests will be stored.
First, open the Git Labs UI by going to http://your_git_server_ip
in your favorite browser. Create an account by filling in the details on the right under New user? Create an account and pressing the green Sign up button. You’ll receive an account activation email, and after activating your account, you’ll be able to sign in on the main page.
Click on the green + New Project button on the main page. Enter “puppet” for the Project path, and click Create project. Enter “puppet” in the Project path field, and choose Public for the Visibility Level, then click the green Create Project button.
Make sure you copy the SSH URL, which you’ll see toward the top of the project screen, as we’ll need it in a later step. It’ll look something like git@your_git_server_ip:username/puppet.git
.
In this step, we will create an SSH key on the Puppet server, then add that key to the Git Labs server.
Log in to the Puppet server as root. (Because Puppet’s files will be owned by root, we need to have rights to setup the initial Git repo in the Puppet folder.)
Create an SSH key for the root user. Make sure not to enter a passphrase because this key will be used by scripts, not a user.
- ssh-keygen -t rsa
Next, display your public key with the following command.
- cat ~/.ssh/id_rsa.pub
Copy this key. It will look something like ssh-rsa long_alphanumeric_string root@hostname
.
Now, on your Git Labs Dashboard page, click on the Profile settings icon on the top bar, second from the right. In the left menu, click SSH Keys, then click the green Add an SSH Key button. In the Title, field add a description of the key (like “Root Puppet Key”), and paste your public key into the Key field. Finally, click Add key.
In this step, we will install Puppet and Git.
On the Puppet server, first download the Puppet package for Ubuntu 14.04.
- wget http://apt.puppetlabs.com/puppetlabs-release-trusty.deb
Install the package.
- dpkg -i /tmp/puppetlabs-release-trusty.deb
Update your system’s package list.
- apt-get update
Finally, install Puppet and git.
- apt-get install puppet git-core
At this point, you should configure your Git environment by following the instructions in this tutorial.
With Puppet and Git installed, we are ready to do our initial push to our Puppet repository.
First, move to the /etc/puppet
directory, where the configuration files live.
- cd /etc/puppet
Initialize a git repository here.
- git init
Add everything in the current directory.
- git add .
Commit these changes with a descriptive comment.
- git commit -m "Initial commit of Puppet files"
Add the Git project we created earlier as origin using the SSH URL you copied in Step 1.
- git remote add origin git@your_server_ip:username/puppet.git
And finally, push the changes.
- git push -u origin master
Now that Puppet is installed, we can put everything together. At this point, you can log out as root and instead log in as the sudo non-root user you created during the prerequisites. It isn’t good practice to operate as the root user unless absolutely necessary.
To get the foundation in place, we need to make a couple of changes. First, we are going to clean up the /etc/puppet/puppet.conf
file. Using your favorite editor (vim, nano, etc.) edit /etc/puppet/puppet.conf
with the following changes.
Let’s start by making a few changes to the /etc/puppet/puppet.conf
file for our specific setup. Open the file using nano or your favorite text editor.
- sudo nano /etc/puppet/puppet.conf
The file will look like this:
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
First, remove everything from the [master]
line down, as we aren’t running a Puppet master. Also delete the last line in the [main]
section which begins with templatedir
, as this is deprecated. Finally, change the line which reads factpath=$vardir/lib/facter
to factpath=$confdir/facter
instead. $confdir
is equivalent to /etc/puppet/
, i.e. our Puppet repository.
Here is what your puppet.conf
should look like once you’re finished with the above changes.
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$confdir/facter
Now Puppet is set up, but it’s not doing any work. The way Puppet works is by looking at files called manifests that define what it should do, so in this step, we’ll create a useful module for Puppet to run.
Our first module, which we will call cron-puppet, will deploy Puppet via Git. It’ll install a Git hook that will run Puppet after a successful merge (e.g. git pull), and it’ll install a cron job to perform a git pull
every 30 minutes.
First, move into the Puppet modules directory.
- cd /etc/puppet/modules
Next, make a cron-puppet
directory containing manifests
and files
directories.
- sudo mkdir -p cron-puppet/manifests cron-puppet/files
Create and open a file called init.pp
in the manifests
directory.
- sudo nano cron-puppet/manifests/init.pp
Copy the following code into init.pp
. This is what tells Puppet to pull from Git every half hour.
class cron-puppet {
file { 'post-hook':
ensure => file,
path => '/etc/puppet/.git/hooks/post-merge',
source => 'puppet:///modules/cron-puppet/post-merge',
mode => 0755,
owner => root,
group => root,
}
cron { 'puppet-apply':
ensure => present,
command => "cd /etc/puppet ; /usr/bin/git pull",
user => root,
minute => '*/30',
require => File['post-hook'],
}
}
Save and close the file, then open another file called post-merge
in the files
directory.
- sudo nano cron-puppet/files/post-merge
Copy the following bash script into post-merge
. This bash script will run after a successful Git merge, and logs the result of the run.
#!/bin/bash -e
## Run Puppet locally using puppet apply
/usr/bin/puppet apply /etc/puppet/manifests/site.pp
## Log status of the Puppet run
if [ $? -eq 0 ]
then
/usr/bin/logger -i "Puppet has run successfully" -t "puppet-run"
exit 0
else
/usr/bin/logger -i "Puppet has ran into an error, please run Puppet manually" -t "puppet-run"
exit 1
fi
Save and close this file
Finally, we have to tell Puppet to run this module by creating a global manifest, which is canonically found at /etc/puppet/manifests/site.pp
.
- sudo nano /etc/puppet/manifests/site.pp
Paste the following into site.pp
. This creates a node classification called ‘default’. Whatever is included in the ‘default’ node will be run on every server. Here, we tell it to run our cron-puppet
module.
node default {
include cron-puppet
}
Save and close the file. Now, let’s make sure our module works by running it.
- sudo puppet apply /etc/puppet/manifests/site.pp
After a successful run you should see some output ending with a line like this.
...
Notice: Finished catalog run in 0.18 seconds
Finally, let’s commit our changes to the Git repository. First, log in as the root user, because that is the user with SSH key access to the repository.
Next, change to the /etc/puppet
directory.
- cd /etc/puppet
Add everything in that directory to the commit.
- git add .
Commit the changes with a descriptive message.
- git commit -m "Added the cron-puppet module"
Finally, push the changes.
- git push -u origin master
To add more servers, simply follow step 3 above to install Puppet and Git on the new server, then clone the Git repository to /etc/puppet
and apply the site.pp
manifest.
You can even automate this installation by using user data when you create a Droplet. Make sure you use an SSH key when you create the Droplet, and have that SSH key added to your GitLab server. Then just tick the Enable User Data checkbox on the Droplet creation screen and enter the following bash script, replacing the variables highlighted in red with your own.
#!/bin/bash -e
## Install Git and Puppet
wget -O /tmp/puppetlabs.deb http://apt.puppetlabs.com/puppetlabs-release-`lsb_release -cs`.deb
dpkg -i /tmp/puppetlabs.deb
apt-get update
apt-get -y install git-core puppet
# Clone the 'puppet' repo
cd /etc
mv puppet/ puppet-bak
git clone http://your_git_server_ip/username/puppet.git /etc/puppet
# Run Puppet initially to set up the auto-deploy mechanism
puppet apply /etc/puppet/manifests/site.pp
That’s all! You now have a masterless Puppet system, and can spin up any number of additional servers without even having to log in to them.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Is there any reason to not use bitbucket or github for the git part? Don’t get me wrong, I like this tutorial but I think it could be easier to follow without installing a git droplet.
Hi, Could you possibly post your puppet.conf file? The reason I ask is that I’m using a Solaris 11 node (urgh) and it comes bundled with Puppet, however I have no puppet.conf and want to use the agent headless.
Thanks and good tutorial! Dave
First of all - this is a great tutorial. I was able to get puppet masterless to run in a couple hours (2 to read and digest and about an hour to take my time and do the work. Since we already had some RHEL 7 VMs and Bitbucket, I used those as my machine and git repo respectively. I downloaded and installed RHEL puppet using Step 4 on the Puppet Installation website (https://docs.puppetlabs.com/guides/install_puppet/install_el.html) - Step 4: Install Puppet on Agent Nodes. I ensured I had an ssh key from my machine to Bitbucket. I then followed the configuration steps in this tutorial for all the puppet config and module creation steps. The cron entry worked on the first puppet apply. Thanks for this!
Sorry hackzilla for not responding sooner, I didn’t get the notification that someone commented on my post, and today I was just browsing around and saw your question.
To add your SSH key to your droplet at the time of creation you first have to add it to your account.
Now when you create a new Droplet at the very bottom of the page, before pressing the “Create Droplet” button, click on the name of your SSH key (it should be in a box) and that will add your key into the root user’s authorized_keys file. Hope that helps, if you haven’t already figured it out.
“Make sure you use an SSH key when you create the Droplet, and have that SSH key added to your GitLab server.”
“That’s all! You now have a masterless Puppet system, and can spin up any number of additional servers without even having to log in to them.”
I don’t understand how/where to add the private key in “create droplet”.
“At that point you have two options: cluster your Puppet Master to handle the load (which will likely require you to buy the commercial version of Puppet)”
You can scale out the open source version of Puppet. What the commercial the version of Puppet will provide you, though, is the ability to add additional masters to the cluster with a few clicks in the Puppet console.