As your infrastructure requirements expand, managing each server by hand becomes an increasingly difficult task. This difficulty is compounded by the requirement for reproducibility, which becomes necessary if a node fails or if horizontal scaling is needed.
Configuration management solutions are designed to address these issues by turning your infrastructure administration into a code base. Instead of performing individual tasks on a number of machines, these tools allow you to commit your requirements to a central location where each component can connect, pull down their configuration, and apply it.
In a previous guide, we talked, on a conceptual level, about the general structure of Chef components and the way in which they interact to achieve the administrator’s objectives. We talked about relevant terminology and discussed the responsibility of each piece.
In this guide, we will install the actual software. We will set up a centralized Chef server which will store and serve configuration instructions and node profiling information. We will also set up a workstation where the administrator can work with the code base and alter the characteristics of the infrastructure. We will follow this up by bootstrapping a new node to bring it under the management of the Chef ecosystem.
We will be setting up version 12 of Chef in this guide. Configuration can be significantly different between versions, so ensure that you are operating within the same major version number as this guide for best results.
The Chef documentation tells us that your Chef server should have at least 4 cores and 4 GB of RAM. It should also have a 64-bit operating system. For our guide, we will be using an 4 core / 8 GB DigitalOcean Droplet with 64-bit Ubuntu 14.04.
The workstation and nodes have very few requirements. We will use Ubuntu 14.04 on those as well for consistency.
When we are finished, we will have a centralized Chef server to store and serve our configuration data. Our workstation will be used to make changes, upload them to the server, and bootstrap and manage new nodes. The node represents a single server within our infrastructure.
We will begin by setting up the Chef server. Remember, Chef recommends at least 4 cores and 4 GB of RAM for this server, so plan accordingly.
Once you are logged into the server you plan on installing the Chef server onto, the first task you need to perform is to ensure that the hostname of the server is a resolvable fully qualified domain name (FQDN) or IP address. You can check this by typing:
hostname -f
The result should be an address where the server can be reached. If this is not the case, you can set this to a domain name or IP address where the server can be reached by editing this file:
sudo nano /etc/hosts
The file will look similar to this:
127.0.1.1 current_hostname current_hostname_alias
127.0.0.1 localhost
. . .
Modify the top line to reflect the fully qualified domain name or the IP address, followed by a space and any alias you want to use for your host. Add a line beneath the two lines shown that has your server’s public IP address in the first column, and the information that you modified at the end of the 127.0.1.1
line to the end. It should look something like this:
127.0.1.1 fqdn_or_IP_address host_alias
127.0.0.1 localhost
IP_address fqdn_or_IP_address host_alias
So, if I do not have a domain name, my public IP address is 123.123.123.123
, and if I also want my host reachable by the hostname “chef”, I could have a file that looks like this:
127.0.1.1 123.123.123.123 chef
127.0.0.1 localhost
123.123.123.123 123.123.123.123 chef
If, on the other hand, this server has the fully qualified domain name of chef.example.com
and an IP address of 234.234.234.234
, my file might look something like this instead:
127.0.1.1 chef.example.com chef
127.0.0.1 localhost
234.234.234.234 chef.example.com chef
Save and close the file when you are finished. You can check that the value was set correctly by typing:
hostname -f
The result should be a value that you can use to reach your Chef server from anywhere in your infrastructure.
Next, we can go ahead and download the Chef 12 server software. You can find the package that must be installed by visiting the Chef site. Specifically, for an Ubuntu installation, you can follow this link.
Under the “Ubuntu Linux 14.04” header, right-click on the download link and copy the link location:
Back on your server, change to your home directory. Paste the link you copied and use the wget
command to download the package. The link you copied may be different from the one below if there has been a minor version update since this writing:
cd ~
wget https://web-dl.packagecloud.io/chef/stable/packages/ubuntu/trusty/chef-server-core_12.0.5-1_amd64.deb
Once the download is complete, install the package by typing:
sudo dpkg -i chef-server-core_*.deb
This will install the base Chef 12 system onto the server. If you have selected a server with less powerful hardware than the recommended amount, this step may fail.
Once the installation is complete, you must call the reconfigure
command, which configures the components that make up the server to work together in your specific environment:
sudo chef-server-ctl reconfigure
Next, we need to create an admin user. This will be the username that will have access to make changes to the infrastructure components in the organization we will be creating.
We can do this using the user-create
subcommand of the chef-server-ctl
command. The command requires a number of fields to be passed in during the creation process. The general syntax is:
chef-server-ctl user-create USERNAME FIRST_NAME LAST_NAME EMAIL PASSWORD
We will include this information, and will also add -f
, an additional flag, onto the end in order to specify a filename in which to output our new user’s private RSA key. We will need this in order to authenticate using the knife
management command later.
For our example, we will create a user with the following information:
The command needed to create a user with this information is (you should change this to reflect your information, especially the password):
sudo chef-server-ctl user-create admin admin admin admin@example.com examplepass -f admin.pem
You should now have a private key called admin.pem
in your current directory.
Now that you have a user, you can create an organization with the org-create
subcommand. An organization is simply a grouping of infrastructure and configuration within Chef. The command has the following general syntax:
chef-server-ctl org-create SHORTNAME LONGNAME --association_user USERNAME
The short name is the name that you will use to refer to the organization from within Chef. The long name is the actual name of the organization. The --association_user
specifies the username that has access to administer the organization. Again, we will add the -f
flag so that we can specify the name of the file to place the private key. The key that will be created is used to validate new clients as part of the organization until they can get their own unique client key.
We will create an organization with the following qualities:
To create an organization with the above qualities, we will use the following command:
sudo chef-server-ctl org-create digitalocean "DigitalOcean, Inc." --association_user admin -f digitalocean-validator.pem
Following this, you should have two .pem
key files in your home directory. In our case, they will be called admin.pem
and digitalocean-validator.pem
. We will need to connect to this server and download these keys to our workstation momentarily. For now though, our Chef server installation is complete.
Now that our Chef server is up and running, our next course of action is to configure a workstation. The actual infrastructure coordination and configuration does not take place on the Chef server. This work is done on a workstation which then uploads the data to the server to influence the Chef environment.
The Chef configuration for your infrastructure is maintained in a hierarchical file structure known collectively as a Chef repo. The general structure of this can be found in a GitHub repository provided by the Chef team. We will use git
to clone this repo onto our workstation to work as a basis for our infrastructure’s Chef repository.
First, we need to install git
through the apt
packaging tools. Update your packaging index and install the tool by typing:
sudo apt-get update
sudo apt-get install git
Once you have git
installed, you can clone the Chef repository onto your machine. For this guide, we will simply clone it to our home directory:
cd ~
git clone https://github.com/chef/chef-repo.git
This will pull down the basic Chef repo structure into a directory called chef-repo
in your home directory.
The configurations authored within the Chef repo itself are best managed within a version control system in the same way that you would manage code. Since we cloned the repo above, a git
repo has already been initialized.
To set your workstation up for new commits, you should do a few things.
First, set the name and email that git
will use to tag any commits you make. This is a requirement for git
to accept commits. We set this globally so that any git
repo we create will use these values:
git config --global user.name "Your Name"
git config --global user.email "username@domain.com"
Next, we will tell git
to ignore any information contained within the ~/chef-repo/.chef
directory. We will create this directory in a few minutes to store some sensitive information. For now, we can add this location to our .gitignore
file so that git
does not store data that should not be exposed to other people:
echo ".chef" >> ~/chef-repo/.gitignore
Since we have made a change to the .gitignore
file, we can go ahead and make our first new commit to the version control system. First, add all of the modified files to the current staging area:
cd ~/chef-repo
git add .
Now, commit the changes. We will use the -m
flag to specify an in-line commit message describing the changes we are making:
git commit -m "Excluding the ./.chef directory from version control"
Our Chef repo is now under version control. As we author configurations for our infrastructure, we can use the above two commands to keep our git
repo up-to-date.
Next, we need to install the Chef Development Kit, a suite of software designed for Chef workstations. This includes many utilities that will be useful when designing configurations for your infrastructure. The tool we are interested in at this point is the bundled knife
command, which can communicate with and control both the Chef server and any Chef clients.
We can find the Chef 12 Development Kit on the Chef website. Since we are using Ubuntu 14.04 as our workstation, the page here will contain the latest download link. Note that at the time of this writing, the download link only references Ubuntu 12.04 and Ubuntu 13.10, but it should still install without issue on Ubuntu 14.04.
Right-click on the download button under “Ubuntu Linux” and copy the link location:
Back on your workstation, change to your home directory. Paste the link you copied and use the wget
command to download the package. The link you copied may be different from the one below if a newer development kit version has been released:
cd ~
wget https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chefdk_0.4.0-1_amd64.deb
Once the .deb
package has been downloaded, you can install it by typing:
sudo dpkg -i chefdk_*.deb
After the installation, you can verify that all of the components are available in their expected location through the new chef
command:
chef verify
If your workstation will primarily be used to manage Chef for your infrastructure, you will likely want to default to the version of Ruby installed with Chef. You can do this by modifying your .bash_profile
so that Chef’s Ruby takes precedence:
echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile
Afterwards, you can source your .bash_profile
file to set the correct environmental variables for the current session:
source ~/.bash_profile
If you wish to manage your Ruby versions independently, you can skip the above steps.
At this point, your workstation has all of the software needed to interact with a Chef server and compose infrastructure configurations. However, it is not yet configured to interact with your Chef server and your environment. In this section, we’ll download the credentials we created on the Chef server.
We will use the scp
utility to download the user key and the organization validator key that we created on the Chef server. Before doing so, we will create the hidden directory where we will store these files:
mkdir ~/chef-repo/.chef
The method that you use to connect to the Chef server will determine how exactly we go about downloading the keys. Follow the method below that matches your setup:
If you connect to your Chef server through SSH using password-based authentication, the scp
command will work without significant modification.
On your workstation, specify the username and domain name or IP address used to connect to the Chef server. Follow this immediately with a colon (:) and the path to the file you wish to download. After adding a space, indicate the directory on the local computer where you wish the download the files to be placed (~/chef-repo/.chef
in our case).
If you log into the Chef server using the root
user account, your commands will look something like this. Remember to change both the domain name or IP address and the name of the key files you are trying to download to match your environment:
scp root@server_domain_or_IP:/root/admin.pem ~/chef-repo/.chef
scp root@server_domain_or_IP:/root/digitalocean-validator.pem ~/chef-repo/.chef
If you connect to your Chef server using a non-root user, the commands will look more like this:
scp username@server_domain_or_IP:/home/username/admin.pem ~/chef-repo/.chef
scp username@server_domain_or_IP:/home/username/digitalocean-validator.pem ~/chef-repo/.chef
If, instead, you connect to your Chef server using SSH keys (recommended), you will need to perform some additional steps.
First, leave your SSH session with the workstation. We will need to reconnect momentarily with a new parameter:
exit
Once you are back on your local computer, you will need to add the SSH keys you use to connect to the Chef server to an SSH agent. OpenSSH, the standard SSH suite, includes an SSH agent that can be started by typing:
eval $(ssh-agent)
You should see output that looks like this (the number will likely be different):
Agent pid 13881
Once the agent is started, you can add your SSH key to it:
ssh-add
Identity added: /home/demo/.ssh/id_rsa (rsa w/o comment)
This will keep your SSH key stored in memory. Now, you can forward the stored key to your workstation as you connect by using the -A
option with ssh
. This will allow you to connect to any computer from your workstation as if you were connecting from your local computer:
ssh -A username@workstation_domain_or_IP
Now, you can connect to your Chef server without needing a password using the forwarded SSH credentials. If the keys on your Chef server were available through the root user, the commands you will need will look similar to this. Remember to change the Chef server domain name or IP address and the key names as needed:
scp root@server_domain_or_IP:/root/admin.pem ~/chef-repo/.chef
scp root@server_domain_or_IP:/root/digitalocean-validator.pem ~/chef-repo/.chef
If the SSH key configured for the Chef server instead is used to authenticate you to a regular user account, your commands will look like this instead:
scp username@server_domain_or_IP:/home/username/admin.pem ~/chef-repo/.chef
scp username@server_domain_or_IP:/home/username/digitalocean-validator.pem ~/chef-repo/.chef
Now that you have your Chef credentials available on your workstation, we can configure the knife
command with the information it needs to connect to and control your Chef infrastructure. This is done through a knife.rb
file that we will place in the ~/chef-repo/.chef
directory along with our keys.
Open up a file called knife.rb
in that directory in your text editor:
nano ~/chef-repo/.chef/knife.rb
In this file, paste the following information:
current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "name_for_workstation"
client_key "#{current_dir}/name_of_user_key"
validation_client_name "organization_validator_name"
validation_key "#{current_dir}/organization_validator_key"
chef_server_url "https://server_domain_or_IP/organizations/organization_name"
syntax_check_cache_path "#{ENV['HOME']}/.chef/syntaxcache"
cookbook_path ["#{current_dir}/../cookbooks"]
The following items should be adjusted to suit your infrastructure:
node_name
: This specifies the name that knife
will use to connect to your Chef server. This should match your user name.client_key
: This should be the name and path to the user key that you copied over from the Chef server. We can use the #{current_dir}
snippet to fill in the path if the key is in the same directory as the knife.rb
file.validation_client_name
: This is the name of the validation client that knife
will use to bootstrap new nodes. This will take the form of your organization short name, followed by -validator
.validation_key
: Like the client_key
, this includes the name and path to the validation key you copied from the Chef server. Again, you can use the #{current_dir}
Ruby snippet to specify the current directory if the validation key is in the same directory as the knife.rb
file.chef_server_url
: This is the URL where the Chef server can be reached. It should begin with https://
, followed by your Chef server’s domain name or IP address. Afterwards, the path to your organization should be specified by appending /organizations/your_organization_name
.For our guide, the knife.rb
file will look similar to this. You still need to adjust the server’s domain name or IP address if you are following along:
current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "admin"
client_key "#{current_dir}/admin.pem"
validation_client_name "digitalocean-validator"
validation_key "#{current_dir}/digitalocean-validator.pem"
chef_server_url "https://server_domain_or_IP/organizations/digitalocean"
syntax_check_cache_path "#{ENV['HOME']}/.chef/syntaxcache"
cookbook_path ["#{current_dir}/../cookbooks"]
When you are finished, save and close the knife.rb
file.
Now, we will test the configuration file by trying out a simple knife
command. We need to be in our ~/chef-repo
directory for our configuration file to be read correctly:
cd ~/chef-repo
knife client list
This first attempt should fail with an error that looks like this:
ERROR: SSL Validation failure connecting to host: server_domain_or_IP - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
ERROR: Could not establish a secure connection to the server.
Use `knife ssl check` to troubleshoot your SSL configuration.
If your Chef Server uses a self-signed certificate, you can use
`knife ssl fetch` to make knife trust the server's certificates.
Original Exception: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
This occurs because we do not have our Chef server’s SSL certificate on our workstation. We can acquire this by typing:
knife ssl fetch
This should add the Chef server’s certificate file to a list in our ~/chef-repo/.chef
directory:
WARNING: Certificates from server_domain_or_IP will be fetched and placed in your trusted_cert
directory (/home/demo/chef-repo/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
Adding certificate for server_domain_or_IP in /home/demo/chef-repo/.chef/trusted_certs/server_domain_or_IP.crt
After the SSL certificate has been fetched, the previous command should now work:
knife client list
digitalocean-validator
If the above command correctly returns, your workstation is now set up to control your Chef environment.
With our Chef server and workstation configured, we can begin using Chef to configure new servers within our infrastructure.
This happens through a process called “bootstrapping” in which the Chef client executable is installed on the new computer and the organizational validator key is passed along as well. The new node then contacts the Chef server with the validator key and, in return, receives its own unique client key and any configuration that has been assigned to it. This process gets the new server into its initial state and sets it up for any future management.
To connect to the new server, we will need a few pieces of information about the new node:
root
, or a user configured with sudo
privileges.root
users, this is unnecessary. For users relying on sudo
privileges, a password is generally necessary.The general syntax of the command will be:
knife bootstrap node_domain_or_IP [options]
Some common options you may end up using are:
-x
: Used to specify the username to authenticate with through SSH. This is usually required.-N
: The new name for the node, as displayed within Chef. Leaving this out will usually result in the hostname being used for the Chef node name.-P
: Used to specify the password for the username on the remote server. This is necessary if either the SSH session requires password authentication or if the username requires a password for sudo
commands.--sudo
: If the username on the remote server will need to use sudo
to perform administrative actions, this flag is needed. By default, it will prompt for the sudo
password.--use-sudo-password
: If you are already providing the password for the user with the -P
flag, using this flag in addition to the --sudo
flag will use the -P
password without prompting.-A
: This option forwards SSH keys to the remote host to login rather than using password authentication.When using the -A
option, you must start an SSH agent on your local computer, add the SSH key that can be used to connect to the new node, and forward that information to your workstation by connecting with the -A
flag initially. More information about how to do this can be found in the workstation configuration section regarding downloading the keys from the Chef server.
Using the above information, it is possible to construct the correct bootstrapping commands for a variety of situations.
For example, to bootstrap a node with the name “testing”, using the username demo
, which is configured with sudo
privileges, and which needs a password for SSH and the sudo
validation, we can type:
knife bootstrap node_domain_or_IP -N testing -x demo -P password --sudo --use-sudo-password
If we want to bootstrap using the root
user, with SSH key authentication using keys available on the workstation, and wish to keep use the node’s hostname as the Chef node name, we can type:
knife bootstrap node_domain_or_IP -x root -A
If we want to use SSH keys to authenticate to a sudo
user, we will still need to provide a password using the -P
flag, the --sudo
flag, and the --use-sudo-password
flag to avoid prompts:
knife bootstrap node_domain_or_IP -x demo -A -P password --sudo --use-sudo-password -N name
If you are in the above scenario, but do not mind being promted for the sudo
password, you can instead just type this:
knife bootstrap node_domain_or_IP -x demo -A --sudo -N name
Once your new node is bootstrapped, you should have a new client:
knife client list
digitalocean-validator
name
You should also have a new node of the same name:
knife node list
name
You can use the above procedure to easily set up new Chef clients on any number of new servers.
If you want to learn about how to automatically add your new DigitalOcean Droplets to your existing Chef infrastructure without having to bootstrap each one, check out this tutorial.
After following this guide, you should have a fully functional Chef server configured for your infrastructure. We have also set up a workstation that can be used to manage and maintain the configurations that Chef will apply to your infrastructure. We have demonstrated how to use the knife
command to bootstrap the servers that will be configured by Chef.
In the next guide, we will demonstrate how to design configurations for your nodes using some Chef constructs. We will go over the fundamentals of Chef recipes and cookbooks as ways to control your infrastructure with declarative configs.
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
Chef is a powerful configuration management system that can be used to programmatically control your infrastructure environment. Leveraging the Chef system allows you to easily recreate your environments in a predictable manner by automating the entire system configuration. In this series, we will introduce you to Chef concepts and demonstrate how to install and utilize the its powerful features to manage your servers.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
After six months but still many thanks !! This is the best chef introduction I’ve found.!
A very useful article. I was able to exactly follow and create the chef server,workstation and client successfully.
Hey thanks for share this article I am in Bootstrapping a New Node with Knife part. Connecting to 192.168.1.106 192.168.1.106 -----> Installing Chef Omnibus (-v 12) 192.168.1.106 downloading https://www.opscode.com/chef/install.sh 192.168.1.106 to file /tmp/install.sh.1843/install.sh 192.168.1.106 trying wget… 192.168.1.106 Downloading Chef 12 for el… 192.168.1.106 downloading https://www.opscode.com/chef/metadata?v=12&prerelease=false&nightlies=false&p=el&pv=6&m=x86_64 … 192.168.1.106 downloaded metadata file looks valid… 192.168.1.106 downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-12.4.1-1.el6.x86_64.rpm
I just wondering is it possible redirect those Internet address to my internal repository because my environment is not able to connect Internet.
This comment has been deleted
Icame to setup the Workstation. but this
echo ".chef" >> ~/chef-repo/.gitignore
command returns No such file or directoryHi! Great tutorial, thanks!
Just a small update. Use of chef-repo is now deprecated.
You should use chef generate repo
After knife client list, 5 times to connect, and failed Network error: error connecting to https://my server name/organizations/my organization name/clients I can view my organization name on opscode-management., but I got error, for this reason I can bootstrap node. Any idea or advice. Thanks,
This comment has been deleted