Tutorial

How To Monitor Hosts and Services with Icinga on Ubuntu 16.04

How To Monitor Hosts and Services with Icinga on Ubuntu 16.04

Introduction

Icinga is an open-source monitoring system used to monitor the health of networked hosts and services. In this tutorial we will use Icinga to set up two different kinds of monitoring configurations. The first is based on simple network checks of your host’s external services, such as making a periodic HTTP request to your website. The other configuration uses a software agent running on the host to gather more detailed system information such as load and number of running processes.

Prerequisites

Before starting this tutorial, you should have completed the previous tutorial in this series, How To Install Icinga and Icinga Web on Ubuntu 16.04. This will leave you with the Icinga core and Icinga Web interface running on a single host, which we’ll refer to as the icinga-master node throughout.

You will also need some servers to monitor. We will use two Ubuntu 16.04 servers with Apache installed for our examples. You can use just the Apache portion of the LAMP tutorial mentioned above to set these up.

Step 1 – Setting up Simple Host Monitoring

One simple way to monitor a server with Icinga is to set up a regular check of its externally available services. So for a web host, we’d regularly ping the server’s IP address and also try to access a web page. This will tell us if the host is up, and if the web server is functioning correctly.

Let’s set up monitoring for a web server. Pick one of the Apache servers mentioned as a prerequisite and make sure the default Apache placeholder page is being served properly. We will call this server web-1.example.com. We won’t need to log into it at all, all the health checks will be configured and executed on the master node.

Note: Icinga always defaults to using the Fully Qualified Domain Name (FQDN) of any host it’s dealing with. A FQDN is a hostname plus its domain name, so web-1.example.com, for example. If you don’t have a proper domain set up for a host, the FQDN will be something like web-1.localdomain.

These are fine to use, just be consistent, and if you don’t have a “real” FQDN always use the server’s IP address in any Icinga address field you configure.

Log into the master node. To add a new host, we need to edit Icinga’s hosts.conf file. Open it in a text editor:

  1. sudo nano /etc/icinga2/conf.d/hosts.conf

This will open a file with some explanatory comments and a single host block defined. The existing object Host NodeName configuration block defines the icinga-master host, which is the host we installed Icinga and Icinga Web on. Position your cursor at the bottom of the file, and add a new host:

/etc/icinga2/conf.d/hosts.conf
. . .
object Host "web-1.example.com" {
  import "generic-host"
  address = "web-1_server_ip"
  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
}

This defines a host called web-1.example.com, imports some default host configs from a template called generic-host, points Icinga to the correct IP address, and then defines a few variables that tell Icinga to check for an HTTP response at the root (/) URL and notify the icingaadmins group via email when problems occur.

Save and close the file, then restart Icinga:

  1. sudo systemctl restart icinga2

Switch back to the Icinga Web interface in your browser. The interface updates itself fairly rapidly, so you don’t need to refresh the page. The new host information should populate in short order, and the health checks will change from Pending to Ok once Icinga gathers enough information.

This is a great way to monitor external services on a host, and there are other checks available for SSH servers, SMTP, and so on. But it’d also be nice to know more details about the internal health of the servers we’re monitoring.

Next, we’ll set up monitoring via an Icinga agent, so we can keep an eye on more detailed system information.

Step 2 – Setting up Agent-based Monitoring

Icinga provides a mechanism for securely communicating between a master and client node in order to run more extensive remote health checks. Instead of only knowing that our web server is successfully serving pages, we could also monitor CPU load, number of process, disk space and so on.

We’re going to set up a second server to monitor. We’ll call it web-2.example.com. We need to install the Icinga software on the remote machine, run some setup wizards to make the connection, then update some configuration files on the Icinga master node.

Note: There are many ways to architect an Icinga installation, complete with multiple tiers of master/satellite/client nodes, high-availability failover, and multiple ways to share configuration details between nodes. We are setting up a simple two tier structure with one master node and multiple client nodes. Further, all configuration will be done on the master node, and our health check commands will be scheduled on the master node and pushed to the clients. The Icinga project calls this setup Top Down Command Endpoint mode.

Set up the Master Node

First, we need to set up the master node to make client connections. We do that by running the node setup wizard on our master node:

  1. sudo icinga2 node wizard

This will start a script that asks us a few questions, and sets things up for us. In the following, we hit ENTER to accept most defaults. Non-default answers are highlighted. Some informational output has been removed for clarity:

Output
Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n Starting the Master setup routine... Please specify the common name (CN) [icinga-master.example.com]: ENTER to accept the default, or type your FQDN Checking for existing certificates for common name 'icinga-master.example.com'... Certificates not yet generated. Running 'api setup' now. Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect. Please specify the API bind host/port (optional): ENTER Bind Host []: ENTER Bind Port []: ENTER Done. Now restart your Icinga 2 daemon to finish the installation!

Restart Icinga to finish updating the configuration:

  1. sudo systemctl restart icinga2

Open up a firewall port to allow external connections to Icinga:

  1. sudo ufw allow 5665

Now we’ll switch to the client node, install Icinga and run the same wizard.

Set up the Client Node

Log into the server we’re calling web-2.example.com. We need to install the Icinga repository again, and then install Icinga itself. This is the same procedure we used on the master node. First install the key:

  1. curl -sSL https://packages.icinga.com/icinga.key | sudo apt-key add -

Open the icinga.list file:

  1. sudo nano /etc/apt/sources.list.d/icinga.list

Paste in the repository details:

/etc/apt/sources.list.d/icinga.list
deb https://packages.icinga.com/ubuntu icinga-xenial main

Save and close the file, then update the package cache:

  1. sudo apt-get update

Then, install icinga2. Note that we do not need the icinga2-ido-mysql package that we installed on the master node:

  1. sudo apt-get install icinga2

Now we run through the node wizard on this server, but we do a satellite config instead of master:

  1. sudo icinga2 node wizard
Output
Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: y Starting the Node setup routine... Please specify the common name (CN) [web-2.example.com]: ENTER Please specify the master endpoint(s) this node should connect to: Master Common Name (CN from your master setup): icinga-master.example.com Do you want to establish a connection to the master from this node? [Y/n]: y Please fill out the master connection information: Master endpoint host (Your master's IP address or FQDN): icinga-master_server_ip Master endpoint port [5665]: ENTER Add more master endpoints? [y/N]: ENTER Please specify the master connection for CSR auto-signing (defaults to master endpoint host): Host [icinga-master_server_ip]: ENTER Port [5665]: ENTER

The wizard will now fetch the public certificate from our master node and show us its details. Confirm the information, then continue:

Output
. . . Is this information correct? [y/N]: y information/cli: Received trusted master certificate. Please specify the request ticket generated on your Icinga 2 master. (Hint: # icinga2 pki ticket --cn 'web-2.example.com'):

At this point, switch back to your master server and run the command the wizard prompted you to. Don’t forget sudo in front of it:

  1. sudo icinga2 pki ticket --cn 'web-2.example.com'

The command will output a key. Copy it to your clipboard, then switch back to the client node, paste it in and hit ENTER to continue with the wizard.

Output
. . . information/cli: Requesting certificate with ticket '5f53864a504a1c42ad69faa930bffa3a12600546'. Please specify the API bind host/port (optional): Bind Host []: ENTER Bind Port []: ENTER Accept config from master? [y/N]: n Accept commands from master? [y/N]: y Done. Now restart your Icinga 2 daemon to finish the installation!

Now open up the Icinga port on your firewall:

  1. sudo ufw allow 5665

And restart Icinga to fully update the configuration:

  1. sudo systemctl restart icinga2

You can verify there’s a connection between the two servers with netstat:

  1. netstat | grep :5665

It might take a moment for the connection to be made. Eventually netstat will output a line showing an ESTABLISHED connection on the correct port.

Output
tcp 0 0 web-2_server_ip:58188 icinga-master_server_ip:5665 ESTABLISHED

This shows that our servers have connected and we’re ready to configure the client checks.

Configure the Agent Monitoring

Even though the master and client are now connected, there’s still some setup to do on the master to enable monitoring. We need to set up a new host file. Switch back to the master node.

One important level of organization in an Icinga install is the concept of a zone. All client nodes must create their own zone, and report to a parent zone, in this case our master node. By default our master node’s zone is named after its FQDN. We’ll make a directory named after our master zone within Icinga’s zones.d directory. This will hold the information for all the master zone’s clients.

Create the zone directory:

  1. sudo mkdir /etc/icinga2/zones.d/icinga-master.example.com

We’re going to create a services configuration file. This will define some service checks that we’ll perform on any remote client node. Open the file now:

  1. sudo nano /etc/icinga2/zones.d/icinga-master.example.com/services.conf

Paste in the following, then save and close:

services.conf
apply Service "load" {
  import "generic-service"
  check_command = "load"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

apply Service "procs" {
  import "generic-service"
  check_command = "procs"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint
}

This defines two service checks. The first will report on the CPU load, and the second will check the number of processes on the server. The last two lines of each service definition are important. command_endpoint tells Icinga that this service check needs to be sent to a remote command endpoint. The assign where line automatically assigns the service check to any host that has a client_endpoint variable defined.

Let’s create such a host now. Open a new file in the zone directory we previously created. Here we’ve named the file after the remote host:

  1. sudo nano /etc/icinga2/zones.d/icinga-master.example.com/web-2.example.com<^>.conf

Paste in the following configuration, then save and close the file:

web-2.example.com.conf
object Zone "web-2.example.com" {
  endpoints = [ "web-2.example.com" ]
  parent = "icinga-master.example.com"
}

object Endpoint "web-2.example.com" {
  host = "web-2_server_ip"
}

object Host "web-2.example.com" {
  import "generic-host"
  address = "web-2_server_ip"
  vars.http_vhosts["http"] = {
    http_uri = "/"
  }
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
  vars.client_endpoint = name
}

This file defines a zone for our remote host, and ties it back to the parent zone. It also defines the host as an endpoint, and then defines the host itself, importing some default rules from the generic-host template. It also sets some vars to create an HTTP check and enable email notifications. Note that because this host has vars.client_endpoint = name defined, it will also be assigned the service checks we just defined in services.conf.

Restart Icinga to update the config:

  1. sudo systemctl restart icinga2

Switch back to the Icinga Web interface, and the new host will show up with checks Pending. After a few moments, those checks should turn Ok. This means our client node is successfully running checks for the master node.

Conclusion

In this tutorial we set up two different types of monitoring with Icinga, external service checks and agent-based host checks. There’s much more to learn about configuring and working with Icinga, so you’ll likely want to dive deeper into Icinga’s extensive documentation.

Note that if you get to a point where you require custom check commands, you’ll need to sync these from the master to the client nodes using a global configuration zone. You can find out more about this specific feature here.

Finally, if you have a large number of servers to monitor, you might look into using configuration management software to automate your Icinga configuration updates. Our tutorial series Getting Started with Configuration Management give a comprehensive overview of the concepts and software involved.

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about us


About the authors

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
5 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

icinga2 will not load with the web-2.example.com.conf above. When starting the service in debug mode
/usr/sbin/icinga2 daemon -x notice

Got this error (the web and master is replaced the the systems FQDNs)

critical/config: Error: Import references unknown template: ‘generic-host’ Location: in /etc/icinga2/zones.d/icinga-master.example.com/web-2.example.com.conf: 11:3-11:23

Solved: Created a directory /etc/icinga2/zones.d/templates and copied templates.conf into the directory. The “generic-host” error went away and now are getting the following error

" critical/config: Error: Validation failed for object ‘web-2.example.com.conf’ of type ‘Host’; Attribute ‘zone’: Object ‘icinga-master.example.com’ of type ‘Zone’ does not exist."

Great addition to the basic tutorial. I had a problem (and could solve it): On my Ubuntu 16.04 the client has to have the plugins installed which was not done by simply installing icinga2. I had to do

apt-get install nagios-plugins 

on my CLIENT machine in addition to get rid of errors

execvpe(/usr/lib/nagios/plugins/check_load) failed: No such file or directory

Was gone after installing the plugins (no config necessary).

Is there a way to install icinga2 on client server running ubuntu 14.04 or different distro?

Question? Does client has to be Ubuntu 16.04? I am getting dependencies error when try to install on Ubuntu 14.04.

sudo apt-get install icinga2 Reading package lists… Done Building dependency tree
Reading state information… Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: icinga2 : Depends: icinga2-bin (= 2.6.3-1~ppa1~xenial1) but it is not going to be installed Depends: icinga2-common (= 2.6.3-1~ppa1~xenial1) but it is not going to be installed E: Unable to correct problems, you have held broken packages.

great

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel