Icinga is a flexible and powerful open-source monitoring system used to oversee the health of networked hosts and services. It could be used to monitor the load and uptime of a cluster of web workers, free disk space on a storage device, memory consumption on a caching service, and so on. Once properly set up, Icinga can give you an at-a-glance overview of the status of large numbers of hosts and services, as well as notifications, downtime scheduling, and long-term storage of performance data.
This tutorial will cover installing the Icinga core, its database backend, and the Icinga Web interface. Finally, we’ll set up email notification so you can receive alerts in your inbox when a service is misbehaving.
Before starting this tutorial you will need:
To get the latest version of Icinga, we first need to add a software repository maintained by the Icinga team. We will then install the software with
apt-get and run through a few configuration screens to set up Icinga’s database backend.
First, download the Icinga developers’ package signing key and add it to the
- curl -sSL https://packages.icinga.com/icinga.key | sudo apt-key add -
This key will be used to automatically verify the integrity of any software we download from the Icinga repository. Now we need to add the repository address to an
apt configuration file. Open up the file with your favorite text editor. We’ll use
nano throughout this tutorial:
- sudo nano /etc/apt/sources.list.d/icinga.list
This will open a new blank text file. Paste in the following line:
deb https://packages.icinga.com/ubuntu icinga-xenial main
Save and close the file, then refresh your package cache:
- sudo apt-get update
apt-get will now download information from the repository we just added, making the Icinga packages available to install:
- sudo apt-get install icinga2 icinga2-ido-mysql
This will install the main Icinga software, along with a database adapter that enables Icinga to put historical data and other information into a MySQL database. You’ll be presented with a few configuration screens for the database adapter:
Now we need to actually enable the Icinga database backend. The
icinga2 command can enable and disable features on the command line. While we’re at it, we’ll also enable the command feature which will eventually let us run manual health checks from the web interface.
- sudo icinga2 feature enable ido-mysql command
icinga2 to use the new features:
- sudo systemctl restart icinga2
And finally, let’s check the status of
icinga2 to make sure it’s running properly:
- sudo systemctl status icinga2
Output● icinga2.service - Icinga host/service/network monitoring system Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2017-04-20 00:54:55 UTC; 3s ago Process: 15354 ExecStartPre=/usr/lib/icinga2/prepare-dirs /usr/lib/icinga2/icinga2 (code=exited, status=0/SUCCESS) Main PID: 15416 (icinga2) Tasks: 11 Memory: 7.7M CPU: 488ms . . .
If you see
Active: active (running), Icinga is up and running. Now that we’ve set up the Icinga core system and database backend, it’s time to get the web interface up and running.
The Icinga core is completely configurable and usable without a web interface, but Icinga Web provides a nice browsable overview of the health of your hosts and services, and allows you to schedule downtime, acknowledge issues, manually trigger health checks, and send notifications, right from your browser.
Let’s install Icinga Web with
- sudo apt-get install icingaweb2
The rest of Icinga Web’s setup is done in a web browser, but before we switch over, there’s one setting we need to update. Icinga Web needs a timezone to be set for the PHP environment, so let’s edit the PHP config file:
- sudo nano /etc/php/7.0/apache2/php.ini
We need to find a specific line to update. In
nano we can press
CTRL-W to bring up a search interface, type in
date.timezone, then hit
ENTER. The cursor will move to the line we need to update. First, uncomment the line by removing the initial
; semicolon, and then type in your correct timezone.
You can find the correct timezone format in the PHP manual’s timezone section. It should look something like this when you’re finished:
date.timezone = America/New_York
Save and close the file. Restart Apache to update:
- sudo systemctl restart apache2
Now it’s time to work through Incinga Web’s browser-based setup.
Before we switch over to our browser for the web-based setup process, we need to create a setup token. This is a key we generate on the command line that authorizes us to use the web setup tool. We create this key with the
- sudo icingacli setup token create
A short token will be printed:
Copy the token to your clipboard, then switch to your browser and load the Icinga Web address. By default this is your server’s domain name or IP address followed by
You’ll be presented with a configuration screen. Paste in the token you copied to your clipboard, and press Next to begin the process. There are many pages of options to go through. We’ll step through them one at a time.
On the second page, you’ll have the option to enable some extra modules for the web interface. We can safely accept the default of only enabling the Monitoring module. Click Next to continue.
The third page shows the status of our PHP environment. You shouldn’t see any red boxes, which would indicate an issue or misconfiguration. You may see some yellow boxes mentioning PostgreSQL modules being missing. We can safely ignore these, as we’re using MySQL, not PostgreSQL. Click Next to continue.
The fourth page lets us choose how we want to authenticate Icinga Web users. If you wanted to integrate with an LDAP service for authentication, this would be the place to choose that. We’ll use the default, Database, to store users in our MySQL database. Click Next to continue.
The fifth page asks us to set up a database to store the user data. This is separate from the database we previously set up during the command line install.
Most of the defaults are fine, but we also need to choose a database name and user/password combination:
Hit Next to continue.
The next page will say that your database doesn’t exist and you don’t have the credentials to create it. Enter root for the username, type in the MySQL root password and click Next to create the Icinga Web database and user.
Now we need to name the authentication backend we just created. The default icingaweb2 is fine. Click Next.
Now that we’ve set up our user database, we can create our first Icinga Web administrative account. Choose a username and password and click Next to continue.
Next we’re presented with options on how to store user preferences and logs. The defaults are fine and will store preferences in the database while logging to syslog. Hit Next to continue.
We are presented with a page to review all of our configurations. Click Next to confirm the configuration details and move on to configuring the monitoring module.
Now we start configuring the actual monitoring module for Icinga Web. Click Next to start.
First up, we select our monitoring backend. The default name of icinga and type of IDO are fine. This indicates that Icinga Web will retrieve information from the ido-mysql database we configured earlier when installing things on the command line.
We need to enter the connection details for the ido-mysql database. We created this password during installation.
This page has all the same options as the user-database setup screen:
Click Next to continue.
Next is a Command Transport prompt. This lets us specify how Icinga Web will pass commands to Icinga when we manually run health checks in the web interface. The default of Local Command File is fine and will work with the command feature we enabled back in Step 1. Click Next to continue.
This lets you specify data that should be masked in the web interface, to prevent any potential onlookers from seeing passwords and other sensitive information. The defaults are fine. Hit Next to continue.
Once again, we’re presented with a summary of our configuration. Hit Finish to finish the setup of Icinga Web. A Congratulations! message will load.
Click Login to Icinga Web 2 and log in with your administrator username and password.
The main interface of Icinga Web will load. Explore a little and familiarize yourself with the interface. If your server has no swap space set up, you may see a red Critical Error box. We can ignore this for now, or you can Acknowledge the issue by clicking the red box, selecting Acknowledge from the right-hand column, filling out a comment, and finally clicking the Acknowledge problem button.
Now that we’ve finished setting up Icinga, and Icinga Web, let’s set email notifications.
Monitoring isn’t too helpful if you can’t receive alerts when something goes wrong. Icinga’s default config has some scripts to email an administrator, but we need to set up email on our server before they’ll work. The simplest way to do that is to use a program called
ssmtp to route all the server’s mail through a standard SMTP server.
ssmtp and some helper mail utilities:
- sudo apt-get install ssmtp mailutils
And now we edit the
ssmtp configuration file with our SMTP details. These should be provided by your ISP, email provider, or IT department. You’ll need a username, pasword, and the address of your SMTP server:
- sudo nano /etc/ssmtp/ssmtp.conf
There will be some existing configuration in the file. Delete it and replace it with this very basic setup that should work with most SMTP servers:
mailhub=mail.example.com:465 UseTLS=yes FromLineOverride=yes AuthUser=smtp_username AuthPass=smtp_password
Save and close the file. To test the connection, use the
- echo "hello world" | mail -s "test subject" email@example.com
You should see an email in your inbox shortly. Now we need to update a few settings for Icinga to send mail.
To get email notifications working, update the email address Icinga is sending to:
- sudo nano /etc/icinga2/conf.d/users.conf
. . . email = "firstname.lastname@example.org" . . .
Restart Icinga one last time:
- sudo systemctl restart icinga2
The icinga-master host is already configured to send notifications when problems arise. Let’s cause a problem and see what happens. We’ll use a command called
stress to increase the system’s load in order to trigger a warning.
- sudo apt-get install stress
stress can manipulate load, disk IO, memory, and other system metrics. The Icinga default configuration will trigger a warning when the system’s load is over five. Let’s cause that now:
- stress --cpu 6
Switch back to the Icinga Web interface and you’ll see the load metric slowly rise. After a few checks it will enter a soft Warning state. Soft means that the check has to fail a few more times before it’s considered a hard state, at which time notifications will be sent. This is to avoid sending notifications for transient issues that quickly fix themselves.
Wait for the warning to reach a hard state and send the notification. You should receive an email with the details of what’s going wrong.
CTRL-C to exit the
stress command. The system load will recover fairly quickly and revert to Ok in the Icinga Web interface. You’ll also receive another email telling you that the issue has cleared up.
In this tutorial we have successfully set up Icinga and Icinga Web, including Icinga’s email notification feature. Currently we are only monitoring the Icinga host itself though. Continue on to our next tutorial How To Monitor Hosts and Services with Icinga on Ubuntu 16.04 where we will set up remote monitoring.
If you’ve enjoyed this tutorial and our broader community, consider checking out our DigitalOcean products which can also help you achieve your development goals.