Tutorial

How To Install and Use Logwatch Log Analyzer and Reporter on a VPS

Published on November 27, 2013
author

O.S Tezer

How To Install and Use Logwatch Log Analyzer and Reporter on a VPS

Introduction


Applications create what are called “log files” to keep track of activities taking place at any given time. These files, which are far from being simple text outputs, can be very complex to go through, especially if the server being managed is a busy one.

When the time comes to refer to log files (e.g. in case of failure, loss of data etc.), making use of all the available help becomes vital. Being able to quickly understand (parse) what they can tell regarding the past events and analyzing what exactly has happened then becomes exceptionally important for coming up with a solution.

Following in the footsteps of our previous articles on Linux system hardening, security monitoring and emailing alerts, in this DigitalOcean article we will talk about Logwatch: a very powerful log parser and analyzer which can make any dedicated system administrator’s life a little bit easier when tackling application related tasks and issues.

Log Files


Much like the black boxes of starships from Startrek, to keep the systems (i.e. servers) running, administrators even today rely on logs. Jokes aside, these application-generated files play a decisive role in tracking back and understanding what has happened in the past [at a given time] for the purposes of full / partial data recovery (i.e. from transaction logs), performance or strategy related analyses (e.g. from server logs) or amendments for the future (e.g. from access logs).

Simply put, log files will consist of actions and events taking place within a given time range.

A good log file should be as detailed as possible in order to help the administrator, who have the responsibility of maintaining the system, find the exact information needed for a certain purpose. Because of this very reason, log files are usually NOT concise and they contain loads of repetitions and loads of (mostly) redundant entries which need thorough analyses and filtering to make sense to a human.

This is where Logwatch, a computer application designed for this job, comes into play.

Enter Logwatch


Log management is an area consisting mostly of search, log rotation / retention and reporting. Logwatch is an application that helps with simple log management by daily analyzing and reporting a short digest from activities taking place on your machine.

Reports created by Logwatch are categorised by services (i.e. applications) running on your system, which can be configured to consist of the ones you like or all of them together by modifying its relatively simple configuration file. Furthermore, Logwatch allows the creation of custom analysis scripts for specific needs.

Installing Logwatch


Please note: Logwatch is a harmless application which should not interfere with your current services or workload. However, as always, it is recommended that you first try it on a new system and make sure to take backups.

On CentOS / RHEL


It is very simple to have Logwatch installed on a RHEL based system (e.g. CentOS). As it is an application consisting of various Perl scripts, certain related dependencies are required. Since we are going to be using yum package manager, this will be automatically taken care of. Unless you have mailx installed already, Logwatch will download it for you during the process as well.

To install Logwatch on CentOS / RHEL, run the following:

$ yum install -y logwatch

On Ubuntu / Debian


Getting Logwatch for Debian based systems (e.g. Ubuntu) is very similar to the process explained above, apart from the differences in package managers (aptitude v. yum).

To install Logwatch on Ubuntu / Debian, run the following:

$ aptitude install -y logwatch

Configuring Logwatch


Although its settings can be overridden during each run manually, in general, you will want to have Logwatch running daily, using common configuration.

Setting The Common Configurations of Logwatch


The default configuration file for Logwatch is located at:

/usr/share/logwatch/default.conf/logwatch.conf

Let’s open up this file using the nano text editor in order to modify its contents:

$ nano /usr/share/logwatch/default.conf/logwatch.conf

Upon running the command above, you will be met with a long list of variables the application uses each time it runs, whether automatically or manually.

In order to begin using it, we will need to make a few changes to these defaults.

Please remember in the future, you might want to come back to modify certain settings defined here. All services (applications) that are analyzed by Logwatch are listed on this file, as explained above (Configuration #5). As you install or remove applications from your virtual server, you can continue to receive reports on all of them or some of them by changing the settings here (see below*).

The important options which we need to set:

Please note: You will need to use your arrow keys to go up or down the lines when you will be making the following changes on the document. Once you are done going through the changes (items 1 - 6), you will need to press CTRL+X and then confirm with Y to save and close. Changes will come into effect automatically the next time logwatch runs.

1. The e-mail address to which daily digest (reports) are sent:

MailTo = root

Replace root with your email address.

Example: MailTo = sysadmin@mydomain.com

2. The e-mail address from which these reports originate:

MailFrom = Logwatch

You might wish to replace Logwatch with your own again.

Example: MailFrom = sysadmin@mydomain.com

3. Setting the range for the reports:

Range = yesterday

You have options of receiving reports for All (all available since the beginning), Today (just today) or Yesterday (just yesterday).

Example: Range = Today

4. Setting the reports’ detail:

Detail = Low

You can modify the reports’ detail here. Options are: Low, Medium and High.

Example: Detail = Medium

5. Setting services (applications) to be analysed:

By default, Logwatch covers a really wide range of services. If you would like to see a full list, you can query the contents of the file scripts/services located at /usr/share/logwatch/.

Example: ls -l /usr/share/logwatch/scripts/services

Service = All

You can choose to receive reports for all services or some specific ones.

For all services, keep the line as: Service = All

If you wish to receive reports for specific ones, modify it similar to the following example, listing each service on a new line (e.g. Service = [name]).

Example:

Service = sendmail
Service = http
Service = identd
Service = sshd2
Service = sudo
..

6. Disabling daily reports:

# DailyReport = No

If you do not wish to have daily repots generated, you should uncomment this line.

Example: DailyReport = No instead of # DailyReport = No

And that’s it! After making these changes, you will receive daily reports based on log files from your server automatically.

To learn more about Logwatch, and creating custom services to receive reports on, you can visit its full documentation by clicking here.

Running Logwatch Manually


It should be mentioned that you have the option to run Logwatch manually whenever you need through the command line.

Here are the available options [from the documentation]:

logwatch  [--detail  level  ] [--logfile log-file-group ] [--service service-name ] [--print]
   [--mailto address ] [--archives] [--range range  ]  [--debug  level  ]  [--save  file-name  ]
   [--logdir  directory ] [--hostname hostname ] [--splithosts] [--multiemail] [--output output-
   type ] [--numeric] [--no-oldfiles-log] [--version] [--help|--usage]

Unless you specify an option, it will be read from the configuration file.

Example:

$ logwatch --detail Low --mailto email@address --service http --range today

And here is what a Logwatch report can look like:

################### Logwatch 7.3.6 (05/19/07) ####################
        Processing Initiated: Wed Nov 15 15:07:00 2013
        Date Range Processed: today
                              ( 2013-Nov-15 )
                              Period is day.
      Detail Level of Output: 0
              Type of Output: unformatted
           Logfiles for Host: host_name
                 ##################################################################
    
 --------------------- Postfix Begin ------------------------

    3.453K  Bytes accepted                             3,536
    3.453K  Bytes delivered                            3,536
 ========   ================================================

        3   Accepted                                 100.00%
 --------   ------------------------------------------------
        3   Total                                    100.00%
 ========   ================================================

        3   Removed from queue
        2   Delivered
        1   Sent via SMTP

        1   Connection failure (outbound)

        1   Postfix start


 ---------------------- Postfix End -------------------------


 --------------------- Connections (secure-log) Begin ------------------------

 New Users:
    apache (48)

 New Groups:
    apache (48)


 **Unmatched Entries**
    groupadd: group added to /etc/group: name=apache, GID=48: 1 Time(s)
    groupadd: group added to /etc/gshadow: name=apache: 1 Time(s)

 ---------------------- Connections (secure-log) End -------------------------

 --------------------- SSHD Begin ------------------------


 SSHD Started: 2 Time(s)

 Users logging in through sshd:
    root:
       ip_addr (ip_addr): 1 time

 ---------------------- SSHD End -------------------------
    
 --------------------- yum Begin ------------------------


 Packages Installed:
    apr-1.3.9-5.el6_2.x86_64
    apr-util-1.3.9-3.el6_0.1.x86_64
    perl-YAML-Syck-1.07-4.el6.x86_64
    4:perl-5.10.1-131.el6_4.x86_64
    mailx-12.4-6.el6.x86_64
    1:perl-Pod-Simple-3.13-131.el6_4.x86_64
    1:perl-Pod-Escapes-1.04-131.el6_4.x86_64
    3:perl-version-0.77-131.el6_4.x86_64
    httpd-2.2.15-29.el6.centos.x86_64
    4:perl-libs-5.10.1-131.el6_4.x86_64
    mailcap-2.1.31-2.el6.noarch
    perl-Date-Manip-6.24-1.el6.noarch
    1:perl-Module-Pluggable-3.90-131.el6_4.x86_64
    httpd-tools-2.2.15-29.el6.centos.x86_64
    apr-util-ldap-1.3.9-3.el6_0.1.x86_64
    logwatch-7.3.6-49.el6.noarch

 ---------------------- yum End -------------------------

<div class=“author”>Submitted by: <a href=“https://twitter.com/ostezer”>O.S. Tezer</div>

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products

About the authors
Default avatar
O.S Tezer

author

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
10 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Adding a line to /etc/logwatch/conf/logwatch.conf

Output = html

Changes the output into easy to read HTML output

Editing the configuration in /usr/share/logwatch/ is probably not the best thing to do. Instead, copy the configuration to /etc/logwatch/ and edit the copy:

$ sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/

This is a good tutorial with one problem.

You should not modify /usr/share/logwatch/default.conf/logwatch.conf to configure the system. The next time logwatch gets updated, your custom configuration will be wiped out. I learned this the hard way.

Instead, copy the config file from usr to /etc/logwatch/conf/logwatch.conf . Make your changes here, then delete all the unchanged lines (if you don’t delete the unchanged lines logwatch will crash because of relative references).

This method will preserve your custom settings when logwatch gets updated.

How about writing an article on creating watches for currently unsupported logfiles?

Thank you for this excellent tutorial. I was getting an error when trying to run logwatch manually as per the instructions:

$ logwatch --detail Low --mailto email@address --service http --range today

The error is below:

ERROR: Date::Manip unable to determine TimeZone.

Execute the following command in a shell prompt: perldoc Date::Manip The section titled TIMEZONES describes valid TimeZones and where they can be defined.

If you happen to get this error one way that I fixed it and the only way I know of is to run this command at your prompt:

$ export DATE_MANIP=DM5

Hope this helps. And thanks again for a great tut!

Oh if if you want HTML format of the email that it sends just add this to the bottom of your logwatch config file:

#Outputs mail content into HTML format instead of text for easier reading Output = HTML

As jclgoodwin said, it is bad to edit the default config. Also, if you want to do anything fancy, it might not even work. I wanted to replace sendmail with a script of my own making to do some pre-processing, and it utterly failed until I did:

mkdir /var/cache/logwatch
cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/

What is the meaning ‘Unmatched Entries’?

I was wondering if I need to setup mail server like postfix on my Debian.

This is not article related but: please remove the “-moz-osx-font-smoothing: grayscale” from your body css… it makes all the text unreadable in MAC… you know really crispy… tested with firefox and safari…

Kamal Nasser
DigitalOcean Employee
DigitalOcean Employee badge
July 25, 2014

@Ethan.richert: You might need to whitelist outgoing port 25:

sudo iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Featured on Community

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more