4 Good Open Source Log Monitoring and Management Tools for Linux

Linux Log Monitoring and Management Tools

4 Linux Log Monitoring and Management Tools

When an operating system such as Linux is running, there are many events happening and processes that run in the background to enable efficient and reliable use of system resources. These events may happen in system software for example the init process or user applications such as Apache, MySQL, FTP and many more.

In order to understand the state of the system and different applications and how they are working, System Administrators have to keep reviewing logfiles on daily basis in production environments.

You can imagine having to review logfiles from several system areas and applications, that is where logging systems come in handy. They help to monitor, review, analyzer and even generate reports from different logfiles as configured by a System Administrator.

  1. How to Monitor System Usages, Outages and Troubleshoot Linux Systems
  2. How to Manage Server Logs (Configure and Rotate) in Linux
  3. How to Monitor Linux Server Logs Real Time with Log.io Tool

In this article, we shall look at the top four most used open source logging management systems in Linux today, the standard logging protocol in most if not all distributions today is syslog.

1. Graylog 2

This is a fully integrated open source log management system that enables System Administrators to collect, index, and analyze both framed, systematic and disorganized data from just about any available source systems.

Graylog Linux Log Management Tool

Graylog Linux Log Management Tool

This logging system is highly pluggable and enables centralized log management from many systems. It is integrated with external components such as MongoDB for metadata and Elasticsearch used to keep logfiles and enable text search.

Graylog 2 has the following features:

  1. Ready for enterprise level production
  2. Includes a dashboard and an alerting system
  3. Can work on data from any log source
  4. Enables real time log processing
  5. Enables parsing of unstructured data
  6. Extensible and highly customizable
  7. Offers an operational data hub

For more information view the Graylog 2 website.

2. Logcheck

Logcheck is an open source log management system that helps System Administrators automatically identify unknown problems and security violations in logfiles. It periodically sends messages about the analysis results to a configured e-mail address.

Logcheck Scans System Logs

Logcheck Scans System Logs

Logcheck is designed as a cronjob on an hourly basis and on every system reboot by default. Three are different levels of logfile filtering are developed in this logging system which include:

  1. Paranoid: is intended for high-security systems that are running very few services as possible.
  2. Server: this is the default filtering level for logcheck and its rules are defined for many different system daemons. The rules defined under paranoid level are also included under this level.
  3. Workstation: it is for sheltered systems and helps to filter most of the messages. It also includes rules defined under paranoid and server levels.

Logcheck is also capable of sorting messages to be reported into three possible layers which include, security events, system events and system attack alerts. A System Administrator can choose the level of details to which system events are reported depending on the filtering level though this does not affect security events and system attack alerts.

Read more about it at the Development team’s logcheck website

3. Logwatch

Logwatch is a Linux/Unix system logfile analyzer and reporter that can be easily customized and it also allows a System Administrator to add additional plugins, create custom scripts that serve specific logging needs.

Logwatch Linux Log Analyzer

Logwatch Linux Log Analyzer

What it does is to review system logfiles for a given period to time and then generates a report based on system areas that you wish to collect information from. One feature of this logging system is that it is easy to use for new System Administrator and it also works on most Linux distributions available and many Unix systems.

Visit the project homepage of Logwatch

4. Logstash

Logstash is also an open source data collection and logging system available on Linux, which capable of real-time pipelining, which was originally designed for data collection but its new versions now integrated several other capabilities such as using a wide range of input data formats, filtering and also output plugins and formats.

LogStash

LogStash

It can effectively unify data from various log source systems and normalize the data into targets of a System Administrators’ choice. Logstash also allows System Administrators to cleanse, compare and standardize all their logging data for distinct advanced analytics and also create visualization use cases as well.

Read more about it at Logstash website.

Summary

That is it for now and remember that these are not all the available log management systems that you can use on Linux. We shall keep reviewing and updating the list in future articles, I hope you find this article useful and you can let us know of other important logging tools or systems out there by leaving a comment.

Best Affordable Linux and WordPress Services For Your Business
Outsource Your Linux and WordPress Project and Get it Promptly Completed Remotely and Delivered Online.

If You Appreciate What We Do Here On TecMint, You Should Consider:

  1. Stay Connected to: Twitter | Facebook | Google Plus
  2. Subscribe to our email updates: Sign Up Now
  3. Get your own self-hosted blog with a Free Domain at ($3.45/month).
  4. Become a Supporter - Make a contribution via PayPal
  5. Support us by purchasing our premium books in PDF format.
  6. Support us by taking our online Linux courses

We are thankful for your never ending support.

Aaron Kili

Aaron Kili is a Linux and F.O.S.S enthusiast, an upcoming Linux SysAdmin, web developer, and currently a content creator for TecMint who loves working with computers and strongly believes in sharing knowledge.

Your name can also be listed here. Got a tip? Submit it here to become an TecMint author.

RedHat RHCE and RHCSA Certification Book
Linux Foundation LFCS and LFCE Certification Preparation Guide

You may also like...

16 Responses

  1. Otis Gospodnetic says:

    I haven’t heard of Logcheck before – I’ll have to check it out, thanks!

    Logstash used to be very popular, but there are lighter log shipper now – e.g. Filebeat from Elastic or Logagent from Sematext.

  2. Robin says:

    Another free and open source log management solution you may want to check out is NXLog, it is scalable and delivers high performance.

  3. Maclean says:

    I have been started using the GrayLog 2 for my Production / Development servers Logs.
    But I’m finding it difficult in sending all the logs from httpd, mongodb etc.. If any one of you have good practical experience in configuring the Graylog 2 with the Extractor please help me.

    • Ravi Saive says:

      @Maclean,

      That’s really good to know that you’ve using GrayLog 2 for your production, I just wanted to know how good is this tool regarding collecting server logs? and why you want to send all logs from httpd or mongodb? I really didn’t understand here, what you’re trying to do so? I never tried GrayLog 2 yet, but will give a try today..

      • Maclean says:

        Hi Ravi,

        The main reason I’m going into the Graylog2 is because for its versatility of collecting the logs from different systems (firewalls, servers, application etc…) and i can create my custom Extractor for filtering the logs. Graylog2 server nodes can be expanded to support the Elastic search (1 web node, 3 Elastic Server node, 2 Data nodes) –> This is expandable as we need.

        A simple example i want a logs to be send is a specified format i can use the json and create the logs format and send it to my GrayLog2 by this we can manage many things inside the application events.

        From the http you have access and error logs if you can send this log details we can get Who all are accessing, what url failed to deliver and errors in the java scripts or icon load or path issue etc…

        But I’m still a beginner on this Apps. But the hardest part is writing the Extractor.

        • Ravi Saive says:

          @Maclean,

          Thanks for detailed explanation, now I know why Graylog2 is more powerful than any other logs tools, I think I should give a try…

  4. Andrey says:

    Graylog 2 vs Logstash. Which system is better?

  5. Bruno says:

    What about Splunk?

Leave a Reply to David Darr Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.