4 Open Source Log Monitoring and Management Tools for Linux

When an operating system such as Linux is running, there are many events happening and processes that run in the background to enable efficient and reliable use of system resources. These events may happen in system software for example the init or systemd process or user applications such as Apache, MySQL, FTP, and many more.

In order to understand the state of the system and different applications and how they are working, System Administrators have to keep reviewing log files on a daily basis in production environments.

You can imagine having to review logfiles from several system areas and applications, that is where logging systems come in handy. They help to monitor, review, analyze and even generate reports from different logfiles as configured by a System Administrator.

In this article, we shall look at the top four most-used open source logging management systems in Linux today, the standard logging protocol in most if not all distributions today is Syslog.

1. Graylog 2

Graylog is a leading open-source and robust centralized logging management tool that is widely used to collect and review logs across various environments including testing and production environments. It is easy to set up and is highly recommended for small businesses.

Graylog - Linux Leading Log Management
Graylog – Linux Leading Log Management

Graylog helps you to easily collect data from multiple devices including network switches, routers, and wireless access points. It integrates with Elasticsearch analytics engine and leverages MongoDB to store data and the logs collected offer deep insights and are helpful in troubleshooting system faults and errors.

With Graylog, you get a neat and sleep WebUI with cool dashboards that help you seamlessly track data. Also, you get a set of nifty tools and functionalities that help in compliance auditing, threat search and so much more. You can enable notifications in such a way that an alert is triggered when a certain condition is met or an issue occurs.

Overall, Graylog does a pretty good job in collating large amounts of data and simplifies searching and analyzing data. The latest version is Graylog 4.0 and offers new features such as Dark mode, integration with slack and ElasticSearch 7 and so much more.

2. Logcheck

Logcheck is yet another open-source log monitoring tool that is run as a cron job. It sifts through thousands of log files to detect violations or systems events that are triggered. Logcheck then sends a detailed summary of the alerts to a configured email address to alert operation teams of an issue such as an unauthorized breach or a system fault.

Logcheck Scans System Logs
Logcheck Scans System Logs

Three are different levels of logfile filtering are developed in this logging system which includes:

  • Paranoid: is intended for high-security systems that are running very few services as possible.
  • Server: this is the default filtering level for logcheck and its rules are defined for many different system daemons. The rules defined under the paranoid level are also included under this level.
  • Workstation: it is for sheltered systems and helps to filter most of the messages. It also includes rules defined under paranoid and server levels.

Logcheck is also capable of sorting messages to be reported into three possible layers which include, security events, system events, and system attack alerts. A System Administrator can choose the level of details to which system events are reported depending on the filtering level though this does not affect security events and system attack alerts.

Logcheck provides the following features:

  • Predefined report templates.
  • A mechanism for filtering logs using regular expressions.
  • Instant email notifications.
  • Instant security alerts.

3. Logwatch

Logwatch is an open-source and highly customizable log collection and analysis application. It parses both system and application logs and generates a report on how applications are running. The report is delivered either on command-line or through a dedicated email address.

Logwatch Linux Log Analyzer
Logwatch Linux Log Analyzer

You can easily customize Logwatch to your preference by modifying the parameters in the /etc/logwatch/conf path. It also provides something extra in the way of pre-written PERL scripts for making log parsing easier.

Logwatch comes with a tiered approach and there are 3 main locations where configuration details are defined:

  • /usr/share/logwatch/default.conf/*
  • /etc/logwatch/conf/dist.conf/*
  • /etc/logwatch/conf/*

All the default settings are defined in the /usr/share/logwatch/default.conf/logwatch.conf file. The recommended practice is to leave this file intact and instead create your own configuration file at the /etc/logwatch/conf/ path by copying the original config file and then define your custom settings.

The latest version of Logwatch is version 7.5.5 and it provides support for querying the systemd journal directly using journalctl. If you cannot afford a proprietary log management tool, Logwatch will give you peace of mind in knowing that all events will be logged and notifications delivered in case something goes amiss.

4. Logstash

Logstash is an open-source server-side data processing pipeline that accepts data from a multitude of sources including local files, or distributed systems like S3. It then processes the logs and funnels them to platforms such as Elasticsearch where they are analyzed and archived later on. It’s quite a powerful tool as it can ingest volumes of logs from multiple applications and later output them to different databases or engines all at the same time.

Logstash: Collect, Parse, Transform Logs
Logstash: Collect, Parse, Transform Logs

Logstash structures unstructured data and performs geolocation lookups, anonymizes personal data, and scales across multiple nodes as well. There’s an extensive list of data sources that you can have Logstash listen to pipe including SNMP, heartbeats, Syslog, Kafka, puppet, windows event log, etc.

Logstash relies on ‘beats’ which are lightweight data shippers which feed data to Logstash for parsing and structuring etc. Data is then sent to other destinations such as Google Cloud, MongoDB, and Elasticsearch for indexing. Logstash is a key component of Elastic Stack that allows users to collate data in any form, parse it and visualize it on interactive dashboards.

What’s more, is that Logstash enjoys widespread community support and regular updates.

Summary

That is it for now and remember that these are not all the available log management systems that you can use on Linux. We shall keep reviewing and updating the list in future articles, I hope you find this article useful and you can let us know of other important logging tools or systems out there by leaving a comment.

If you liked this article, then do subscribe to email alerts for Linux tutorials. If you have any questions or doubts? do ask for help in the comments section.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

Support Us

We are thankful for your never ending support.

16 thoughts on “4 Open Source Log Monitoring and Management Tools for Linux”

  1. I haven’t heard of Logcheck before – I’ll have to check it out, thanks!

    Logstash used to be very popular, but there are lighter log shipper now – e.g. Filebeat from Elastic or Logagent from Sematext.

    Reply
  2. Another free and open source log management solution you may want to check out is NXLog, it is scalable and delivers high performance.

    Reply
  3. I have been started using the GrayLog 2 for my Production / Development servers Logs.
    But I’m finding it difficult in sending all the logs from httpd, mongodb etc.. If any one of you have good practical experience in configuring the Graylog 2 with the Extractor please help me.

    Reply
    • @Maclean,

      That’s really good to know that you’ve using GrayLog 2 for your production, I just wanted to know how good is this tool regarding collecting server logs? and why you want to send all logs from httpd or mongodb? I really didn’t understand here, what you’re trying to do so? I never tried GrayLog 2 yet, but will give a try today..

      Reply
      • Hi Ravi,

        The main reason I’m going into the Graylog2 is because for its versatility of collecting the logs from different systems (firewalls, servers, application etc…) and i can create my custom Extractor for filtering the logs. Graylog2 server nodes can be expanded to support the Elastic search (1 web node, 3 Elastic Server node, 2 Data nodes) –> This is expandable as we need.

        A simple example i want a logs to be send is a specified format i can use the json and create the logs format and send it to my GrayLog2 by this we can manage many things inside the application events.

        From the http you have access and error logs if you can send this log details we can get Who all are accessing, what url failed to deliver and errors in the java scripts or icon load or path issue etc…

        But I’m still a beginner on this Apps. But the hardest part is writing the Extractor.

        Reply
        • @Maclean,

          Thanks for detailed explanation, now I know why Graylog2 is more powerful than any other logs tools, I think I should give a try…

          Reply

Leave a Reply to Otis Gospodnetic Cancel reply

Have a question or suggestion? Please leave a comment to start the discussion. Please keep in mind that all comments are moderated and your email address will NOT be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.