Setup a Centralized Log Server with Rsyslog in CentOS/RHEL 8

In order for a system administrators to recognize or analyze problems on a CentOS 8 or RHEL 8 server, it is important to know and view the events that occurred on the server in a particular period of time from log files found in /var/log the directory in the system.

The Syslog (System Logging Protocol) system on the server can act as a central log monitoring point over a network where all servers, network devices, switches, routers and internal services that create logs, whether linked to the particular internal issue or just informative messages can send their logs.

On a CentOS/RHEL 8 server, Rsyslog daemon is the most important log server that comes preinstalled by default, followed by Systemd Journal Daemon (journald).

Rsyslog is an open-source utility, developed as a client/server architecture service and can achieve both roles independently. It can run as a server and gather all logs transmitted by other devices over the network or it can run as a client by sending all internal system events logged to a remote Syslog server.

Requirements

  1. Installation of “CentOS 8.0″ with Screenshots
  2. Installation of RHEL 8 with Screenshots

In order to set up a centralized log server on a CentOS/RHEL 8 server, you need to check an confirm that the /var partition has enough space (a few GB minimum) to store all recorded log files on the system that send by other devices on the network. I recommend you to have a separate drive (LVM or RAID) to mount the /var/log/ directory.

How to Configure Rsyslog Server in CentOS/RHEL 8

1. As I said, Rsyslog service is installed and running automatically in CentOS/RHEL 8 server. In order to verify that the daemon is running in the system, run the following command.

# systemctl status rsyslog.service
Check Rsyslog Status in CentOS 8
Check Rsyslog Status in CentOS 8

If the service is not running by default, run the following command to start rsyslog daemon.

# systemctl start rsyslog.service

2. If the Rsyslog utility is not installed by default on the system that you plan to use as a centralized logging server, run the following dnf command to install the rsyslog package and start the daemon.

# dnf install rsyslog
# systemctl start rsyslog.service

3. Once Rsyslog utility installed, you can now configure rsyslog as a centralized logging server by opening the main configuration file /etc/rsyslog.conf, in order to receive log messages for external clients.

# vi /etc/rsyslog.conf

In the /etc/rsyslog.conf configuration file, find and uncomment the following lines to grant UDP transport reception to the Rsyslog server via 514 port. Rsyslog uses the standard UDP protocol for log transmission.

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
Configure Rsyslog Server in CentOS 8
Configure Rsyslog Server in CentOS 8

4. The UDP protocol doesn’t have the TCP overhead, and it makes data transmission faster than the TCP protocol. On the other hand, the UDP protocol doesn’t guarantee the reliability of transmitted data.

However, if you want to use TCP protocol for log reception you must find and uncomment the following lines in the /etc/rsyslog.conf the configuration file in order to configure Rsyslog daemon to bind and listen to a TCP socket on 514 port.

module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

5. Now create a new template for receiving remote messages, as this template will guide the local Rsyslog server, where to save the received messages send by Syslog network clients.

$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 
*.* ?RemoteLogs
Create Template for Rsyslog Server
Create Template for Rsyslog Server

The $template RemoteLogs directive guides Rsyslog daemon to gather and write all of the transmitted log messages to distinct files, based on the client name and remote client application that created the messages based on the outlined properties added in the template configuration: %HOSTNAME% and %PROGRAMNAME%.

All received log files will be written to the local filesystem to an allocated file named after the client machine’s hostname and kept in /var/log/ directory.

The & ~ redirect rule directs the local Rsyslog server to stop processing the received log message further and remove the messages (not write them to internal log files).

The RemoteLogs is an arbitrary name given to this template directive. You can use whatever name you want that best suitable for your template.

To configure more complex Rsyslog templates, read the Rsyslog configuration file manual by running the man rsyslog.conf command or consult Rsyslog online documentation.

# man rsyslog.conf

6. After making the above configuration changes, you can restart the Rsyslog daemon in order to apply recent changes by running the following command.

# service rsyslog restart

7. Once you restarted the Rsyslog server, it should now act as a centralized log server and record messages from Syslog clients. To confirm the Rsyslog network sockets, run netstat command and use grep utility to filter rsyslog string.

# netstat -tulpn | grep rsyslog 

If netstat command not intall on CentOS 8, you can install it using the following command.

# dnf whatprovides netstat
# dnf install net-tools
Check Rsyslog Listening Sockets
Check Rsyslog Listening Sockets

8. If you have SELinux active in CentOS/RHEL 8, run the following command to allow rsyslog traffic depending on network socket type.

# semanage port -a -t syslogd_port_t -p udp 514
# semanage port -a -t syslogd_port_t -p tcp 514

If semanage command not install on CentOS 8, you can install it using the following command.

# dnf whatprovides semanage
# dnf install policycoreutils-python-utils

9. If you have a firewall active on the system, run the following command in order to add the needed rules for allowing rsyslog traffic on ports in Firewalld.

# firewall-cmd --permanent --add-port=514/tcp
# firewall-cmd --permanent --add-port=514/udp
# firewall-cmd --reload

You can also limit incoming connections on port 514 from whitelisted IP ranges as shown.

# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="tcp" accept'
# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="udp" accept'
# firewall-cmd --reload

That’s all! Rsyslog is now configured as a centralize logs server and can collect logs from remote clients. In the next article, we will see how to configure Rsyslog client on CentOS/RHEL 8 server.

Ravi Saive
I am an experienced GNU/Linux expert and a full-stack software developer with over a decade in the field of Linux and Open Source technologies

Each tutorial at TecMint is created by a team of experienced Linux system administrators so that it meets our high-quality standards.

Join the TecMint Weekly Newsletter (More Than 156,129 Linux Enthusiasts Have Subscribed)
Was this article helpful? Please add a comment or buy me a coffee to show your appreciation.

19 thoughts on “Setup a Centralized Log Server with Rsyslog in CentOS/RHEL 8”

  1. Hi everyone,

    Has anyone migrated from CentOS 8 to another Linux flavor since RedHat is pulling the plug on it? Any good documentation developed for it? I THOROUGHLY appreciated this one!!!

    Reply
  2. Hi,

    I am trying to forward logs from RHEL 8 server to the RHEL 7 server. As we know the rsyslog format is different. Can anyone suggest, how we can make this compatible that the logs can e forwarded to the lower rhel rsyslog version?

    The below error is seen, when logs are forwarded from RHEL 8 to RHEL 7.

    Mar 14 03:12:42 ban-ext-srv01 rsyslogd: unexpected GnuTLS error -15 in nsdsel_gtls.c:178: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2078 ]
    Mar 14 03:12:42 ban-ext-srv01 rsyslogd: netstream session 0x7f0709799e30 from 192.150.31.67 will be closed due to error  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
    Mar 14 03:12:54 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:12:56 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:12:56 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:12:57 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:12:58 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:03 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:04 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:06 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:19 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:20 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:21 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:37 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:57 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:58 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:59 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    Mar 14 03:13:59 ban-ext-srv01 rsyslogd: gnutls returned error on handshake: An unexpected TLS packet was received.  [v8.24.0-34.el7 try http://www.rsyslog.com/e/2083 ]
    
    Reply
  3. This article needs a couple of minor revisions.

    All references to yum should be replaced with the new CentOS 8 package manager, dnf. Fortunately, since dnf is born of yum, the commands are backwards-compatible. “yum install xyz” should be replaced with “dnf install xyz“.

    Minor typo, “yum whatproviders semanage” should be “dnf whatprovides semanage” (“whatproviders” should be “whatprovides“)

    ". ?RemoteLogs & ~" throws errors in /var/log/messages. It should read "*.* ?RemoteLogs". The "& ~" is no longer needed in the new version of rsyslog shipped by default in CentOS 8.

    Also, there are major security risks in having a rsyslog server open to the world on port 514. You can expand the article to include examples of using rich rules to limit incoming connections on port 514 from whitelisted IP ranges.

    Examples:

    # firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="tcp" accept'
    # firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="123.123.123.0/21" port port="514" protocol="udp" accept'
    # firewall-cmd --reload

    Reply
  4. Hi

    I hope I can get some help with this. I am trying to use rsyslog to collect logs from our firewalls. So I configured the rsyslog.conf file as below.

    #### MODULES ####
    
    module(load="imuxsock" 	  # provides support for local system logging (e.g. via logger command)
           SysSock.Use="off") # Turn off message reception via local log socket; 
    			  # local messages are retrieved through imjournal now.
    module(load="imjournal" 	    # provides access to the systemd journal
           StateFile="imjournal.state") # File to store the position in the journal
    #module(load="imklog") # reads kernel messages (the same are read from journald)
    #module(load"immark") # provides --MARK-- message capability
    
    # Provides UDP syslog reception
    # for parameters see http://www.rsyslog.com/doc/imudp.html
    module(load="imudp") # needs to be done just once
    input(type="imudp" port="514")
    # TEMPLATE
    $template RemoteLogs,"/home/logs/%HOSTNAME%/%$YEAR-%$MONTH-%$DAY%/%syslogfacility-text%.log"
    . ?RemoteLogs & ~
    

    Unfortunately, when I check the /home/logs directory I do not see any files or folders created but I see traffic from the firewall going to the CentOS 8 VM. Can anyone help or suggest what I am missing? I have disabled firewalld and SELinux but still cannot get the log files generated.

    Reply
    • Template should read.

      $template RemoteLogs,"/home/logs/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/%syslogfacility-text%.log"
      

      You are missing some %. I hope that helps.

      Reply
      • Finally got it to work. Thank you so much Gator.

        Had to redo the template line as:

        $template RemoteLogs,"/home/logs/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%/%syslogfacility-text%.log"
        *.* ?RemoteLogs 
        & ~ 
        

        I think this was necessary to be on its own line also..

        Reply
  5. The next step after centralizing the logs comes the need to analyze, classify and report on the events. This is where a solution like Graylog beats a rsyslog server hands down.

    Reply
  6. Also, I was wondering if there was a way to separate the log entries incoming from different servers into separate files or directories (perhaps create a new directory per day)? It would make it easier to perform lookups, archiving, and clean-ups. Thank you, again, for the OUTSTANDING job!!!

    Reply
    • @Gator,

      You can use the following configuration parameters to write all received messages from clients in a single log file named after the IP Address of the remote client.

      $template FromIp,"/var/log/%FROMHOST-IP%.log" 
      . ?FromIp & ~
      
      Reply
      • I could not get this to work quite right. I wound up doing it a bit differently.

        I wanted to split up the logs to be generated into date folders with the clients as subfolders. The reason for this is to make it easy for archiving and cleanup as the directories are all listed by date. I went with the date/client method, but some people may prefer the client/date way in which they would just need to switch around the order of the directory creation.

        To do this, I replaced the template lines with:

        $template DynaFile,"/var/log/Remote-Logs/%$YEAR%-%$MONTH%-%$DAY%/%FROMHOST%/%syslogfacility-text%.log"
        *.* -?DynaFile.   
        

        I then changed it to

        $template DynaFile,"/var/log/Remote-Logs/%$YEAR%-%$MONTH%-%$DAY%/%FROMHOST%/%PROGRAMNAME%.log"
        

        to get a better break down of individual process logging. So far it looks good.

        Thank you very much again!!!

        Reply
  7. Hi Ravi,

    Thank you for creating a new cookbook for this on CentOS 8.

    I did find a few errors that you may want to update:

    1. With the basic installation of CentOS, semanage and netstat are not loaded. You want to add “yum whatproviders semanage” and then “yum install policycoreutils-python-utils“.

    For netstat, “yum whatproviders netstat” and then “yum install net-tools“.

    2. The semanage lines are incorrect. They should read “semanage port -a -t syslogd_port_t -p udp 514” and “semanage port -a -t syslogd_port_t -p tcp 514“.

    3. One of the firewall commands is incorrect. It should have 2 dashes: “firewall_cmd –reload“.

    Excellent job and very much thanks!!!!

    Reply

Leave a Reply to Gator Nation Cancel reply

Thank you for taking the time to share your thoughts with us. We appreciate your decision to leave a comment and value your contribution to the discussion. It's important to note that we moderate all comments in accordance with our comment policy to ensure a respectful and constructive conversation.

Rest assured that your email address will remain private and will not be published or shared with anyone. We prioritize the privacy and security of our users.