Creating a central syslog server

(Last Updated On: 02/15/2019)

Your home network might already contain some devices or systems like a home server, a WiFi router, a media player, or home automation system. It is a best practice creating a central syslog server and storing logs of various sources in one place.

A pile of timber logs

In this post I will show you the way of creating a central syslog server and comply with use cases like:

Log management

In reality many companies have log management solutions. It is a must have, usually it is also required by law or compliance. Creating a central syslog server on your home server has many benefits. On one hand you can get some log management skills, which you could mention in your CV. On the other hand you may find it extremely helpful for troubleshooting issues you may ran into and creating different analytics.

Log management as a concept is pretty deep. Usually it contains some of the following topics.

  • Getting logs from various sources and store them in one place.
  • Manage life cycle of logs by having backups, archives.
  • Create alerts, analyitics or performance metrics from the logs.
  • And many more.

My goal is not to have an all in one solution. Remember creating a central syslog server for a a DIY home server is not the same as creating one (or more) for companies.

Create a central syslog server

All Linux distributions have a solution for storing syslogs locally. Usually it is either rsyslog or syslog-ng or maybe just systemd-journald or a mix of them. These software have different capabilities. My choice is syslog-ng but I will also utilize journald later.

Syslog-ng on openSUSE can be installed from the default repository, although it is a bit outdated and lacks some features I need. Therefore I use a third party repository to install a more recent version of it. Generally speaking it is up to you decide which version do you need. For having decent GeoIP support I recommend at least version 3.18.

[email protected]:~> sudo zypper ar -cfp 90 https://download.opensuse.org/repositories/home:/czanik:/syslog-ng319/openSUSE_Leap_15.0/ syslog-ng319
[email protected]:~> sudo zypper ref
[email protected]:~> sudo zypper in -y syslog-ng syslog-ng-geoip
[email protected]:~> sudo systemctl enable --now syslog-ng

Wasn’t that simple? The output is slightly shortened to improve readability.

Basic configuration

The default configuration coming with the package is pretty good and already covers the use case of storing the logs of the host system in textual format. You will find most of your system’s logs files here: /var/log/{messages,localmessages} ,

As we connect more systems, we will use syslog-ng configuration’s include ability to extend the basic configuration. The idea behind including additional configuration parts is pretty similar to the one I did with autofs.

I think the configuration syntax is very user friendly and the software’s admin guide is also a great resource.

Adding OpenWRT logs via network sources

One of the simplest use is to collecting log messages from OpenWRT devices via RFC3164 (BSD Syslog) Protocol and store them locally in textual format.

I placed this configuration as /etc/syslog-ng/conf.d/network.conf and its content is visible below. You can find some explanation after that.

source s_network{
    network(
        ip("10.1.1.10")
        port(514)
        transport("tcp")
    );
};

destination d_network{
    file(
        "/var/log/network/$HOST/$S_YEAR.$S_MONTH.$S_DAY/messages"
        create-dirs(yes)
    );
};

log {
    source(s_network);
    destination(d_network);
};

Some explanation:

  • We set up a source driver called “s_network” to listen for incoming logs on the network with IP address 10.1.1.10 on TCP port 514.
  • A destination driver called “d_network” is responsible for storing the received logs on the file system.
  • We connect the source and destination drivers to each other in a log path called “log”.

Note: The names “s_network” and “d_network” can be anything, you shall use names what describes your use case better.

Note: Macros like “$S_YEAR” and “$S_DAY” enables you to use values from the parsed log messages in various parts of the configuration. In this example it creates file names like this: /var/log/network/router/2019.01.13/messages .

Do not forget to reload syslog-ng for the changes to take effect and also open up the required port on the firewall to be able to receive the logs.

[email protected]:~> sudo systemctl reload syslog-ng
[email protected]:~> sudo firewall-cmd --add-port=514/tcp --zone=internal
[email protected]:~> sudo firewall-cmd --add-port=514/tcp --zone=internal --permanent

Set up log forwarding on OpenWRT LuCI interface

Although you could do this in the CLI both by hand or by using UCI, I will use LuCI as a configuration interface.
You should find the logging settings under System → System → External system log server, which should point to your syslog-ng server.

Note: If you also change the name of your OpenWRT device, then completely reboot the device for the change to take effect, otherwise you will still found the previous name in the logs you receive regardless of the change. I am not sure whether this is the expected behavior but I have seen this many times.

Parsing fail2ban’s logs to get GeoIP metadata

My DIY home server is reachable from the public Internet and is being scanned by botnets all the time. One solution I use as a basic defense is Fail2ban. Fail2ban analyzes login attempts on SSH by parsing authentication logs of sshd. When it finds hosts with failed authentication attempts, it bans their IP addresses with the firewall for a while. It can be extended to other services too.

Some time ago, one of my friend created a post about visualizing login attempts on a world map based on their GeoIP data. His guide used iptables logs, but I wanted to use Fail2ban’s logs to get the same information.

This use case is special as we do not use anything from the original log message except the IP address. We create new metadata based on the IP address itself.

Setup collecting and parsing logs of Fail2ban

  1. Fail2ban by default logs into a file. Although I could use that file by syslog-ng, I wanted it to log to syslog instead, so syslog-ng will get all the logs by default.
    To do that I had to change “logtarget” option to “SYSLOG” in Fail2ban’s configuration.
    You can find that setting in file: /etc/fail2ban/fail2ban.conf

    # Option: logtarget 
    # Values: [ STDOUT | STDERR | SYSLOG | SYSOUT | FILE ]  Default: STDERR
    # 
    #logtarget = /var/log/fail2ban.log
    logtarget = SYSLOG 
    
  2. You should save and place the following configuration example as /etc/syslog-ng/conf.d/fail2ban-geoip.conf .
    filter f_fail2ban {
        program("fail2ban-server");
    };
    
    filter f_fail2ban_messages {
        match('Ban' value("MESSAGE"))
        or
        match('Found' value("MESSAGE"));
    };
    
    parser p_patterndb {
        db_parser(
            file("/etc/syslog-ng/patterndb-fail2ban.xml")
        );
    };
    
    parser p_geoip2 {
        geoip2(
            "${ip}", prefix( "geoip2." )
            database( "/etc/syslog-ng/GeoLite2-City.mmdb" )
        );
    };
    
    rewrite r_geoip2 {
        set(
            "${geoip2.location.latitude},${geoip2.location.longitude}",
            value( "geoip2.location2" ),
            condition(not "${geoip2.location.latitude}" == "")
        );
    };
    
    destination d_geoip2_file {
        file(
            "/var/log/fail2ban-geoip2.log"
            template(
                "${ISODATE} -> $(format-json --key ip --key jail --key pid --key HOST --key ISODATE --key geoip2.*)\n"
            )
        );
    };
    
    log {
        source(src);
        filter(f_fail2ban);
        filter(f_fail2ban_messages);
        parser(p_patterndb);
        parser(p_geoip2);
        rewrite(r_geoip2);
        destination(d_geoip2_file);
    };

    This configuration was made with Elasticsearch in mind. However currently syslog-ng only writes the logs to the local filesystem as JSON bodies.

  3. The patterndb feature of syslog-ng will parse the logs of Fail2ban. It parses the original log message with the patterns below. You can even use the parsed values in syslog-ng macros. The macro’s name is our case is “ip” and its value is the parsed IP address. Should you want to read more about it, then check the admin guide.
    A working example can be found below. Place it in /etc/syslog-ng/patterndb-fail2ban.xml

    <!-- https://superuser.com/questions/1018927/what-is-found-in-fail2ban-log-file -->
    <?xml version='1.0' encoding='UTF-8'?>
    <patterndb version="4" pub_date="2018-06-05">
      <ruleset id='c16cd58c-37d3-42c1-b7ea-50ca2e24f864' name='fail2ban-server'>
        <pattern>fail2ban-server</pattern>
        <rules>
          <rule class='violation' id='c16cd58c-37d3-42c1-b7ea-50ca2e24f864' provider='balage'>
          <description></description>
            <patterns>
              <pattern>fail2ban.filter @QSTRING:pid:[]@: INFO @QSTRING:jail:[]@ Found @IPvANY:[email protected]</pattern>
              <pattern>fail2ban.actions @QSTRING:pid:[]@: NOTICE @QSTRING:jail:[]@ Ban @IPvANY:[email protected]</pattern>
            </patterns>
          </rule>
        </rules>
      </ruleset>
    </patterndb>
    
  4. The GeoIP database is a snapshot. If you do not have any automation to periodically fetch an updated version of the database, then you can use this SHELL script in a cronjob to that.
    #!/bin/bash
    
    set -o nounset
    DB="GeoLite2-City.tar.gz"
    
    TMPDIR=$(mktemp -d)
    cd ${TMPDIR}
    
    wget -q http://geolite.maxmind.com/download/geoip/database/${DB}
    wget -q http://geolite.maxmind.com/download/geoip/database/${DB}.md5
    
    MD5SUM_CREATED=$(md5sum ${DB} | cut -d" " -f1)
    MD5SUM_HOSTED=$(head -n1 ${DB}.md5)
    
    if [ ${MD5SUM_CREATED} != ${MD5SUM_HOSTED} ]; then
        echo "md5sums do not match"
        echo "'${MD5SUM_CREATED}'"
        echo "'${MD5SUM_HOSTED}'"
        exit 1
    fi
    tar -xf ${DB} -C ${TMPDIR}
    rm ${DB}
    rm ${DB}.md5
    SUBDIR=$(ls -d -- */)
    cd ${SUBDIR}
    cp GeoLite2-City.mmdb /etc/syslog-ng/
    chown root.root /etc/syslog-ng/GeoLite2-City.mmdb
    systemctl reload syslog-ng
    rm -rf ${TMPDIR}
    

    Here you can find and example cronjob which calls the script “geoip_db_fetcher.sh” on every Sunday at 4 AM.

    0 4 * * 6 /usr/local/bin/geoip_db_fetcher.sh
  5. Once you reloaded syslog-ng, you should be able to find messages like this one below in /var/log/fail2ban-geoip.log
    2019-01-22T10:57:33+01:00 -> {"pid":"1984","jail":"sshd","ip":"1.2.3.4","geoip2":{"registered_country":{"names":{"en":"France"},"iso_code":"FR","is_in_european_union":"true","geoname_id":"3017382"},"location2":"48.858200,2.338700","location":{"time_zone":"Europe/Paris","longitude":"2.338700","latitude":"48.858200","accuracy_radius":"500"},"country":{"names":{"en":"France"},"iso_code":"FR","is_in_european_union":"true","geoname_id":"3017382"},"continent":{"names":{"en":"Europe"},"geoname_id":"6255148","code":"EU"}},"ISODATE":"2019-01-22T10:57:33+01:00","HOST":"microchuck"}
    

    Note: There are still cases when you receive an empty JSON body. Usually this happens when an IP is already banned but tries to login again. I will try to fix that and later post an update here.
    Update: I have corrected the message filters and updated both the examples above and the GitHub repository.

A GitHub repository was also created to contain the snippets used in this blog post.

Next steps

In the next posts I will write about how to logging Docker to Elasticsearch. Also you can read about how visualizing Fail2ban logs in Kibana can be done for instance with failed login attempts of SSH.

If you found this post helpful then please share it with others. I will highly appreciate it.

Thank you.

4 thoughts on “Creating a central syslog server

  1. Dave

    Using ubuntu 18.04 and getting an error when trying to syslog-ng and cant figure out why. No info on the internet about it.

    “Mar 03 18:18:21 trumpfsmurica.com syslog-ng[16847]: Error parsing parser expression, parser plugin geoip2 not found in /etc/syslog-ng/conf.d/fail2ban-geoip.conf at line 18, column 5:
    Mar 03 18:18:21 trumpfsmurica.com syslog-ng[16847]: included from /etc/syslog-ng/syslog-ng.conf line 162, column 1
    Mar 03 18:18:21 trumpfsmurica.com syslog-ng[16847]: geoip2(
    Mar 03 18:18:21 trumpfsmurica.com syslog-ng[16847]: ^^^^^^”

    Reply
  2. Balázs Németh Post author

    According to the log message the geoip2 plugin is not installed. It can be installed by adding the following package: ‘apt-get install syslog-ng-mod-geoip’.
    The geoip2 plugin was a rewrite of the older plugin simply called ‘geoip’ which is considered legacy.
    Even though the version in Ubuntu 18.04 should support both plugins, as far as I can see the binary only supports the legacy version. You should either use ‘geoip’ instead of geoip2 or update syslog-ng from a third party repository.

    I suggest to use László Budai’s repository. He is an official developer of syslog-ng.
    You can get the repo details from here.
    https://lists.balabit.hu/pipermail/syslog-ng/2019-March/025089.html

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

4 + 2 =