Installing and configuring logcheck

Installing logcheck
The Debian readme says on logcheck: Logcheck compares recent log entries with a series of sets of egrep pattern-matching rules that flag them as urgent or filter them out as routine, and mails the results to the administrator. It's another great tool to enchance the security on your system. As with a lot of tools, it takes some effort to customize but it's worthwhile in the end.

Installation

Install logcheck:

apt update
apt install logcheck

Tune the settings via /etc/logcheck/logcheck.conf. Here's an example:

DATE="$(date +'%H:%M %d/%m/%Y')"
REPORTLEVEL="server"
SENDMAILTO="logcheck"
MAILASATTACH=0
FQDN=1
ATTACKSUBJECT="Security Alerts"
SECURITYSUBJECT="Security Events"
EVENTSSUBJECT="System Events"
TMP="/tmp"

Set the report level to workstation, server or paranoid. To summarize the different levels:

  • paranoid
    • minimal set of filters in ignore.d.paranoid
    • for systems like firewalls, with few services
    • lot's of output (verbose), make sure you can handle it
  • server
    • more selective set of filters in ignore.d.server
    • also contains the paranoid filters
    • for machines running more services
  • workstation
    • elaborate set of filters in ignore.d.workstation
    • also uses ignore.d.paranoid and ignore.d.server filter
    • for non critical machines

syslog-summary

Logcheck can use another tool, syslog-summary to minimize the number of log entries reported. According to the man page of syslog-summary:

syslog-summary summarizes the contents of log files via the syslog(3)
service, by displaying each unique (except for the time) line once, and
also the number of times such a line occurs in the input. The lines are
displayed in the order they occur in the input.

If you want to use this functionality, you need to install syslog-summary and enable the option in logcheck.conf. Logcheck will run syslog-summary over each section it encounters.:

apt install syslog-summary

vi /etc/logcheck/logcheck.conf
...
SYSLOGSUMMARY=1

There is also a log-summary-ssh tool. The script is placed in /usr/share/doc/logcheck/. Quote from the script /usr/share/doc/logcheck/log-summary-ssh:

Selects two lines that are very common with ssh scans.
This script removes those from output and prints out aggregate
statistics for those (both by host and by attempted user names)

If you want to use this tool instead of syslog-summary, you will need to add these line to your config:

SYSLOGSUMMARY=1
SYSLOG_SUMMARY=/usr/local/sbin/log-summary-ssh

Copy the script to /usr/local/sbin/ and make it executable:

cp usr/local/sbin/log-summary-ssh /usr/local/sbin/
chmod 750 /usr/local/sbin/log-summary-ssh

The script also explains how to use both syslog-summary and the log-summary-ssh tool:

If you want to use both syslog-summary and this script, you need to
write some kind of wrapper around those.

To do this, we create a wrapper script:

vi /usr/local/sbin/my-log-summary

#!/bin/sh
syslog-summary $* | /usr/local/sbin/log-summary-sh

Make it executable:

chmod 750 /usr/local/sbin/my-log-summary

Change the logcheck config:

SYSLOG_SUMMARY=/usr/local/sbin/my-log-summary

Files to scan

Next you need to specify the logfiles you want logcheck to go over. This is done by specifying the logfiles in /etc/logcheck/logcheck.logfiles.:

vi /etc/logcheck/logcheck.logfiles

/var/log/syslog
/var/log/auth.log

By default, syslog and auth.log are specified, as most syslog entries appear in one of those.

Mail reports

In order to mail reports the option SENDMAILTO needs to be set to a valid system user or e-mail address. If you use a system user, there should be a valid alias for logcheck, and a mailer (mail, sendmail, sSMTP, Postfix, ...) should be installed.:

vi /etc/aliases
…
logcheck: root
root: my-offsite-email-address

How it works

Logcheck reads each log entry from the files specified in /etc/logcheck/logcheck.logfiles. In this case, the logfiles /var/log/syslog and /var/log/auth.log. It also keeps track of the messages it has read so far. Logcheck uses a Perl utility named "logtail" to do this which bookmarks its place in the logs, so that events aren't reported twice in successive logcheck runs.

More info can be found in:

  • /usr/share/doc/logcheck-database/README.logcheck-database.gz
  • /usr/share/doc/logcheck/README.logcheck

Here's the gist of it: There are three layers of sets of filtering rules, all of which are normal egrep pattern-matches, applied in turn. log messages handled at one layer are not carried over to lower layers. So, for each new log entry, the following is done:

  1. the "SECURITY ALERTS" layer, designed to detect the traces of active intrusion attempts.
    • Patterns raising the alarm go in "/etc/logcheck/cracking.d"
      • if pattern match: "Security Alerts" report, with the relevant event moved to a special section.
    • Patterns cancelling maximum-priority alarms
      • not enabled by default, enable in config file
      • rules go in the directory "/etc/logcheck/cracking.ignore.d".
      • if pattern match: reclassify the alert as a false alarm (=not reported)
  2. the "SECURITY EVENTS" layer, designed to detect less critical events still considered worthy of special attention.
    • Patterns raising the alarm go in "/etc/logcheck/violations.d"
      • if pattern match: "Security Events" report, with the relevant event moved to a special section.
    • Patterns cancelling such alarms
      • rules go in the directory "/etc/logcheck/violations.ignore.d"
      • if pattern match: "Security Events" that match with violations.ignore patterns are discarded as false alarms.
  3. the "SYSTEM EVENTS" layer, handling leftover log messages.
    • This layer doesn't have an equivalent to the alarm-raising cracking.d and violations.d. All remaining lines are considered for inclusion in the main "System Events" section.
    • Patterns in the three "/etc/logcheck/ignore.d.*" directories overrule alerts
      • if pattern match: excluded from the report
      • the directories used are depended on the REPORTLEVEL (workstation|server|paranoid) specified.
      • the bare minimum set of ignore filters is in ignore.d.paranoid.

If logcheck doesn't select any messages, no email is not sent.

Tuning

Now it's time to have our first test run, as we'll probably need to tune what get's mailed:

sudo -u logcheck logcheck -m root -t

The flags used:

-m        Mail report to recipient.
-t        Testing mode does not update offset.

Shortly after logcheck finishes it's checks, you will probably receive an e-mail. If you don't, and your mail settings are ok, this would mean there was nothing worthwhile reporting.

You can also use logcheck-test. To test our filter rulefile against syslog:

logcheck-test -s -r local-server

The flags used:

-s|--syslog                 : Parse /var/log/syslog
-r|--rule-file RULEFILE     : Use file RULEFILE for rule input

Another interesting and useful flag is -i:

-i|--invert-match           : Show lines that don't match the RULE or RULEFILE

If you do get an e-mail, go over the entries to see if they require your attention or not. If not, then those lines should be ignored in future runs of logcheck. These are good candidates for our ignore file.

Logcheck allows you to specify which entries it needs to ignore. This is done by putting regular expressions in a file in one of the /etc/logcheck/ignore.d.<REPORTLEVEL> directories. For instance, if you use REPORTLEVEL=server in the config file, the directory where you would put your own ignore file is /etc/logcheck/ignore.d.server.

Note

egrep uses "Extended Regular Expressions" to filter/match so not all character classes might be supported. In the webtool we'll use to test our regular expressions, you can choose PCRE (perl regular expressions) but always check your regex with egrep to make sure all classes are supported. For instance, don't use s but use [:space:] instead. Instead you could use a different online tester such as RexV.2

In this example we'll tune logcheck by adding an ignore file in /etc/logcheck/ignore.d.server/

Note

Do not use a period in the name if you don't want run-parts to ignore it. It's a good idea to use local-<name> for your rules. You could use the online regex tester to test your regex lines.

Let's create such a file and put some entries in it. Use the output of the test run we have previously done to tune the output on your machine.:

cd /etc/logcheck/ignore.d.server/
vi local-server

# SSSD
sssd_be: GSSAPI client step 2
sssd_be: GSSAPI client step 1
sssd_be: message repeated 2 times
# Postfix
postfix/local\[[[:digit:]]*\]: warning: database /etc/aliases.db is older than source file
# Kernel
perf: interrupt took too long
CIFS VFS: Free previous auth_key.response

Note

These rules are very simple. It's better to use more strict matching rules in order to avoid log lines being ignored that really shouldn't be ignored. Have a look at the standard ignore rules for a better understanding of how the included ignore rules work. Further in this article we will rewrite these rules to be more strictly matching.

Since logcheck uses egrep to match the regular expressions in the ignore directories to filter the entries, we can use grep to test what entries are going to be ignored.:

egrep -f /etc/logcheck/ignore.d.server/local-server /var/log/syslog

If you want to test only one line for instance:

egrep "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sensord:[[:space:]]*(temp[0-9]*|Core [0-9]*): .*$" /var/log/syslog

The result is the messages that will be suppressed. These messages will not be e-mailed to you so be careful not to be over zealous. If you're happy with the result, do another test run.:

sudo -u logcheck logcheck -m root -t -d

This time, we add the -d flag. It produces more output and should be an easier way of determining if our ignore file is used.:

-d        Debug mode.

An excerpt of the output:

D: [1523427703] Turning debug mode on
D: [1523427703] Sourcing - /etc/logcheck/logcheck.conf
D: [1523427703] Setting SENDMAILTO to root
D: [1523427703] Setting LOGTAIL_OPTS to -t
D: [1523427703] Finished getopts c:dhH:l:L:m:opr:RsS:tTuvw
D: [1523427703] Trying to get lockfile: /run/lock/logcheck/logcheck.lock
D: [1523427703] Running lockfile-touch /run/lock/logcheck/logcheck.lock
D: [1523427703] cleanrules: /etc/logcheck/cracking.d/kernel
D: [1523427703] cleanrules: /etc/logcheck/cracking.d/rlogind
D: [1523427703] cleanrules: /etc/logcheck/cracking.d/rsh
D: [1523427703] cleanrules: /etc/logcheck/cracking.d/smartd
D: [1523427703] cleanrules: /etc/logcheck/cracking.d/tftpd
D: [1523427703] cleanrules: /etc/logcheck/cracking.d/uucico
D: [1523427703] cleanrules: /etc/logcheck/violations.d/kernel
D: [1523427703] cleanrules: /etc/logcheck/violations.d/logcheck
D: [1523427703] cleanrules: /etc/logcheck/violations.d/smartd
D: [1523427703] cleanrules: /etc/logcheck/violations.d/su
D: [1523427703] cleanrules: /etc/logcheck/violations.d/sudo
D: [1523427703] cleanrules: /etc/logcheck/violations.ignore.d/local-server
D: [1523427703] cleanrules: /etc/logcheck/violations.ignore.d/logcheck-su
D: [1523427703] cleanrules: /etc/logcheck/violations.ignore.d/logcheck-sudo
D: [1523427703] cleanrules: /etc/logcheck/ignore.d.workstation/automount
D: [1523427703] cleanrules: /etc/logcheck/ignore.d.workstation/bind
D: [1523427703] cleanrules: /etc/logcheck/ignore.d.workstation/bluetooth-alsa
D: [1523427703] cleanrules: /etc/logcheck/ignore.d.workstation/bluez-utils
D: [1523427703] cleanrules: /etc/logcheck/ignore.d.workstation/bonobo
...
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.workstation/xdm
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.workstation/xlockmore
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/acpid
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/amandad
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/amavisd-new
...
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/libpam-krb5
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/libpam-mount
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/libsasl2-modules
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/local-server
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/logcheck
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/login
...
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/qpopper
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.server/rbldnsd
...
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.paranoid/bind
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.paranoid/cracklib-runtime
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.paranoid/cron
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.paranoid/incron
D: [1523427704] cleanrules: /etc/logcheck/ignore.d.paranoid/logcheck
...
D: [1523427704] logoutput called with file: /var/log/syslog
D: [1523427704] Running /usr/sbin/logtail2 on /var/log/syslog
D: [1523427704] logoutput called with file: /var/log/auth.log
D: [1523427704] Running /usr/sbin/logtail2 on /var/log/auth.log
D: [1523427705] Sorting logs
D: [1523427705] Setting the Intro
D: [1523427705] Checking for security alerts
D: [1523427705] greplogoutput: kernel
D: [1523427705] greplogoutput: rlogind
D: [1523427705] greplogoutput: rsh
D: [1523427705] greplogoutput: smartd
D: [1523427705] greplogoutput: tftpd
D: [1523427705] greplogoutput: uucico
D: [1523427705] greplogoutput: returning 1
D: [1523427705] Checking for security events
D: [1523427705] greplogoutput: kernel
D: [1523427705] greplogoutput: logcheck
D: [1523427705] greplogoutput: smartd
D: [1523427705] greplogoutput: su
D: [1523427705] greplogoutput: sudo
D: [1523427705] greplogoutput: Entries in checked
D: [1523427705] cleanchecked - file: /tmp/logcheck.vcRlSi/violations-ignore/logcheck-sudo
D: [1523427705] cleanchecked - file: /tmp/logcheck.vcRlSi/violations-ignore/local-workstation
D: [1523427705] greplogoutput: returning 1
D: [1523427705] Checking for system events
D: [1523427705] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore
D: [1523427705] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/acpid
D: [1523427705] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/amandad
...
D: [1523427706] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/libsasl2-modules
D: [1523427706] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/local-workstation
D: [1523427706] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/logcheck
...
D: [1523427707] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/xdm
D: [1523427707] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/xinetd
D: [1523427707] cleanchecked - dir - /tmp/logcheck.vcRlSi/ignore/xlockmore
D: [1523427707] Setting the footer text
D: [1523427707] cleanup: Killing lockfile-touch - 8427
D: [1523427707] cleanup: Removing lockfile: /run/lock/logcheck/logcheck.lock
D: [1523427707] cleanup: Removing - /tmp/logcheck.vcRlSi

Our ignore file is processed. Repeat the tuning and testing as many times as you need to minimize the reporting to the bare minimum but important log entries. If too many lines are reported, some entries are bound to be overlooked.

There is interesting info on the package specific ignore filters and how they interact with the non-package specific patterns. For instance, a line by a package is logged containing "rejected". If the standard keyword "reject" is listed in the generic "/etc/logcheck/violations.d/logcheck" file, a "Security Events" will be reported.

If you want to prevent this "reject" to trigger, you would probably thinlk about putting a filtering pattern in "/etc/logcheck/violations.ignore.d/<package>".

The solution is to use a file named in the specially-privileged ./logcheck-<packagename> format: "/etc/logcheck/violations.ignore.d/logcheck-fooserver". This file can contain patterns provided by that particular package which nonetheless need to take precedence over the generic rules.

For local SysAdmin usage, prefix your ignore pattern file with local- so you get "local-<package>".

Stricter rules

As mentioned before, you want the rules to match more strictly. For example, consider this line received as System event:

Apr 10 09:36:23 mymachine /usr/bin/x2goumount-session[1390]: successfully unmounted "/tmp/.x2go-user/spool/C-user-51-1523359618_stDXFCE_dp32"

First, we verify this is not something we should investigate further. It's not. Next, we search for a default ignore rule to use as an example:

cat /etc/logcheck/ignore.d.server/x2goserver

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /usr/bin/x2gomountdirs\[[ 0-9]{1,6}\]: WARNING: mounting of /.*/spool failed$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /usr/bin/x2gosuspend-session: session with ID .* has been suspended successfully$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /usr/bin/x2gostartagent: successfully started X2Go agent session with ID .*$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /usr/bin/x2goresume-session: client .* has successfully resumed session with ID .*$

This gives us a pretty good idea how to build stricter rules. Go to RexV.2, online regex tester or another regex tester, choose ere if possible, or pcre for regex flavour and paste the log line in the test string text area.

Note

You might have to escape the / path symbol with a (regex escape character). This is not going to be necessary in the ignore rule for logcheck so you can remove those escape characters from the tested regex.

In the regular expression text area, paste this rule we came up with base on the standard rules.:

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ \/usr\/bin\/x2goumount-session\[[ 0-9]{1,6}\]: successfully unmounted .*$

We get a match and put the rule without the escape chars in our previously made ignore file:

vi /etc/logcheck/ignore.d.server/local-server
...
# x2go umount
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /usr/bin/x2goumount-session\[[ 0-9]{1,6}\]: successfully unmounted .*$

Check if the rule matches:

egrep -f /etc/logcheck/ignore.d.server/local-server /var/log/syslog

An improved version of our ignore rulefile:

# SSSD
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sssd_be: GSSAPI client step 1.*$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sssd_be: GSSAPI client step 2.*$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sssd_be: message repeated [[:digit:]] times: \[ GSSAPI client step 1\].*$
# Postfix
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix\/local\[[0-9]+\]: warning: database /etc/aliases.db is older than source file.*$
# Kernel
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: \[[0-9]+\.[0-9]+\] perf: interrupt took too long \(.*\), lowering.*$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: \[[0-9]+\.[0-9]+\] CIFS VFS: Free previous.*$
# Kaspersky
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kesl: result_t klif::FanotifyInterceptor::UpdateMarks().*$
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kesl: result_t klif::FanotifyInterceptor::ResetCache().*$
# x2go
^\w{3} [ :0-9]{11} [._[:alnum:]-]+ /usr/bin/x2goumount-session\[[ 0-9]{1,6}\]: successfully unmounted .*$

Try a logcheck test:

sudo -u logcheck logcheck -m root -t -d

Cron

On Debian/Ubuntu, logcheck comes with a cron entry to run it after reboot and 2 minutes past every hour:

# /etc/cron.d/logcheck: crontab entries for the logcheck package

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

@reboot         logcheck    if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R; fi
2 * * * *       logcheck    if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi

# EOF

If you want another schedule, adjust this crontab entry.