Systemd & Syslog-ng

Systemd although very good, has caused problems for logging (for me).  Systemd does not play nice with logging applications such as Splunk.  The reason for this is that systemd takes over syslog and stores all log data in its journald system which uses tmpfs (RAM) until flushed to disk in its proprietary  format.  This is to make log data more secure.

Here is how I got around that so that I could analyse my logs.  It's pretty straight forward and isn't too involved.

Configure journald.conf

I have set the following options in my config and everything else is commented out.

# cat /etc/systemd/journald.conf 

[Journal]
Storage=volatile
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=no

Configure Syslog-ng

The journald config above will now send everything to syslog which by default will store in /var/log/messages.  To split out specific logs, you'll need to tell syslog what to do with them.  Below is a basic config to split a few logs that I'm interested in.  Some apps may use their own logger, so be aware of this.

# cat /etc/syslog-ng/syslog-ng.conf 
@version: 3.7
# $Id$
#
# Syslog-ng default configuration file for Gentoo Linux

# https://bugs.gentoo.org/show_bug.cgi?id=426814
@include "scl.conf"

options { 
 threaded(yes);
 chain_hostnames(no); 

 # The default action of syslog-ng is to log a STATS line
 # to the file every 10 minutes. That's pretty ugly after a while.
 # Change it to every 12 hours so you get a nice daily update of
 # how many messages syslog-ng missed (0).
 stats_freq(43200); 
 # The default action of syslog-ng is to log a MARK line
 # to the file every 20 minutes. That's seems high for most
 # people so turn it down to once an hour. Set it to zero
 # if you don't want the functionality at all.
 mark_freq(3600); 
};

source src {
 system();
 internal();
};

destination messages { file("/var/log/messages"); };

# By default messages are logged to tty12...
destination console_all { file("/dev/tty12"); };
# ...if you intend to use /dev/console for programs like xconsole
# you can comment out the destination line above that references /dev/tty12
# and uncomment the line below.
#destination console_all { file("/dev/console"); };

# iptables log
destination firewall { file("/var/log/firewall.log"); };
filter f_firewall { program("iptables") or match("Dropped" value(MESSAGE)); };
log { source(src); filter(f_firewall); destination(firewall); flags(final); };

# ssh log
destination sshd { file("/var/log/sshd.log"); };
filter f_sshd { program("^sshd$"); }; 
log { source(src); filter(f_sshd); destination(sshd); flags(final); };

# named log
destination named { file("/var/log/named.log" owner(named) group(named) perm(0600) dir_perm(0700)); };
filter f_named { program("^named$"); }; 
log { source(src); filter(f_named); destination(named); flags(final); };

# dhcp log
destination dhcpd { file("/var/log/dhcpd.log"); };
filter f_dhcpd { program("^dhcpd$"); }; 
log { source(src); filter(f_dhcpd); destination(dhcpd); flags(final); };

# spamd log
destination spamd { file("/var/log/spamd.log"); };
filter f_spamd { program("^spamassassin$") or program("^/usr/sbin/spamd$"); }; 
log { source(src); filter(f_spamd); destination(spamd); flags(final); };

# ALWAYS AT THE END
log { source(src); destination(messages); };
log { source(src); destination(console_all); };