Tag Archives: systemd

Systemd & Syslog-ng

Systemd although very good, has caused problems for logging (for me).  Systemd does not play nice with logging applications such as Splunk.  The reason for this is that systemd takes over syslog and stores all log data in its journald system which uses tmpfs (RAM) until flushed to disk in its proprietary  format.  This is to make log data more secure.

Here is how I got around that so that I could analyse my logs.  It's pretty straight forward and isn't too involved.

Configure journald.conf

I have set the following options in my config and everything else is commented out.

# cat /etc/systemd/journald.conf 

[Journal]
Storage=volatile
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=no

Configure Syslog-ng

The journald config above will now send everything to syslog which by default will store in /var/log/messages.  To split out specific logs, you'll need to tell syslog what to do with them.  Below is a basic config to split a few logs that I'm interested in.  Some apps may use their own logger, so be aware of this.

# cat /etc/syslog-ng/syslog-ng.conf 
@version: 3.7
# $Id$
#
# Syslog-ng default configuration file for Gentoo Linux

# https://bugs.gentoo.org/show_bug.cgi?id=426814
@include "scl.conf"

options { 
 threaded(yes);
 chain_hostnames(no); 

 # The default action of syslog-ng is to log a STATS line
 # to the file every 10 minutes. That's pretty ugly after a while.
 # Change it to every 12 hours so you get a nice daily update of
 # how many messages syslog-ng missed (0).
 stats_freq(43200); 
 # The default action of syslog-ng is to log a MARK line
 # to the file every 20 minutes. That's seems high for most
 # people so turn it down to once an hour. Set it to zero
 # if you don't want the functionality at all.
 mark_freq(3600); 
};

source src {
 system();
 internal();
};

destination messages { file("/var/log/messages"); };

# By default messages are logged to tty12...
destination console_all { file("/dev/tty12"); };
# ...if you intend to use /dev/console for programs like xconsole
# you can comment out the destination line above that references /dev/tty12
# and uncomment the line below.
#destination console_all { file("/dev/console"); };

# iptables log
destination firewall { file("/var/log/firewall.log"); };
filter f_firewall { program("iptables") or match("Dropped" value(MESSAGE)); };
log { source(src); filter(f_firewall); destination(firewall); flags(final); };

# ssh log
destination sshd { file("/var/log/sshd.log"); };
filter f_sshd { program("^sshd$"); }; 
log { source(src); filter(f_sshd); destination(sshd); flags(final); };

# named log
destination named { file("/var/log/named.log" owner(named) group(named) perm(0600) dir_perm(0700)); };
filter f_named { program("^named$"); }; 
log { source(src); filter(f_named); destination(named); flags(final); };

# dhcp log
destination dhcpd { file("/var/log/dhcpd.log"); };
filter f_dhcpd { program("^dhcpd$"); }; 
log { source(src); filter(f_dhcpd); destination(dhcpd); flags(final); };

# spamd log
destination spamd { file("/var/log/spamd.log"); };
filter f_spamd { program("^spamassassin$") or program("^/usr/sbin/spamd$"); }; 
log { source(src); filter(f_spamd); destination(spamd); flags(final); };

# ALWAYS AT THE END
log { source(src); destination(messages); };
log { source(src); destination(console_all); };

Kodi with Systemd

kodi-logo-weis3
Kodi As An Appliance

After installing a base gentoo OS (no gui)
install evilvm and kodi (or use upstream kodi)

If using upstream kodi, create the file: /etc/portage/sets/kodi and populate with: (adjust accordingly)

app-eselect/eselect-java
dev-java/java-config
dev-java/openjdk-bin
dev-lang/swig
dev-libs/crossguid
dev-libs/flatbuffers
dev-libs/libcdio
dev-libs/libfmt
dev-libs/libfstrcmp
dev-libs/libinput
dev-libs/libtomcrypt
dev-libs/libtommath
dev-libs/rapidjson
dev-libs/spdlog
dev-libs/tinyxml
dev-python/cffi
dev-python/olefile
dev-python/pillow
dev-python/pycparser
dev-python/pycryptodome
dev-python/xkbcommon
dev-vcs/git
media-fonts/roboto
media-libs/intel-hybrid-codec-driver
media-libs/libass
media-libs/libdisplay-info
media-libs/libdisplay-info
media-libs/libdvdcss
media-libs/libdvdnav
media-libs/libdvdread
media-libs/libva-intel-driver
media-libs/libva-intel-media-driver
media-libs/taglib
media-sound/pulseaudio
net-fs/nfs-utils
net-fs/samba
net-libs/libmicrohttpd
sci-libs/kissfft
sys-apps/baselayout-java
sys-devel/clang
=llvm-core/clang-17.0.6
=llvm-core/llvm-17.0.6
x11-wm/evilwm

install the deps with: emerge -quUND @kodi

Ensure you have all the correct use flags defined in /etc/portage/package.use/package.use

Create the file: /usr/lib/systemd/system/kodi.service and populate with:

[Unit]
Description = Starts instance of Kodi
After = systemd-user-sessions.service network.target sound.target startx.service

[Service]
User = kodi
Group = kodi
PAMName=login
Type = simple
ExecStart = sh /usr/bin/kodi-standalone2 -- :0 -nolisten tcp
Restart = on-abort

[Install]
WantedBy = multi-user.target

Then create the file: /usr/lib/systemd/system/startx.service and populate with:


[Unit]
Description=StartX service

[Service]
User = kodi
Group = kodi
PAMName=login
Type = simple
ExecStart=/usr/bin/startx :0 /usr/bin/evilwm

[Install]
WantedBy=multi-user.target

Only enable the kodi server: systemctld enable kodi

If you have Intel GFX, create the file: /etc/X11/xorg.conf.d/20-intel.conf and populate with: (adjust accordingly)

Section "Module"
Load "glx"
EndSection

Section "Device"
Identifier "Card0"
Driver "modesetting"
#BusID "PCI:0:2:0"
Option "TripleBuffer" "false"
Option "TearFree" "true"
Option "SwapbuffersWait" "true"
Option "AccelMethod" "glamor"
Option "DRI" "iris"
EndSection

Section "dri"
Mode 0666
EndSection

#Breaks Gnome when set to false, but breaks kodi if enabled
Section "Extensions"
Option "Composite" "False"
EndSection

Create the kodi user and assign to the following groups:

useradd -G tty,audio,video,render,pipewire kodi

If everything went to plan, executing: systemctl start kodi should fire up X and land you at kodi.

Splunk & Systemd

No longer required from Splunk version 7.2.2 onwards

Universal Forwarder

Copy this text into a file named "/usr/lib/systemd/system/splunkforwarder.service".

[Unit]
Description=Splunk Universal Forwarder
Wants=network.target
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
ExecStart=/opt/splunkforwarder/bin/splunk start
ExecStop=/opt/splunkforwarder/bin/splunk stop
ExecReload=/opt/splunkforwarder/bin/splunk restart
StandardOutput=syslog
Restart=always

[Install]
WantedBy=multi-user.target

Next, create a symlink that systemd will use to fetch this control file.

systemctl enable splunkforwarder

That's it.  You can either reboot or run

systemctl start splunkforwarder

SearchHead, Indexer, Heavy Forwarder

Copy this text into a file named "/usr/lib/systemd/system/splunkforwarder.service".

[Unit]
Description=Splunk
Wants=network.target
After=network.target

[Service]
User=splunk
Group=splunk
Type=forking
RemainAfterExit=yes
ExecStart=/opt/splunk/bin/splunk start --answer-yes --no-prompt --accept-license
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
StandardOutput=syslog
#ExecStart=/bin/sh -c "echo never >/sys/kernel/mm/transparent_hugepage/enabled"
#ExecStart=/bin/sh -c "echo never >/sys/kernel/mm/transparent_hugepage/defrag"
#ulimit -Sn 65535
#ulimit -Hn 65535
LimitNOFILE=65535
#ulimit -Su 20480
#ulimit -Hu 20480
LimitNPROC=20480
#ulimit -Hf unlimited
#ulimit -Sf unlimited
LimitFSIZE=infinity
LimitCORE=infinity
#Restart=always

[Install]
WantedBy=multi-user.target
If you have THP compiled into your kernel (hint: it's enabled by default), then you'll need to uncomment the two ExecStart lines