Systemd although very good, has caused problems for logging (for me). Systemd does not play nice with logging applications such as Splunk. The reason for this is that systemd takes over syslog and stores all log data in its journald system which uses tmpfs (RAM) until flushed to disk in its proprietary format. This is to make log data more secure.
Here is how I got around that so that I could analyse my logs. It's pretty straight forward and isn't too involved.
Configure journald.conf
I have set the following options in my config and everything else is commented out.
The journald config above will now send everything to syslog which by default will store in /var/log/messages. To split out specific logs, you'll need to tell syslog what to do with them. Below is a basic config to split a few logs that I'm interested in. Some apps may use their own logger, so be aware of this.
# cat /etc/syslog-ng/syslog-ng.conf
@version: 3.7
# $Id$
#
# Syslog-ng default configuration file for Gentoo Linux
# https://bugs.gentoo.org/show_bug.cgi?id=426814
@include "scl.conf"
options {
threaded(yes);
chain_hostnames(no);
# The default action of syslog-ng is to log a STATS line
# to the file every 10 minutes. That's pretty ugly after a while.
# Change it to every 12 hours so you get a nice daily update of
# how many messages syslog-ng missed (0).
stats_freq(43200);
# The default action of syslog-ng is to log a MARK line
# to the file every 20 minutes. That's seems high for most
# people so turn it down to once an hour. Set it to zero
# if you don't want the functionality at all.
mark_freq(3600);
};
source src {
system();
internal();
};
destination messages { file("/var/log/messages"); };
# By default messages are logged to tty12...
destination console_all { file("/dev/tty12"); };
# ...if you intend to use /dev/console for programs like xconsole
# you can comment out the destination line above that references /dev/tty12
# and uncomment the line below.
#destination console_all { file("/dev/console"); };
# iptables log
destination firewall { file("/var/log/firewall.log"); };
filter f_firewall { program("iptables") or match("Dropped" value(MESSAGE)); };
log { source(src); filter(f_firewall); destination(firewall); flags(final); };
# ssh log
destination sshd { file("/var/log/sshd.log"); };
filter f_sshd { program("^sshd$"); };
log { source(src); filter(f_sshd); destination(sshd); flags(final); };
# named log
destination named { file("/var/log/named.log" owner(named) group(named) perm(0600) dir_perm(0700)); };
filter f_named { program("^named$"); };
log { source(src); filter(f_named); destination(named); flags(final); };
# dhcp log
destination dhcpd { file("/var/log/dhcpd.log"); };
filter f_dhcpd { program("^dhcpd$"); };
log { source(src); filter(f_dhcpd); destination(dhcpd); flags(final); };
# spamd log
destination spamd { file("/var/log/spamd.log"); };
filter f_spamd { program("^spamassassin$") or program("^/usr/sbin/spamd$"); };
log { source(src); filter(f_spamd); destination(spamd); flags(final); };
# ALWAYS AT THE END
log { source(src); destination(messages); };
log { source(src); destination(console_all); };
First thing is to ensure the correct options are set in the kernel.
IP sets support the following type of sets:
bitmap:ip
The bitmap:ip set type uses a memory range, where each bit represents one IP address and can store up to 65535 (B-class network) entries. You can store same size network addresses in this kind of sets as well and an IP address will be in the set if the network address it belongs to can be found in the set.
bitmap:ip,mac
The bitmap:ip,mac set type uses a memory range, where each 8 bytes represents one IP and a MAC addresses. A bitmap:ip,mac set type can store up to 65535 (B-class network) IP addresses with MAC.
bitmap:port
The bitmap:port set type uses a memory range, where each bit represents one TCP/UDP port. A bitmap:port type of set can store up to 65535 ports.
hash:ip
The hash:ip set type uses a hash to store IP addresses where clashing is resolved by storing the clashing elements in an array and, as a last resort, by dynamically growing the hash. Same size network addresses can be stored in an hash:ip type of set as well.
hash:net
The hash:net set type also uses a hash to store CIDR netblocks, which may be of different sizes. The same techique is used to avoid clashes as at the hash:ip set type.
hash:ip,port
The hash:ip,port is similar to hash:ip but you can store IP address and protocol-port pairs in it. TCP, SCTP, UDP, UDPLITE, ICMP and ICMPv6 are supported with port numbers/ICMP(v6) types and other protocol numbers without port information.
hash:ip,port,ip
You can store IP address, port number, and IP address triples in an hash:ip,port,ip type of set.
hash:ip,port,net
You can store IP address, port number and network address triples in this kind of set.
hash:net,port
The set type supports to store network address and port number pairs.
hash:net,iface
In this kind of set one can store network address and interface name pairs.
list:set
In a list:set kind of set you can store other sets; it is like an ordered union of different sets.
-*- Networking support --->
Networking options --->
[*] Network packet filtering framework (Netfilter) --->
<*> IP set support --->
(256) Maximum number of IP sets
<*> bitmap:ip set support
<*> bitmap:ip,mac set support
<*> bitmap:port set support
<*> hash:ip set support
<*> hash:ip,mark set support
<*> hash:ip,port set support
<*> hash:ip,port,ip set support
<*> hash:ip,port,net set support
<*> hash:ip,mac set support
<*> hash:mac set support
<*> hash:net,port,net set support
<*> hash:net set support
<*> hash:net,net set support
<*> hash:net,port set support
<*> hash:net,iface set support
<*> list:set set support
If you're having to now add this, you'll obviously need to reboot once you have built and installed the updated kernel.
Now let's install ipset.
# emerge -av ipset
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] net-firewall/ipset-6.29::gentoo USE="-modules"
Ipset has the modules flag set by default. With the above kernel config, the build will fail as I haven't used modules DUH! So just disable the flag. If you've build the kernel with modules instead, just leave as is.
So on to the fun stuff. Ipset has a pretty good help section and the manpage is detailed :)
I use systemd, so there is no service to restart for iptables or ipset.
Obviously you'll have iptables configured and running :) By default I block everything and only have specific ports open to the world. But that doesn't mean we don't need ipsets. I'm going to use ipset to block spiders (crawlers) that I don't want or that bypass my robots.txt. I could limit the blocking to specific ports eg 80 & 443, but I'm just going to block everything. Just because I can ;) I've also set a timeout (optional) so the IP's can be rotated and the sets don't grow too big.
So let's create an ipset. I only want to block CIDR, so I don't need to add port, mac etc etc.
ipset create netSpiders hash:net timeout 86400
You can add IPs in here also, but I like to keep things clean so I also have:
ipset create ipSpiders hash:ip timeout 86400
NOTE: If you haven't rebooted into the ipset enabled kernel, you'll get the error:
ipset v6.29: Kernel error received: set type not supported
You can list the set and any rules within by executing:
# ipset list
Name: spiders
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 86400
Size in memory: 368
References: 0
Number of entries: 0
Members:
As you can see, we have nothing defined yet. So let's add some spiders. Unfortunately it's per CIDR.
su - apache -c "php /path/to/nextcloud/occ twofactorauth:disable username"
This will disable 2FA regardless of it it is enabled in the GUI. You will need to enable 2FA by running the command again, but with enable instead.
Delete the undeleteables.
I had a situation where a user had ~30Gb of data in their nextcloud trash which they could not delete. There were also files/directories in their active files that they also could not delete. No errors in the logs, just unable to delete :( The work around is to manually delete them from the server and then run the following command.
su - apache -c "php /path/to/nextcloud/occ files:scan <user>"
This removes invalid file references from the database.
Here I will show how to configure VSFTPD for basic authentication so that we have a base working daemon. Then we will build on that by implementing SSL and then virtual users.
Obviously first thing is first ;) If you haven't already, install vsftpd.
The above will tell vsftpd to not allow any anonymous connections eg mandatory login. What and where to log. What IP and port to listen on. To lock users into their home directory (defined in /etc/passwd).
The bottom 3 lines userlist_* we define so we don't grant all local users ftp access. If you answer YES to userlist_deny, the user list will deny any users listed in the file and allow everything else.
$ ftp cdstealer.com
Connected to cdstealer.com (<IP>).
220 (vsFTPd 3.0.2)
Name (cdstealer.com:user1):
530 Please login with USER and PASS.
SSL not available
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
SSL
So let's get some encryption so we aren't transmitting plain text credentials.
Generate a self signed cert.
# openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout /etc/ssl/apache2/vsftp.pem -out /etc/ssl/apache2/vsftp.pem
Generating a 4096 bit RSA private key
..........................................................................................++
....++
writing new private key to '/etc/ssl/apache2/vsftp.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:GB
State or Province Name (full name) [Some-State]:Somewhere nice
Locality Name (eg, city) []:Leeds
Organization Name (eg, company) [Internet Widgits Pty Ltd]:cdstealer.com
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:cdstealer.com
Email Address []:
The openssl command I've used, generates a 4096bit encrypted cert (this is good) that is valid for 10 years. :)
Execute the following to remove unwanted access to the cert.
chmod 600 /etc/ssl/apache2/vsftp.pem
Add this section to your /etc/vsftpd/vsftpd.conf file.
Let's test and make sure everything is still working.
$ ftp cdstealer.com
Connected to cdstealer.com (<IP>).
220 (vsFTPd 3.0.2)
Name (cdstealer.com:user1):
234 Proceed with negotiation.
[SSL Cipher AES128-SHA]
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
GREAT! So now we have a secure FTP service running.
In the interest of security, I'm not comfortable having system user accounts, even though we have secured things with only specific users able to ftp. I believe that standard ftp authentication does not support encryption for system users, but does for virtual users?
This is a little more involved than having standard system users :(
So, add the following to your /etc/vsftpd/vsftpd.conf file.
Here we define that virtual users get the same permissions as the ftp system user, are unable to browse outside their directory, use a PAM database for credentials and define custom settings per user.
NOTE: You may notice that the file extension is missing from the path in /etc/pam.d/vsftpd. This is intentional as PAM automatically adds the .db suffix.
You will need to add the users from the database to the access list file.
/etc/vsftpd/vsftpd.user_list
For your convenience, I've written a user management script, here :)
Define custom settings.
Create the directory which will store the configs. We defined this earlier as user_config_dir=/etc/vsftpd/virtualUsers.
mkdir /etc/vsftpd/virtualUsers
Create a user config.
vi /etc/vsftpd/virtualUsers/user1
write_enable=NO
local_root=/FTP/user1
chroot_local_user=YES
dirlist_enable=YES
download_enable=YES
Here we set the users CHROOT directory, deny write permissions and allow downloading. Options here (not all) override specific options defined in the main config.
If you get the following when logging in or listing a directory, it maybe due to the user directory not existing or not having permission.
ssl_getc: SSL_read failed -1 = 0
421 Service not available, remote server has closed connection
Login failed.
No control connection for command: Success
The following error may occur on ftp clients with vsftpd 3.0.x:
500 OOPS: priv_sock_get_cmd
This is caused by seccomp filter sanboxing, which is enabled by default on amd64. To workaround this issue, disable seccomp filter sanboxing:
Add the following line to /etc/vsftpd/vsftpd.conf.
Want to batch download a bunch of videos in one swoop? Here's how. This also works with individual videos :)
Step 1) Install youtube-dl
# emerge -av youtube-dl
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] net-misc/youtube-dl-2016.12.22::gentoo USE="offensive {-test}" PYTHON_TARGETS="python2_7 python3_4 (-python3_5)"
This is more of an audit thing, but sometimes it's useful to know when you did something.
If you type the command history into your shell. You get back a list of the last X number of command executed.
$ history | tail -n1
502 history | tail -n1
But by adding the following to your ~/.bashrc (local user) or /etc/bash/bashrc (all users) file, you can inject the time/date. Please see man date for the options you can use.
HISTTIMEFORMAT='%F %T '
Example
$ history | tail -n1
508 2017-01-14 11:29:58 history | tail -n1
Bash History Duplicate Removal
It's an annoyance when you execute a command consecutively many time and then have to search further back to get to the last different command. Behold! Add this to your .bashrc(local user) or /etc/bash/bashrc (all users) and no matter how many times you execute that command consecutively, it will only store the one time.
HISTCONTROL=ignoreboth
Example
user@server ~ $ vi .bashrc
user@server ~ $ vi .bashrc
user@server ~ $ vi .bashrc
user@server ~ $ vi .bashrc
user@server ~ $ history | tail -n5
508 2017-01-14 11:29:58 history | tail -n1
509 2017-01-14 11:31:36 man date
510 2017-01-14 11:32:43 history
511 2017-01-14 11:40:52 vi .bashrc
512 2017-01-14 11:42:36 history | tail -n5
Double tap exit/logoff
Accidentally logging out of a shell session or user session can be annoying. But there is hope :) Add this to your .bashrc(local user) or /etc/bash/bashrc (all users) and now you have to double tap to get out.
This one ideally needs to go into the /etc/bash/bashrc or else it would only work when closing your own shell session.
IGNOREEOF=1
Example
server ~ # Use "logout" to leave the shell.
server ~ # logout
user@server ~ $
Extract Path or File
$ export VAR=/home/me/mydir/file.c
$ echo "${VAR%/*}" /home/me/mydir
$ echo "${VAR##*/}" file.c
Cron Execution Only
Pop this at the top of a bash script.
Eg. The below is for a cronjob that is executed at reboot:
The "!senders = +whitelist_senders" line will lookup against the file you created. It will also create an empty file within the path of the first line of this section for the time based rejection. So to keep things "tidy", we'll run a cronjob every 30 mins to remove files.
Want Vim to automatically do things to new or existing files? Autocmd to the rescue! Here I will demo a very simple example that I use for when I create/edit bash/python files.
I created the directory .vim in the root of my home directory and then created the "header" files that I wanted.
$ cat .vim/python_header
:insert
#!/usr/bin/python3
################################################################################
####
#### Filename:
####
#### Purpose:
####
#### Created on:
####
#### Author:
####
#### Last Modified:
####
################################################################################
.
The first line must be :insert and the last line must be a period '.'
I then added the following lines to the .vimrc file in the root of my home directory.
