For a while now I'd been thinking about how to block spam that is mascarading as myself. Today, I fixed it :) In my exim.conf, I have a section for spam filtering under the "acl_check_data" section.
I also added the following under the acl_check_helo:
# If the HELO pretend to be this host
deny condition = ${if or { \
{eq {${lc:$sender_helo_name}}{cdstealer.com}} \
{eq {${lc:$sender_helo_name}}{127.0.0.1}} \
{eq {${lc:$sender_helo_name}}{198.16.181.0/24}} \
} {true}{false} }
So, a situation may arise where you are required to install Windows where the only media you have is a USB drive. Now if you're already running Windows, you can do this with the media creation tool.
You can download a Windows 10 image direct from M$ here.
Obviously, we're running Linux ;) So download the ISO image and let's get cracking ;)
USB Prepare
You will need a drive >4Gb (I used a 2Gb :\)
Insert your USB and execute (where 'X' is the drive eg /dev/sda):
fdisk /dev/sdX
I'm assuming there are no partitions defined. So we'll start from scratch.
# fdisk /dev/sdj
Welcome to fdisk (util-linux 2.30.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdj: 1.9 GiB, 2058354688 bytes, 4020224 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0003dd8d
-------------------------------------------------------------------------------
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-4020223, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-4020223, default 4020223):
Created a new partition 1 of type 'Linux' and of size 1.9 GiB.
-------------------------------------------------------------------------------
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 7
Changed type of partition 'Linux' to 'HPFS/NTFS/exFAT'.
-------------------------------------------------------------------------------
Command (m for help): a
Selected partition 1
The bootable flag on partition 1 is enabled now.
-------------------------------------------------------------------------------
Command (m for help): p
Disk /dev/sdj: 1.9 GiB, 2058354688 bytes, 4020224 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0003dd8d
Device Boot Start End Sectors Size Id Type
/dev/sdj1 * 2048 4020223 4018176 1.9G 7 HPFS/NTFS/exFAT
-------------------------------------------------------------------------------
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Make sure you have sys-fs/ntfs3g installed.
Execute
mkntfs -Q /dev/sdj1
# mkntfs -Q /dev/sdj1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.
Install boot sector
Make sure you have ms-sys installed.
Execute ms-sys -n /dev/sdj1
# ms-sys -n /dev/sdj1
NTFS Windows 7 boot record successfully written to /dev/sdj1
Now for the installation files.
Mount the downloaded ISO and the USB drive.
mount -o loop /$HOME/Downloads/Win10_1709_English_x64.iso /mnt/THIS
mount /dev/sdj1 /mnt/THIS
Now copy the files.
cp -Rv /mnt/THIS/* /run/media///
And that's it. Unmount the ISO/USB and go install.
Knocking up a simple and quick web server can be extremely useful without the need to install and configure a full fat web server. However, this isn't secure and will expose ALL files from the current directory executed in!
The official doc can be found here (v2) and here (v3).
NOTE: You will need to generate the certs for HTTPS version.
Just because! So sometimes you may need to patch a kernel source file, or a script, or maybe even your life who knows. But whatever the reason, creating a patch file and applying it are extremely easy.
Let's take the config from this post as an example.
Copy the file /etc/grub.d/10_linux to /etc/grub.d/10_linux_patched.
Open /etc/grub.d/10_linux_patched in your favourite editor and make the required changes (line 204), then save.
Original:
initrd=
for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \
"initrd-${version}" "initramfs-${version}.img" \
"initrd.img-${alt_version}" "initrd-${alt_version}.img" \
"initrd-${alt_version}" "initramfs-${alt_version}.img" \
"initramfs-genkernel-${version}" \
"initramfs-genkernel-${alt_version}" \
"initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \
"initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}"; do
if test -e "${dirname}/${i}" ; then
initrd="$i"
break
fi
done
Patched:
initrd=
for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \
"initrd-${version}" "initramfs-${version}.img" \
"initrd.img-${alt_version}" "initrd-${alt_version}.img" \
"initrd-${alt_version}" "initramfs-${alt_version}.img" \
"initramfs-genkernel-${version}" \
"initramfs-genkernel-${alt_version}" \
"initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \
"initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}"; do
if test -e "${dirname}/${i}" ; then
initrd="early_ucode.cpio ${rel_dirname}/$i"
break
else
initrd="early_ucode.cpio"
fi
done
If you just run a diff on the files, you see the changes.
The filenames that precede the '---' & '+++' are the files to be read '+++' and the file to be changed '---'. These 2 lines may also have a full or partial path to the files.
To apply the patch, just execute: patch < 10_linux.patch
If you now diff the 2 files, they should match ;)
So if not applying in the root of the files, we need to inform patch to omit the preceding segments.
example:
If the first 2 lines of the patch have a relative path.
Here we will get the firmware for the CPU loaded at boot. This helps the stability and efficiency of the processor.
Let's make sure we have the required options enabled in the kernel and rebuild if not.
General setup --->
[*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
Processor type and features --->
<*> CPU microcode loading support
[*] Intel microcode loading support
If these were not enabled, enable them as above (not modules!) and recompile and install your kernel.
Next we'll install 2 packages required for this to work.
# emerge -av intel-microcode iucode_tool
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] sys-firmware/intel-microcode-20161104::gentoo USE="split-ucode -initramfs -monolithic" 0 KiB
[ebuild R ] sys-apps/iucode_tool-2.1.1::gentoo 0 KiB
Mount your /boot filesystem and execute the following to generate the cpio archive.
You should see output like this with a different HEX value.
iucode_tool: system has processor(s) with signature 0x000206a7
iucode_tool: Writing selected microcodes to: /boot/early_ucode.cpio
Now we'll configure grub2 and install the new boot config.
Open the file /etc/grub.d/10_linux in your favourite editor and (at line ~204) make the following change.
From this:
initrd=for i in "initrd.img-${version}""initrd-${version}.img""initrd-${version}.gz"\"initrd-${version}""initramfs-${version}.img"\"initrd.img-${alt_version}""initrd-${alt_version}.img"\"initrd-${alt_version}""initramfs-${alt_version}.img"\"initramfs-genkernel-${version}"\"initramfs-genkernel-${alt_version}"\"initramfs-genkernel-${GENKERNEL_ARCH}-${version}"\"initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}";doiftest -e "${dirname}/${i}";theninitrd="$i"breakfidone
To:
initrd=
for i in "initrd.img-${version}" "initrd-${version}.img" "initrd-${version}.gz" \
"initrd-${version}" "initramfs-${version}.img" \
"initrd.img-${alt_version}" "initrd-${alt_version}.img" \
"initrd-${alt_version}" "initramfs-${alt_version}.img" \
"initramfs-genkernel-${version}" \
"initramfs-genkernel-${alt_version}" \
"initramfs-genkernel-${GENKERNEL_ARCH}-${version}" \
"initramfs-genkernel-${GENKERNEL_ARCH}-${alt_version}"; do
if test -e "${dirname}/${i}" ; then
initrd="early_ucode.cpio ${rel_dirname}/$i" break else initrd="early_ucode.cpio" fi done
Edit /etc/default/grub (at line ~59)
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to kernel
GRUB_DISABLE_LINUX_UUID=true
And last off, install the new grub config:
grub-mkconfig -o /boot/grub/grub.cfg
And that is it!
If there is an update to sys-firmware/intel-microcode from this point that contains firmware for your CPU, just mount your boot FS and execute the iucode_tool command to generate a new cpio archive.
As always, first thing is first ;) Make sure you have the correct kernel options enabled.
File systems --->
[*] Dnotify support
[*] Network File Systems --->
<*> NFS client support
<*> NFS client support for NFS version 4
[*] NFS client support for NFSv4.1
<*> NFS server support
[*] NFS server support for NFS version 4
[*] NFSv4.1 server support for Parallel NFS (pNFS)
Depending on other options enabled in the kernel, version 3 may be auto selected and non-optional.
Once compiled and built. We can install the required apps. Make sure you enable to required USE flags.
emerge -av net-fs/nfs-utils
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ~] net-fs/nfs-utils-2.1.1::gentoo USE="libmount nfsidmap nfsv4 nfsv41 tcpd uuid -caps -ipv6 -kerberos -nfsdcld (-selinux)"
SERVER:
Now we'll create the directory structure. The first line is the virtual root followed by each filesystem.
Systemd although very good, has caused problems for logging (for me). Systemd does not play nice with logging applications such as Splunk. The reason for this is that systemd takes over syslog and stores all log data in its journald system which uses tmpfs (RAM) until flushed to disk in its proprietary format. This is to make log data more secure.
Here is how I got around that so that I could analyse my logs. It's pretty straight forward and isn't too involved.
Configure journald.conf
I have set the following options in my config and everything else is commented out.
The journald config above will now send everything to syslog which by default will store in /var/log/messages. To split out specific logs, you'll need to tell syslog what to do with them. Below is a basic config to split a few logs that I'm interested in. Some apps may use their own logger, so be aware of this.
# cat /etc/syslog-ng/syslog-ng.conf
@version: 3.7
# $Id$
#
# Syslog-ng default configuration file for Gentoo Linux
# https://bugs.gentoo.org/show_bug.cgi?id=426814
@include "scl.conf"
options {
threaded(yes);
chain_hostnames(no);
# The default action of syslog-ng is to log a STATS line
# to the file every 10 minutes. That's pretty ugly after a while.
# Change it to every 12 hours so you get a nice daily update of
# how many messages syslog-ng missed (0).
stats_freq(43200);
# The default action of syslog-ng is to log a MARK line
# to the file every 20 minutes. That's seems high for most
# people so turn it down to once an hour. Set it to zero
# if you don't want the functionality at all.
mark_freq(3600);
};
source src {
system();
internal();
};
destination messages { file("/var/log/messages"); };
# By default messages are logged to tty12...
destination console_all { file("/dev/tty12"); };
# ...if you intend to use /dev/console for programs like xconsole
# you can comment out the destination line above that references /dev/tty12
# and uncomment the line below.
#destination console_all { file("/dev/console"); };
# iptables log
destination firewall { file("/var/log/firewall.log"); };
filter f_firewall { program("iptables") or match("Dropped" value(MESSAGE)); };
log { source(src); filter(f_firewall); destination(firewall); flags(final); };
# ssh log
destination sshd { file("/var/log/sshd.log"); };
filter f_sshd { program("^sshd$"); };
log { source(src); filter(f_sshd); destination(sshd); flags(final); };
# named log
destination named { file("/var/log/named.log" owner(named) group(named) perm(0600) dir_perm(0700)); };
filter f_named { program("^named$"); };
log { source(src); filter(f_named); destination(named); flags(final); };
# dhcp log
destination dhcpd { file("/var/log/dhcpd.log"); };
filter f_dhcpd { program("^dhcpd$"); };
log { source(src); filter(f_dhcpd); destination(dhcpd); flags(final); };
# spamd log
destination spamd { file("/var/log/spamd.log"); };
filter f_spamd { program("^spamassassin$") or program("^/usr/sbin/spamd$"); };
log { source(src); filter(f_spamd); destination(spamd); flags(final); };
# ALWAYS AT THE END
log { source(src); destination(messages); };
log { source(src); destination(console_all); };
First thing is to ensure the correct options are set in the kernel.
IP sets support the following type of sets:
bitmap:ip
The bitmap:ip set type uses a memory range, where each bit represents one IP address and can store up to 65535 (B-class network) entries. You can store same size network addresses in this kind of sets as well and an IP address will be in the set if the network address it belongs to can be found in the set.
bitmap:ip,mac
The bitmap:ip,mac set type uses a memory range, where each 8 bytes represents one IP and a MAC addresses. A bitmap:ip,mac set type can store up to 65535 (B-class network) IP addresses with MAC.
bitmap:port
The bitmap:port set type uses a memory range, where each bit represents one TCP/UDP port. A bitmap:port type of set can store up to 65535 ports.
hash:ip
The hash:ip set type uses a hash to store IP addresses where clashing is resolved by storing the clashing elements in an array and, as a last resort, by dynamically growing the hash. Same size network addresses can be stored in an hash:ip type of set as well.
hash:net
The hash:net set type also uses a hash to store CIDR netblocks, which may be of different sizes. The same techique is used to avoid clashes as at the hash:ip set type.
hash:ip,port
The hash:ip,port is similar to hash:ip but you can store IP address and protocol-port pairs in it. TCP, SCTP, UDP, UDPLITE, ICMP and ICMPv6 are supported with port numbers/ICMP(v6) types and other protocol numbers without port information.
hash:ip,port,ip
You can store IP address, port number, and IP address triples in an hash:ip,port,ip type of set.
hash:ip,port,net
You can store IP address, port number and network address triples in this kind of set.
hash:net,port
The set type supports to store network address and port number pairs.
hash:net,iface
In this kind of set one can store network address and interface name pairs.
list:set
In a list:set kind of set you can store other sets; it is like an ordered union of different sets.
-*- Networking support --->
Networking options --->
[*] Network packet filtering framework (Netfilter) --->
<*> IP set support --->
(256) Maximum number of IP sets
<*> bitmap:ip set support
<*> bitmap:ip,mac set support
<*> bitmap:port set support
<*> hash:ip set support
<*> hash:ip,mark set support
<*> hash:ip,port set support
<*> hash:ip,port,ip set support
<*> hash:ip,port,net set support
<*> hash:ip,mac set support
<*> hash:mac set support
<*> hash:net,port,net set support
<*> hash:net set support
<*> hash:net,net set support
<*> hash:net,port set support
<*> hash:net,iface set support
<*> list:set set support
If you're having to now add this, you'll obviously need to reboot once you have built and installed the updated kernel.
Now let's install ipset.
# emerge -av ipset
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] net-firewall/ipset-6.29::gentoo USE="-modules"
Ipset has the modules flag set by default. With the above kernel config, the build will fail as I haven't used modules DUH! So just disable the flag. If you've build the kernel with modules instead, just leave as is.
So on to the fun stuff. Ipset has a pretty good help section and the manpage is detailed :)
I use systemd, so there is no service to restart for iptables or ipset.
Obviously you'll have iptables configured and running :) By default I block everything and only have specific ports open to the world. But that doesn't mean we don't need ipsets. I'm going to use ipset to block spiders (crawlers) that I don't want or that bypass my robots.txt. I could limit the blocking to specific ports eg 80 & 443, but I'm just going to block everything. Just because I can ;) I've also set a timeout (optional) so the IP's can be rotated and the sets don't grow too big.
So let's create an ipset. I only want to block CIDR, so I don't need to add port, mac etc etc.
ipset create netSpiders hash:net timeout 86400
You can add IPs in here also, but I like to keep things clean so I also have:
ipset create ipSpiders hash:ip timeout 86400
NOTE: If you haven't rebooted into the ipset enabled kernel, you'll get the error:
ipset v6.29: Kernel error received: set type not supported
You can list the set and any rules within by executing:
# ipset list
Name: spiders
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 86400
Size in memory: 368
References: 0
Number of entries: 0
Members:
As you can see, we have nothing defined yet. So let's add some spiders. Unfortunately it's per CIDR.
su - apache -c "php /path/to/nextcloud/occ twofactorauth:disable username"
This will disable 2FA regardless of it it is enabled in the GUI. You will need to enable 2FA by running the command again, but with enable instead.
Delete the undeleteables.
I had a situation where a user had ~30Gb of data in their nextcloud trash which they could not delete. There were also files/directories in their active files that they also could not delete. No errors in the logs, just unable to delete :( The work around is to manually delete them from the server and then run the following command.
su - apache -c "php /path/to/nextcloud/occ files:scan <user>"
This removes invalid file references from the database.