All posts by cdstealer

Nextcloud

Just a few little tips I've had to use.

Disable 2FA on CLI.
su - apache -c "php /path/to/nextcloud/occ twofactorauth:disable username"

This will disable 2FA regardless of it it is enabled in the GUI.  You will need to enable 2FA by running the command again, but with enable instead.

Delete the undeleteables.

I had a situation where a user had ~30Gb of data in their nextcloud trash which they could not delete.  There were also files/directories in their active files that they also could not delete.  No errors in the logs, just unable to delete :(  The work around is to manually delete them from the server and then run the following command.

su - apache -c "php /path/to/nextcloud/occ files:scan <user>"

This removes invalid file references from the database.

 

VSFTPD

Here I will show how to configure VSFTPD for basic authentication so that we have a base working daemon.  Then we will build on that by implementing SSL and then virtual users.

Obviously first thing is first ;)  If you haven't already, install vsftpd.

emerge -av vsftpd
...
net-ftp/vsftpd-3.0.2-r1::gentoo USE="pam ssl tcpd -caps (-selinux) -xinetd"

We want to enable pam and ssl for later on.  Once done, pop this into /etc/vsftpd/vsftpd.conf

##################
# Basic Settings #
##################
anonymous_enable=NO
dirmessage_enable=YES
xferlog_enable=YES
vsftpd_log_file=/var/log/vsftpd.log
connect_from_port_20=YES
xferlog_file=/var/log/vsftpd.log
data_connection_timeout=120
nopriv_user=ftp
ftpd_banner=Insert Welcome Message
listen=YES
listen_address=<server IP>
listen_port=21

###############
# Local Users #
###############
local_enable=YES
write_enable=YES
chroot_local_user=YES
passwd_chroot_enable=YES
allow_writeable_chroot=NO
userlist_enable=YES
userlist_deny=NO
userlist_file=/etc/vsftpd/vsftpd.user_list

The above will tell vsftpd to not allow any anonymous connections eg mandatory login.  What and where to log. What IP and port to listen on.  To lock users into their home directory (defined in /etc/passwd).

The bottom 3 lines userlist_* we define so we don't grant all local users ftp access.  If you answer YES to userlist_deny, the user list will deny any users listed in the file and allow everything else.

cat /etc/vsftpd/vsftpd.user_list
user1
user2
user3

Restart vsftpd to activate the new config.

Let's test and make sure everything is working.

$ ftp cdstealer.com
Connected to cdstealer.com (<IP>).
220 (vsFTPd 3.0.2)
Name (cdstealer.com:user1): 
530 Please login with USER and PASS.
SSL not available
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

SSL

So let's get some encryption so we aren't transmitting plain text credentials.

Generate a self signed cert.

# openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout /etc/ssl/apache2/vsftp.pem -out /etc/ssl/apache2/vsftp.pem
Generating a 4096 bit RSA private key
..........................................................................................++
....++
writing new private key to '/etc/ssl/apache2/vsftp.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:GB
State or Province Name (full name) [Some-State]:Somewhere nice
Locality Name (eg, city) []:Leeds
Organization Name (eg, company) [Internet Widgits Pty Ltd]:cdstealer.com
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:cdstealer.com
Email Address []:

The openssl command I've used, generates a 4096bit encrypted cert (this is good) that is valid for 10 years. :)

Execute the following to remove unwanted access to the cert.

chmod 600 /etc/ssl/apache2/vsftp.pem

Add this section to your /etc/vsftpd/vsftpd.conf file.

##############
# Enable SSL #
##############
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
require_ssl_reuse=NO
ssl_tlsv1=NO
ssl_sslv2=NO
ssl_sslv3=NO
implicit_ssl=NO
ssl_ciphers=TLSv1.2+HIGH:TLSv1.3+HIGH:@STRENGTH:!eNULL:!aNULL
rsa_cert_file=/etc/letsencrypt/live/cdstealer.com/fullchain.pem
rsa_private_key_file=/etc/letsencrypt/live/cdstealer.com/privkey.pem

Restart vsftpd to activate the new config.

Let's test and make sure everything is still working.

$ ftp cdstealer.com
Connected to cdstealer.com (<IP>).
220 (vsFTPd 3.0.2)
Name (cdstealer.com:user1): 
234 Proceed with negotiation.
[SSL Cipher AES128-SHA]
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

GREAT!  So now we have a secure FTP service running.

Extra Options:

##################
# Other Settings #
##################
dirlist_enable=YES
download_enable=YES
force_dot_files=NO
hide_ids=YES
max_clients=2
max_per_ip=2

#######################
# Passive Connections #
#######################
pasv_enable=YES
pasv_address=<domainname>
pasv_min_port=63899
pasv_max_port=63999
pasv_addr_resolve=YES

Virtual Users:

In the interest of security, I'm not comfortable having system user accounts, even though we have secured things with only specific users able to ftp.  I believe that standard ftp authentication does not support encryption for system users, but does for virtual users?

This is a little more involved than having standard system users :(

So, add the following to your /etc/vsftpd/vsftpd.conf file.

##################
## Virtual Users #
##################
virtual_use_local_privs=YES
pam_service_name=vsftpd
user_sub_token=$USER
local_root=/FTP/$USER
secure_chroot_dir=/var/run/vsftpd
guest_enable=YES
guest_username=ftp
user_config_dir=/etc/vsftpd/virtualUsers

Here we define that virtual users get the same permissions as the ftp system user, are unable to browse outside their directory, use a PAM database for credentials and define custom settings per user.

Create the PAM config.
vi /etc/pam.d/vsftpd
Populate with the following.
auth required /lib/security/pam_userdb.so db=/etc/vsftpd/virtualUsers
account required /lib/security/pam_userdb.so db=/etc/vsftpd/virtualUsers
session required /lib/security/pam_loginuid.so

pam_userdb.so is part of pam, so nothing else to do.

Create user database.
cd /etc/vsftpd
vi virtualUsers.txt

The format of this file is:

username
password
username
password
<empty line>
Execute the following to create the database.
db5.3_load -T -t hash -f /etc/vsftpd/virtualUsers.txt /etc/vsftpd/virtualUsers.db
Update the permissions.
chmod 600 virtualUsers.txt virtualUsers.db

NOTE: You may notice that the file extension is missing from the path in /etc/pam.d/vsftpd.  This is intentional as PAM automatically adds the .db suffix.

You will need to add the users from the database to the access list file.

/etc/vsftpd/vsftpd.user_list

For your convenience, I've written a user management script, here :)

Define custom settings.

Create the directory which will store the configs.  We defined this earlier as user_config_dir=/etc/vsftpd/virtualUsers.

mkdir /etc/vsftpd/virtualUsers
Create a user config.
vi /etc/vsftpd/virtualUsers/user1

write_enable=NO
local_root=/FTP/user1
chroot_local_user=YES
dirlist_enable=YES
download_enable=YES

Here we set the users CHROOT directory, deny write permissions and allow downloading.  Options here (not all) override specific options defined in the main config.

RTFM

NOTES:

If you get the following when logging in or listing a directory, it maybe due to the user directory not existing or not having permission.

ssl_getc: SSL_read failed -1 = 0
421 Service not available, remote server has closed connection
Login failed.
No control connection for command: Success

The following error may occur on ftp clients with vsftpd 3.0.x:

500 OOPS: priv_sock_get_cmd

This is caused by seccomp filter sanboxing, which is enabled by default on amd64. To workaround this issue, disable seccomp filter sanboxing:

Add the following line to /etc/vsftpd/vsftpd.conf.

seccomp_sandbox=NO

Don't forget to restart vsftpd :)

Generate SSL Certificate Request.

Generate the RSA key

Run the following commands to create a directory in which to store your RSA key, substituting a directory name of your choice:

mkdir ~/domain.com.ssl/
cd ~/domain.com.ssl/

Run the following command to generate a private key:

openssl genrsa -out ~/domain.com.ssl/domain.com.key 2048

Create a CSR

Type the following command to create a CSR with the RSA private key (output is in PEM format):

openssl req -new -sha256 -key ~/domain.com.ssl/domain.com.key -out ~/domain.com.ssl/domain.com.csr

When prompted, enter the necessary information for creating a CSR by using the conventions shown in the following table.

Note: The following characters cannot be used in the Organization Name or theOrganizational Unit: < > ~ ! @ # $ % ^ * / \ ( ) ?.,&

DN field Explanation Example
Common Name The fully qualified domain name for your web server. This must be an exact match. If you intend to secure the URL https://www.yourdomain.com, then your CSR’s common name must be www.yourdomain.com. If you plan to get a wildcard certificate, make sure to prefix your domain name with an asterisk, for example: *.domain.com.
Organization Name The exact legal name of your organization. Do not abbreviate your organization name. domain.com
Organizational Unit Section of the organization. IT
City or Locality The city where your organization is legally located. Wellesley Hills
State or Province The state or province where your organization is legally located. Do not use an abbreviation. Massachusetts
Country The two-letter ISO abbreviation for your country. US

Warning: Leave the challenge password blank (press Enter).

Verify your CSR

Run the following command to verify your CSR:

openssl req -noout -text -in ~/domain.com.ssl/domain.com.csr

Submit your CSR

Submit the CSR that you created to a certificate authority.

Trouble Shooting:
A really cool list of commands you can run here

WordPress Theme Customization

At the time of writing this, I'm using the Twentyfourteen theme which is the one I prefer.  However, I wanted it to use as much of the screen estate as possible.  I didn't like any other theme that had this, so I looked into modifying what I had.  Here is what I did...

Step 1) Go to your site admin page.

Step 2) Go to "appearance" and select "theme"

Step 3) Select the theme you wish to change and activate.

Step 4) Go to "appearance" and select "customise"

Step 5) Click the "Additional CSS" option at the bottom of the left sidebar.

Step 6) Add the CSS sections of what you want to change.

To achieve the look I wanted, I added the following:

.site {
background-color: #fff;
max-width: 100%;
position: relative;
}

.site-header {
background-color: #000;
max-width: 100%;
position: relative;
width: 100%;
z-index: 4;
}

.site-content .entry-header,
.site-content .entry-content,
.site-content .entry-summary,
.site-content .entry-meta,
.page-content {
margin: 0 auto;
max-width: 90%;
}

youtube video downloader

Want to batch download a bunch of videos in one swoop?  Here's how.  This also works with individual videos :)

Step 1)  Install youtube-dl

# emerge -av youtube-dl

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild R ] net-misc/youtube-dl-2016.12.22::gentoo USE="offensive {-test}" PYTHON_TARGETS="python2_7 python3_4 (-python3_5)"

Step 2) Create a playlist on youtube

Step 3) Execute the command and sit back.

$ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]' --merge-output-format 'mp4' https://www.youtube.com/playlist?list=PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_ -o '%(title)s'

So this command takes the best mp4 video and the best m4a, merges them into mp4 and saves it as the title of the source from youtube.

Example output is:

[youtube:playlist] PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_: Downloading webpage
[download] Downloading playlist: Python 3.4 Programming Tutorials
[youtube:playlist] playlist Python 3.4 Programming Tutorials: Downloading 56 videos
[download] Downloading video 1 of 56
[youtube] HBxCHonP6Ro: Downloading webpage
[youtube] HBxCHonP6Ro: Downloading video info webpage
[youtube] HBxCHonP6Ro: Extracting video information
[youtube] HBxCHonP6Ro: Downloading MPD manifest
[download] Python Programming Tutorial - 1 - Installing Python.f137 has already been downloaded
[download] 100% of 11.66MiB
[download] Destination: Python Programming Tutorial - 1 - Installing Python.f140
[download] 100% of 3.73MiB in 00:01
[ffmpeg] Merging formats into "Python Programming Tutorial - 1 - Installing Python.mp4"
Deleting original file Python Programming Tutorial - 1 - Installing Python.f137 (pass -k to keep)
Deleting original file Python Programming Tutorial - 1 - Installing Python.f140 (pass -k to keep)

You can get the official docs from here.

Bash Shell Tips

Just a few things I find helpful :)


Bash history with timestamp.

This is more of an audit thing, but sometimes it's useful to know when you did something.

If you type the command history into your shell.  You get back a list of the last X number of command executed.

$ history | tail -n1
502 history | tail -n1

But by adding the following to your ~/.bashrc (local user) or /etc/bash/bashrc (all users) file, you can inject the time/date.  Please see man date for the options you can use.

HISTTIMEFORMAT='%F %T '

Example

$ history | tail -n1
508 2017-01-14 11:29:58 history | tail -n1

Bash History Duplicate Removal

It's an annoyance when you execute a command consecutively many time and then have to search further back to get to the last different command.  Behold!  Add this to your .bashrc(local user) or /etc/bash/bashrc (all users) and no matter how many times you execute that command consecutively, it will only store the one time.

HISTCONTROL=ignoreboth

Example

user@server ~ $ vi .bashrc
user@server ~ $ vi .bashrc
user@server ~ $ vi .bashrc
user@server ~ $ vi .bashrc
user@server ~ $ history | tail -n5
508 2017-01-14 11:29:58 history | tail -n1
509 2017-01-14 11:31:36 man date
510 2017-01-14 11:32:43 history
511 2017-01-14 11:40:52 vi .bashrc
512  2017-01-14 11:42:36 history | tail -n5

Double tap exit/logoff

Accidentally logging out of a shell session or user session can be annoying.  But there is hope :)  Add this to your .bashrc(local user) or /etc/bash/bashrc (all users) and now you have to double tap to get out.

This one ideally needs to go into the /etc/bash/bashrc or else it would only work when closing your own shell session.

IGNOREEOF=1

Example

server ~ # Use "logout" to leave the shell.
server ~ # logout
user@server ~ $

Extract Path or File

$ export VAR=/home/me/mydir/file.c

$ echo "${VAR%/*}"
/home/me/mydir

$ echo "${VAR##*/}"
file.c
Cron Execution Only

Pop this at the top of a bash script.

Eg. The below is for a cronjob that is executed at reboot:

# crontab -l
@reboot /bin/bash /path/to/something/bad.sh

#!/usr/bin/env bash

if [[ ! $(< /proc/${PPID}/comm) == cron* ]]; then
echo "Do NOT manually execute this script. ONLY at reboot!!!!"
exit
fi

Greylisting with Exim

I decided to apply greylisting on my server, not because I had to as spamassassin was blocking 99.999999999% of spam, but because I can ;)

I set this up on Exim-4.88 which was the current stable release at the time of writing.

It actually easier than I thought.

Under the "MAIN CONFIGURATION SETTINGS" I added the following line to define my whitelist:

addresslist whitelist_senders = wildlsearch;/etc/exim/greylist_whitelist

The file should contain either full email addresses or wildcard domains.  eg this@email.com or *@email.com on separate lines.

Then under the "ACL CONFIGURATION" section, I have the following.  This precedes the spam section of my config.

warn set acl_m_greyfile = /var/spool/exim/greylist/${length_255:\
${sg{$sender_host_address}{\N\.\d+$\N}{}},\
${sg{$sender_address,$local_part@$domain}{\N[^\w.,=@-]\N}{} }}

defer log_message = greylisted
!senders = +whitelist_senders
condition = ${if exists{$acl_m_greyfile}\
{${if >{${eval:$tod_epoch-\
${extract{mtime}{${stat:$acl_m_greyfile}} }}\
}{180}{0}{1}}\
}{${if eq{${run{/usr/bin/touch $acl_m_greyfile} }}{}{1}{1} }} }
message = Deferred: Temporary error, please try again later

The "!senders = +whitelist_senders" line will lookup against the file you created.  It will also create an empty file within the path of the first line of this section for the time based rejection.  So to keep things "tidy", we'll run a cronjob every 30 mins to remove files.

# Expire greylisters
*/30 * * * * /usr/bin/find /var/spool/exim/greylist -cmin +363 -type f -delete

And that's it!  Restart exim, send an email from an outside source and check your exim log ;)

2017-01-01 12:49:44 H=(209-182-113-49.ip.twinvalley.net) [209.182.113.49] F=<czgxi@biz2net.com> temporarily rejected RCPT <this@email.com>: greylisted

If the mail is from a genuine and correctly configured email server, when it retries (after 3 minutes), the mail will be accepted ;)

Here are some visualisations to show how effective greylisting is.

This is a graph showing the number of spam blocked every day.

Spam blocked
Can you guess when I applied greylisting?

This is a graph showing the grey listing applied every day and how it actually reduces the number of spam emails received by the server.

Greylisting in effect
Greylisting in effect!

VIM autocmd

Want Vim to automatically do things to new or existing files?  Autocmd to the rescue!  Here I will demo a very simple example that I use for when I create/edit bash/python files.

I created the directory .vim in the root of my home directory and then created the "header" files that I wanted.

$ cat .vim/python_header
:insert
#!/usr/bin/python3
################################################################################
####
#### Filename:
####
#### Purpose:
####
#### Created on:
####
#### Author:
####
#### Last Modified:
####
################################################################################
.

The first line must be :insert and the last line must be a period '.'

I then added the following lines to the .vimrc file in the root of my home directory.

" Python file header
autocmd bufnewfile *.py so /home/cdstealer/.vim/python_header
autocmd bufnewfile *.py exe "4" "g/Filename:.*/s//Filename:\t\t\t" .expand("%")
autocmd bufnewfile *.py exe "6" "g/Purpose:.*/s//Purpose:\t\t\t"
autocmd bufnewfile *.py exe "8" "g/Created on:.*/s//Created on:\t\t" .strftime("%c")
autocmd bufnewfile *.py exe "10" "g/Author:.*/s//Author:\t\t\t" .$USER
autocmd Bufwritepre,filewritepre *.py execute "normal ma"
autocmd BufWritePre,FileWritePre *.py exe "12" "g/Last Modified:.*/s/Last Modified:.*/Last Modified:\t\t" .strftime("%c") . " by " . $USER
autocmd bufwritepost,filewritepost *.py execute "normal `a"

Note: Lines that start with a double quote are treated as comments.

The official docs can be found here.

Explanation:

The first autocmd line specifies the file.  Due to the :insert, this is then inserted into the new file.

bufnewfile: means take the following action on a new file.
Bufwritepre,filewritepre: means update the file only if editing not creating.

Lines 2, 3, 4 & 5 are basic search and replace for the header file being inserted.

The number in quotes after the exe is the line number to work on.  You can set this to a range instead of being specific.  The syntax is:

"<starting line number>," . <end line number> .

Once saved, you can test straight away by starting a new file:

$ vi python_test.py

1 #!/usr/bin/python3
2 ################################################################################
3 ####
4 #### Filename: python_test.py
5 ####
6 #### Purpose:
7 ####
8 #### Created on: Fri 30 Dec 2016 19:18:58 GMT
9 ####
10 #### Author: cdstealer
11 ####
12 #### Last Modified:
13 ####
14 ################################################################################

Save the file and then re-open, make a small change and the "Last Modified" line will now contain a date stamp.

1 #!/usr/bin/python3
2 ################################################################################
3 ####
4 #### Filename: python_test.py
5 ####
6 #### Purpose: To test
7 ####
8 #### Created on: Fri 30 Dec 2016 19:18:58 GMT
9 ####
10 #### Author: cdstealer
11 ####
12 #### Last Modified: Fri 30 Dec 2016 19:20:38 GMT by cdstealer
13 ####
14 ################################################################################

Very very useful ;)