Yes there are millions of password generators out there in the wild. But I wanted to write my own. So without further ado, here it is :)
Category Archives: Linux
WordPress Theme Customization
At the time of writing this, I'm using the Twentyfourteen theme which is the one I prefer. However, I wanted it to use as much of the screen estate as possible. I didn't like any other theme that had this, so I looked into modifying what I had. Here is what I did...
Step 1) Go to your site admin page.
Step 2) Go to "appearance" and select "theme"
Step 3) Select the theme you wish to change and activate.
Step 4) Go to "appearance" and select "customise"
Step 5) Click the "Additional CSS" option at the bottom of the left sidebar.
Step 6) Add the CSS sections of what you want to change.
To achieve the look I wanted, I added the following:
.site {
background-color: #fff;
max-width: 100%;
position: relative;
}
.site-header {
background-color: #000;
max-width: 100%;
position: relative;
width: 100%;
z-index: 4;
}
.site-content .entry-header,
.site-content .entry-content,
.site-content .entry-summary,
.site-content .entry-meta,
.page-content {
margin: 0 auto;
max-width: 90%;
}
youtube video downloader
Want to batch download a bunch of videos in one swoop? Here's how. This also works with individual videos :)
Step 1) Install youtube-dl
# emerge -av youtube-dl
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] net-misc/youtube-dl-2016.12.22::gentoo USE="offensive {-test}" PYTHON_TARGETS="python2_7 python3_4 (-python3_5)"
Step 2) Create a playlist on youtube
Step 3) Execute the command and sit back.
$ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]' --merge-output-format 'mp4' https://www.youtube.com/playlist?list=PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_ -o '%(title)s'
So this command takes the best mp4 video and the best m4a, merges them into mp4 and saves it as the title of the source from youtube.
Example output is:
[youtube:playlist] PL6gx4Cwl9DGAcbMi1sH6oAMk4JHw91mC_: Downloading webpage [download] Downloading playlist: Python 3.4 Programming Tutorials [youtube:playlist] playlist Python 3.4 Programming Tutorials: Downloading 56 videos [download] Downloading video 1 of 56 [youtube] HBxCHonP6Ro: Downloading webpage [youtube] HBxCHonP6Ro: Downloading video info webpage [youtube] HBxCHonP6Ro: Extracting video information [youtube] HBxCHonP6Ro: Downloading MPD manifest [download] Python Programming Tutorial - 1 - Installing Python.f137 has already been downloaded [download] 100% of 11.66MiB [download] Destination: Python Programming Tutorial - 1 - Installing Python.f140 [download] 100% of 3.73MiB in 00:01 [ffmpeg] Merging formats into "Python Programming Tutorial - 1 - Installing Python.mp4" Deleting original file Python Programming Tutorial - 1 - Installing Python.f137 (pass -k to keep) Deleting original file Python Programming Tutorial - 1 - Installing Python.f140 (pass -k to keep)
You can get the official docs from here.
Bash Shell Tips
Just a few things I find helpful :)
Bash history with timestamp.
This is more of an audit thing, but sometimes it's useful to know when you did something.
If you type the command history into your shell. You get back a list of the last X number of command executed.
$ history | tail -n1 502 history | tail -n1
But by adding the following to your ~/.bashrc (local user) or /etc/bash/bashrc (all users) file, you can inject the time/date. Please see man date for the options you can use.
HISTTIMEFORMAT='%F %T '
Example
$ history | tail -n1 508 2017-01-14 11:29:58 history | tail -n1
Bash History Duplicate Removal
It's an annoyance when you execute a command consecutively many time and then have to search further back to get to the last different command. Behold! Add this to your .bashrc(local user) or /etc/bash/bashrc (all users) and no matter how many times you execute that command consecutively, it will only store the one time.
HISTCONTROL=ignoreboth
Example
user@server ~ $ vi .bashrc user@server ~ $ vi .bashrc user@server ~ $ vi .bashrc user@server ~ $ vi .bashrc user@server ~ $ history | tail -n5 508 2017-01-14 11:29:58 history | tail -n1 509 2017-01-14 11:31:36 man date 510 2017-01-14 11:32:43 history 511 2017-01-14 11:40:52 vi .bashrc 512 2017-01-14 11:42:36 history | tail -n5
Double tap exit/logoff
Accidentally logging out of a shell session or user session can be annoying. But there is hope :) Add this to your .bashrc(local user) or /etc/bash/bashrc (all users) and now you have to double tap to get out.
This one ideally needs to go into the /etc/bash/bashrc or else it would only work when closing your own shell session.
IGNOREEOF=1
Example
server ~ # Use "logout" to leave the shell. server ~ # logout user@server ~ $
Extract Path or File
$ export VAR=/home/me/mydir/file.c
$ echo "${VAR%/*}"
/home/me/mydir
$ echo "${VAR##*/}"
file.c
Cron Execution Only
Pop this at the top of a bash script.
Eg. The below is for a cronjob that is executed at reboot:
# crontab -l
@reboot /bin/bash /path/to/something/bad.sh
#!/usr/bin/env bash
if [[ ! $(< /proc/${PPID}/comm) == cron* ]]; then
echo "Do NOT manually execute this script. ONLY at reboot!!!!"
exit
fi
Kernel Upgrade Script
Here I wrote a small(ish) bash script to semi automate a kernel upgrade on Gentoo Linux.
(right click -> save as)
Script
Greylisting with Exim
I decided to apply greylisting on my server, not because I had to as spamassassin was blocking 99.999999999% of spam, but because I can ;)
I set this up on Exim-4.88 which was the current stable release at the time of writing.
It actually easier than I thought.
Under the "MAIN CONFIGURATION SETTINGS" I added the following line to define my whitelist:
addresslist whitelist_senders = wildlsearch;/etc/exim/greylist_whitelist
The file should contain either full email addresses or wildcard domains. eg this@email.com or *@email.com on separate lines.
Then under the "ACL CONFIGURATION" section, I have the following. This precedes the spam section of my config.
warn set acl_m_greyfile = /var/spool/exim/greylist/${length_255:\
${sg{$sender_host_address}{\N\.\d+$\N}{}},\
${sg{$sender_address,$local_part@$domain}{\N[^\w.,=@-]\N}{} }}
defer log_message = greylisted
!senders = +whitelist_senders
condition = ${if exists{$acl_m_greyfile}\
{${if >{${eval:$tod_epoch-\
${extract{mtime}{${stat:$acl_m_greyfile}} }}\
}{180}{0}{1}}\
}{${if eq{${run{/usr/bin/touch $acl_m_greyfile} }}{}{1}{1} }} }
message = Deferred: Temporary error, please try again later
The "!senders = +whitelist_senders" line will lookup against the file you created. It will also create an empty file within the path of the first line of this section for the time based rejection. So to keep things "tidy", we'll run a cronjob every 30 mins to remove files.
# Expire greylisters */30 * * * * /usr/bin/find /var/spool/exim/greylist -cmin +363 -type f -delete
And that's it! Restart exim, send an email from an outside source and check your exim log ;)
2017-01-01 12:49:44 H=(209-182-113-49.ip.twinvalley.net) [209.182.113.49] F=<czgxi@biz2net.com> temporarily rejected RCPT <this@email.com>: greylisted
If the mail is from a genuine and correctly configured email server, when it retries (after 3 minutes), the mail will be accepted ;)
Here are some visualisations to show how effective greylisting is.
This is a graph showing the number of spam blocked every day.
This is a graph showing the grey listing applied every day and how it actually reduces the number of spam emails received by the server.
VIM autocmd
Want Vim to automatically do things to new or existing files? Autocmd to the rescue! Here I will demo a very simple example that I use for when I create/edit bash/python files.
I created the directory .vim in the root of my home directory and then created the "header" files that I wanted.
$ cat .vim/python_header :insert #!/usr/bin/python3 ################################################################################ #### #### Filename: #### #### Purpose: #### #### Created on: #### #### Author: #### #### Last Modified: #### ################################################################################ .
The first line must be :insert and the last line must be a period '.'
I then added the following lines to the .vimrc file in the root of my home directory.
" Python file header
autocmd bufnewfile *.py so /home/cdstealer/.vim/python_header
autocmd bufnewfile *.py exe "4" "g/Filename:.*/s//Filename:\t\t\t" .expand("%")
autocmd bufnewfile *.py exe "6" "g/Purpose:.*/s//Purpose:\t\t\t"
autocmd bufnewfile *.py exe "8" "g/Created on:.*/s//Created on:\t\t" .strftime("%c")
autocmd bufnewfile *.py exe "10" "g/Author:.*/s//Author:\t\t\t" .$USER
autocmd Bufwritepre,filewritepre *.py execute "normal ma"
autocmd BufWritePre,FileWritePre *.py exe "12" "g/Last Modified:.*/s/Last Modified:.*/Last Modified:\t\t" .strftime("%c") . " by " . $USER
autocmd bufwritepost,filewritepost *.py execute "normal `a"
Note: Lines that start with a double quote are treated as comments.
The official docs can be found here.
Explanation:
The first autocmd line specifies the file. Due to the :insert, this is then inserted into the new file.
bufnewfile: means take the following action on a new file.
Bufwritepre,filewritepre: means update the file only if editing not creating.
Lines 2, 3, 4 & 5 are basic search and replace for the header file being inserted.
The number in quotes after the exe is the line number to work on. You can set this to a range instead of being specific. The syntax is:
"<starting line number>," . <end line number> .
Once saved, you can test straight away by starting a new file:
$ vi python_test.py 1 #!/usr/bin/python3 2 ################################################################################ 3 #### 4 #### Filename: python_test.py 5 #### 6 #### Purpose: 7 #### 8 #### Created on: Fri 30 Dec 2016 19:18:58 GMT 9 #### 10 #### Author: cdstealer 11 #### 12 #### Last Modified: 13 #### 14 ################################################################################
Save the file and then re-open, make a small change and the "Last Modified" line will now contain a date stamp.
1 #!/usr/bin/python3 2 ################################################################################ 3 #### 4 #### Filename: python_test.py 5 #### 6 #### Purpose: To test 7 #### 8 #### Created on: Fri 30 Dec 2016 19:18:58 GMT 9 #### 10 #### Author: cdstealer 11 #### 12 #### Last Modified: Fri 30 Dec 2016 19:20:38 GMT by cdstealer 13 #### 14 ################################################################################
Very very useful ;)
Linux Troubleshooting
Structure needs cleaning.
Raspberry Pi 3B Gentoo Install
NOTE: Although the CPU is 64bit, the rest of it is 32bit. Trying to run in 64bit will cause pain and suffering.
Other NOTE: I cannot YET get the display to work with hardware acceleration. Blank screen when enabled and kodi goes nuts, writing hundreds of lines to the log per second.
15:24:07 T:1943678976 ERROR: Previous line repeats 1 times. 15:24:07 T:1943678976 ERROR: Invalid GUI Shader selected - [guishader_frag_fonts.glsl] 15:24:07 T:1943678976 ERROR: Invalid GUI Shader selected - [guishader_frag_texture_noblend.glsl]
Contents:
Partition SD
Boot
Root
Portage
Get a few things sorted
Screen
Offbox Building
Time to get compiling!
Distcc Server
Distcc Client
Systemd
WIFI
Locale
Hostname
Keymap
Tweaks
Caveats
Kodi
Mame
Partition SD
Example setup:
# fdisk -H255 -S63 /dev/sdX -------------------------------------------------------------------------------- Disk /dev/sdg: 59 GiB, 63367544832 bytes, 123764736 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00000000 Device Boot Start End Sectors Size Id Type /dev/sdg1 * 2048 104447 102400 50M c W95 FAT32 (LBA) /dev/sdg2 104448 115447807 115343360 55G 83 Linux /dev/sdg3 115447808 123764735 8316928 4G 5 Extended /dev/sdg5 115449856 123764735 8314880 4G 82 Linux swap / Solaris --------------------------------------------------------------------------------
Format Filesystems:
mkfs.vfat -n BOOT /dev/sdg1 mke2fs -t ext4 -L ROOTFS /dev/sdg2 mkswap /dev/sdg5
The boot filesystem needs to be VFAT but the rest are up to you :)
Boot!
Download the firmware archive from here
extract the /boot/* to the boot filesystem of the SD card.
Create a file called cmdline.txt to specify any necessary kernel parameters.
For example:
root=/dev/mmcblk0p2 rootdelay=2 rootfstype=ext4 rootwait
Create the file config.txt for more options.
For example:
gpu_mem=64 arm_control=0x200
That should be the boot system sorted :)
Root
Download stage3
stage3-armv7a_hardfp-YYYYMMDD.tar.bz2
Download portage
Unpack the stage3 tarball:
tar xpjf stage3-arm64-YYYYMMDD.tar.bz2 -C /path/to/sd/ROOTFS
Adjust the make.conf file:
/path/to/sd/ROOTFS/etc/portage/make.conf
CFLAGS="-O2 -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard"
CXXFLAGS="${CFLAGS}"
Generate the root password hash:
openssl passwd -1
Add this to the /path/to/sd/ROOTFS/etc/shadow file
eg (replace the * between the first and second :)
root:$1$SnXkVgp0$AhbB/IogzcV.opa9suOx21:9797:0:::::
Adjust the /path/to/sd/ROOTFS/etc/fstab file. The SD card is recognized as /dev/mmcblk0
| <fs> | <mountpoint> | <type> | <opts> | <dump> | <pass> |
| /dev/mmcblk0p1 | /boot | vfat | noauto,noatime | 0 | 0 |
| /dev/mmcblk0p2 | / | ext4 | noatime | 0 | 1 |
| /dev/mmcblk0p3 | none | swap | sw | 0 | 0 |
Portage
Unpack the portage tarball:
tar xpjf /path/ to/Downloads/portage-latest.tar.bz2 -C /path/to/sd/ROOTFS/usr/
At this point we should now have a bootable RPi3. :) It can't do very much, but we'll work on that.
Hint: Boot up the RPi
Get a few things sorted
Once booted and logged in, do this before we do anything else.
Get the network up:
Assuming the use of DHCP on the eth0 network interface.
cd /etc/init.d/ cp net.lo net.eth0 rc-config start net.eth0 rc-update add net.eth0 boot rc-update add sshd default
If you intend on doing remotely, we need to create a user and assign to the wheel group. We could configure SSH to accept direct root connections, but that's bad practice.
execute the command:
useradd -m <username> -G wheel
This will create the user, the home directory and amend the assigned groups with the wheel group.
Now set the passwd:
passwd <username>
Edit rc.conf
nano /etc/rc.conf
uncomment rc_sys=""
Stop the annoying "s0 respawning too fast" error:
nano /etc/inittab
Comment the s0:12345... line
The RaspberryPi does not have a hwclock:
rc-update add swclock boot rc-update del hwclock boot
Set a hostname (optional unless you run your own DHCP/DNS)
nano -w /etc/conf.d/hostname hostname="pifukka"
Set the correct keymap for the console.
nano -w /etc/conf.d/keymaps set keymap="uk"
Set the correct timezone
ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
Create the portage directories.
mkdir /etc/portage/package.{keywords,mask,use}
Until we have NTP installed and working, each time the RPi is booted, we'll need to set the time/date. This is done by executing:
date --set="04/09/2016 19:55:00"
MM/DD/YYYY HH:MM:SS
Failure to do this will cause pain if you try and compile anything. eg distcc goes into a configure loop :(
Select the profile of the OS:
Execute:
eselect profile list
will get you this:
[1] default/linux/arm/13.0 [2] default/linux/arm/13.0/desktop [3] default/linux/arm/13.0/desktop/gnome [4] default/linux/arm/13.0/desktop/gnome/systemd [5] default/linux/arm/13.0/desktop/kde [6] default/linux/arm/13.0/desktop/kde/systemd [7] default/linux/arm/13.0/developer [8] default/linux/arm/13.0/armv4 [9] default/linux/arm/13.0/armv4/desktop [10] default/linux/arm/13.0/armv4/desktop/gnome [11] default/linux/arm/13.0/armv4/desktop/kde [12] default/linux/arm/13.0/armv4/developer [13] default/linux/arm/13.0/armv4t [14] default/linux/arm/13.0/armv4t/desktop [15] default/linux/arm/13.0/armv4t/desktop/gnome [16] default/linux/arm/13.0/armv4t/desktop/kde [17] default/linux/arm/13.0/armv4t/developer [18] default/linux/arm/13.0/armv5te [19] default/linux/arm/13.0/armv5te/desktop [20] default/linux/arm/13.0/armv5te/desktop/gnome [21] default/linux/arm/13.0/armv5te/desktop/kde [22] default/linux/arm/13.0/armv5te/developer [23] default/linux/arm/13.0/armv6j [24] default/linux/arm/13.0/armv6j/desktop [25] default/linux/arm/13.0/armv6j/desktop/gnome [26] default/linux/arm/13.0/armv6j/desktop/kde [27] default/linux/arm/13.0/armv6j/developer [28] default/linux/arm/13.0/armv7a * [29] default/linux/arm/13.0/armv7a/desktop [30] default/linux/arm/13.0/armv7a/desktop/gnome [31] default/linux/arm/13.0/armv7a/desktop/kde [32] default/linux/arm/13.0/armv7a/developer [33] hardened/linux/arm/armv7a [34] hardened/linux/arm/armv6j [35] hardened/linux/musl/arm/armv7a [36] default/linux/uclibc/arm/armv7a [37] hardened/linux/uclibc/arm/armv7a
As you can see by the `*`, I have left it at the default (option 28)
We are going to be required to build a few packages natively on the RPi3. This will take a while due to the ARM CPU having to compile power of a decomposing turd :|
I also found it useful to get screen installed :) Should take approx. 5 minutes.
emerge app-misc/screen
Screen
I find screen extremely useful. I'm sure you had situations where you were part way through something on a remote terminal and you gotten disconnected and lost everything. With screen, that doesn't matter. If you get disconnected, just reconnect to the server then reconnect to the screen session. :)
start a screen session
screen -S sesh1
Offbox Building
We are going to configure both the RPi and full fat system as a server/client compiler array as compiling anything on the RPi is beyond time and space.
SERVER:
CROSSDEV
Add the crossdev package to keywords to install the latest version.
|
If you need to remove cross building:
execute crossdev -C cross-armv7a-hardfloat-linux-gnueabi * Uninstalling target 'armv7a-hardfloat-linux-gnueabi' ... <<< cross-armv7a-hardfloat-linux-gnueabi/glibc-2.22-r4 <<< cross-armv7a-hardfloat-linux-gnueabi/linux-headers-4.3 <<< cross-armv7a-hardfloat-linux-gnueabi/binutils-2.25.1-r1 PORTAGE_BZIP2_COMMAND setting is invalid: 'bzip2' PORTAGE_BZIP2_COMMAND setting from make.globals is invalid: 'bzip2' <<< cross-armv7a-hardfloat-linux-gnueabi/gcc-4.9.3 /usr/armv7a-hardfloat-linux-gnueabi: directory still exists; remove recursively? [y/N] Then reverse the steps below. |
emerge crossdev
Tell portage about our repo.
mkdir -p /usr/local/portage-crossdev/{profiles,metadata}
echo 'portage-crossdev' > /usr/local/portage-crossdev/profiles/repo_name
echo 'masters = gentoo' > /usr/local/portage-crossdev/metadata/layout.conf
chown -R portage:portage /usr/local/portage-crossdev
vi /etc/portage/repos.conf/gentoo.conf
[portage-crossdev]
location=/usr/local/portage-crossdev
masters=gentoo
priority=10
Build the Toolchain:
crossdev -S -v -t armv7a-hardfloat-linux-gnueabi
You should get the following output:
-------------------------------------------------------------------------------- * crossdev version: 20160602 * Host Portage ARCH: amd64 * Target Portage ARCH: arm * Target System: armv7a-hardfloat-linux-gnueabi * Stage: 4 (C/C++ compiler) * ABIs: default * binutils: binutils-[stable] * gcc: gcc-[stable] * headers: linux-headers-[stable] * libc: glibc-[stable] * CROSSDEV_OVERLAY: /usr/local/portage-crossdev * PORT_LOGDIR: /var/log/portage * PORTAGE_CONFIGROOT: * Portage flags: -v _ - ~ - _ - ~ - _ - ~ - _ - ~ - _ - ~ - _ - ~ - _ - * leaving metadata/layout.conf alone in /usr/local/portage-crossdev _ - ~ - _ - ~ - _ - ~ - _ - ~ - _ - ~ - _ - ~ - _ - * Log: /var/log/portage/cross-armv7a-hardfloat-linux-gnueabi-binutils.log * Emerging cross-binutils ... These are the packages that would be merged, in order: Calculating dependencies -------------------------------------------------------------------------------- THIS IS WHERE SHIT GETS BUILT!
NOTE:
If everything fails with the following error, that means you borked the setup eg NOT FOLLOWED THE INSTRUCTIONS ;).
Calculating dependencies /usr/lib/portage/python3.4/ebuild.sh: line 624: /path/to/sd/ROOTFS/usr/local/portage-crossdev/cross-aarch64-unknown-linux-gnu/binutils/binutils-9999.ebuild: Permission denied
DISTCCD
Add "sys-devel/distcc crossdev" to /etc/portage/package.use/package.use
emerge distcc These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] sys-devel/distcc-3.2_rc1-r4::gentoo USE="crossdev ipv6 -avahi -gnome -gssapi -gtk -hardened (-selinux) -xinetd" PYTHON_TARGETS="python2_7" 0 KiB
----------------------------------------------------------
INITRC:
/etc/conf.d/distccd
DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"
----------------------------------------------------------
SYSTEMD:
/etc/systemd/system/distccd.service.d/00gentoo.conf
[Service]
Environment="ALLOWED_SERVERS=192.168.0.0/24"
----------------------------------------------------------
This can be space separated IPs.
/etc/distcc/hosts just IPs of the clients. eg 192.168.0.1,cpp,lzo
CLIENT:
DISTCC
emerge distcc
Takes approx. 5 minutes.
execute distcc-config --set-hosts "<build_host>,cpp,lzo"
Once installed add `FEATURES="distcc distcc-pump"` to /etc/portage/make.conf.
Also add `MAKEOPTS="-jX -lX"` to /etc/portage/make.conf where `jX` is the number of CPUs on the build server times 2+1 and lX is the number of CPUs on the client.
eg MAKEOPTS="-j17 -l4"
Edit /etc/portage/make.conf and add EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=1" (optional)
NOTE: Although the main work is now off loaded, all the pre and post compile work is still done natively.
Time to get compiling!
Install a time daemon.
pump emerge ntp
(2 packages & approx. 12 minutes)
Auto start ntp.
rc-update add ntp-client default /etc/init.d/ntp-client start
KERNEL:
We're going to need this at some point. Some applications need a kernel source to compile. This should be relatively painless now we have a build server. Compiling natively takes ~6 hours!
You will need to add sys-kernel/raspberrypi-sources ** to package.keywords
pump emerge raspberrypi-sources
This will download an uncompressed kernel source from git ~1.45Gb. That and about 21 other deps that take roughly 2 hours to install (not including kernel build).
cd /usr/src/linux-4.4.9999-raspberrypi
CROSSDEV ONLY!!!
x64:
make ARCH=arm64 defconfig make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- oldconfig make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- modules_install INSTALL_MOD_PATH=/run/media/cdstealer/ROOTFS
NOTE: Ensure python2.7 is enabled as >=3.4 gets the error
"TypeError: 'str' does not support the buffer interface"
imagetool-uncompressed.py arch/arm/boot/Image /run/media/cdstealer/BOOT/boot/kernel.img
x32:
make ARCH=arm bcm2709_defconfig make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- oldconfig make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- -j$(nproc) make ARCH=arm CROSS_COMPILE=armv7a-hardfloat-linux-gnueabi- modules_install INSTALL_MOD_PATH=/run/media/cdstealer/ROOTFS
cp arch/arm/boot/zImage /run/media/cdstealer/BOOT/boot/kernel.img
NATIVE:
make bcm2709_defconfig make make modules_install cp arch/arm/boot/zImage /boot/kernel.img
Another way is by the official docs:
make -j4 zImage modules dtbs cp arch/arm/boot/dts/*.dtb /boot/ cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/ cp arch/arm/boot/dts/overlays/README /boot/overlays/ scripts/mkknlimg arch/arm/boot/zImage /boot/kernel7.img make modules_install
Which should then output:
Version: Linux version 4.4.19-raspberrypi-v7+ (root@pifukka) (gcc version 4.9.3 (Gentoo 4.9.3 p1.5, pie-0.6.4) ) #3 SMP Wed Sep 7 00:07:23 BST 2016 DT: y DDT: y 270x: y 283x: n
With a slightly modified config (I removed most drivers not relevant to me. This is the config I used (4.4.21-raspberrypi-v7+) and got a compile time of 1 hour!! So the config is mainly to power the bare RPi3b, a MCE remote and a Playstation controller.
Executed this:
ks=$(date); make -j4 zImage modules dtbs; kf=$(date) && cp arch/arm/boot/dts/*.dtb /boot/ && cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/ && cp arch/arm/boot/dts/overlays/README /boot/overlays/ && scripts/mkknlimg arch/arm/boot/zImage /boot/kernel7-custom.img && make modules_install && echo -e "Kernel start: ${ks}\nKernel end: ${kf}"
and got this:
Version: Linux version 4.4.21-raspberrypi-v7-custom+ (root@pifukka) (gcc version 4.9.3 (Gentoo 4.9.3 p1.5, pie-0.6.4) ) #1 SMP Fri Sep 16 13:55:51 BST 2016 DT: y DDT: y 270x: y 283x: n Kernel start: Fri Sep 16 13:00:09 BST 2016 Kernel end: Fri Sep 16 14:00:00 BST 2016
You can add kernel=kernel-myconfig.img to config.txt to specify your own kernel name.
NOTE: There is a hell of a lot of bloat in this, so if you know what you're doing, you could do a make menuconfig to trim the fat before you run make.
Using pump didn't work when compiling the kernel and failed very quickly. So it's a 6 hour native compile for the default .config :(
It seems that there are just too many versions of the firmware floating about the tinterwebs. So I managed to get this working by downloading the Jessie image from here, mounting the img file and copying the contents of /lib/firmware/brcm to the same location on the RPi.
Although the RPi3B has 802.11n, it will not use 5Ghz frequency band, only 2.4Ghz.
Once you have rebooted with the new kernel and firmware, hopefully you will now see the following in dmesg:
[ 7.270284] usbcore: registered new interface driver brcmfmac [ 7.460455] cfg80211: World regulatory domain updated: [ 7.460475] cfg80211: DFS Master region: unset [ 7.460484] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time) [ 7.460498] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A) [ 7.460513] cfg80211: (2457000 KHz - 2482000 KHz @ 20000 KHz, 92000 KHz AUTO), (N/A, 2000 mBm), (N/A) [ 7.460525] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A) [ 7.460539] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A) [ 7.460553] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s) [ 7.460565] cfg80211: (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s) [ 7.460577] cfg80211: (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A) [ 7.460589] cfg80211: (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A) [ 8.339256] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50 [ 9.349147] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50 [ 10.359184] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50
Systemd
OK. So far we've got a bootable RPi using RC. If you want to use systemd, we'll need to change a few things and also configure the daemons.
I added USE="systemd" to /etc/portage/make.conf
Don't forget to add init=/usr/lib/systemd/systemd to the line in cmdline.txt.
Yes systemd has it's pitfalls, but I've gotten used to it and actually quite like it <don't shoot me>
You'll then need to rebuild everything to use it.
emerge -uvND world
WIFI Network
Create the file:
vi /etc/systemd/network/wireless.network
And populate with content:
[Match] Name=wlan0 [Network] DHCP=ipv4 Domains=cdstealer.com
Also create this file:
vi /etc/wpa_supplicant/wpa_supplicant-wlan0.conf
And populate with content:
ctrl_interface=/var/run/wpa_supplicant
eapol_version=1
ap_scan=1
fast_reauth=1
network={
ssid="<yourSSID"
proto=WPA
key_mgmt=WPA-PSK
pairwise=CCMP TKIP
group=CCMP TKIP WEP104 WEP40
psk="yourSecretPassPhrase"
priority=2
}
Then enable the services:
systemctl enable sshd systemctl enable wpa_supplicant@wlan0 systemctl enable systemd-networkd
Locale
As locales may vary, execute localectl list-locales and make a note of yours. Mine is en_GB.utf8. Then execute:
localectl set-locale LANG=en_GB.utf8
I also had to add LC_ALL="en_GB.utf8" to /etc/environment.
Hostname
hostnamectl set-hostname <hostname>
Keymap
localectl set-keymap uk
Execute localectl to confirm the settings:
# localectl status System Locale: LANG=en_GB.utf8 VC Keymap: uk X11 Layout: gb X11 Model: pc105 X11 Options: terminate:ctrl_alt_bksp
NTP
timedatectl set-ntp true
Drivers
Don't forget to install the driver package.
pump emerge raspberrypi-userland
pump emerge xorg-server
Should yield:
Total: 25 packages (25 new), Size of downloads: 11,256 KiB
Tweaks
edit /boot/config.txt and add the following to the existing line dtoverlay=lirc-rpi, i2c_arm=on, dtoverlay=mmc,overclock_50=100
dtoverlay=lirc-rpi = loads the lirc modules for remotes.
i2c_arm=on = enabled i2c for the CPU.
dtoverlay=mmc,overclock_50=100 = this should speed up the SD access. You may need to reduce the 100 value as I notice considerable file corruption. Some forums have suggested a value of 83.
I also added the following to /etc/portage/make.conf
LINGUAS="en en_GB" L10N="en en-GB" VIDEO_CARDS="vc4"
Caveats
There are a few packages that are not compatible with distcc. So if any of these give you butthurt when emerging, Try emerging them individually. Eg. FEATURES="-distcc -distcc-pump" MAKEOPTS="-j4" emerge -av llvm. Unfortunately that does mean slooooooooow compiles :(
Note: When compiling anything without distcc, ensure you specify the MAKEOPTS as omitting this will crash the Pi. The -j17 that was specified in /etc/portage/make.conf will be used which will be counter productive and slow down the builds until BOOM!
The common error that highlights this is at configure time:
Checking uname sysname type : Traceback (most recent call last):
Affected packages:
Package (native compile time)
sys-libs/talloc (~3 minutes)
sys-devel/llvm (~200 minutes)
sys-libs/tdb (~7 minutes)
sys-libs/ntdb (~3 minutes)
sys-libs/tevent (~3 minutes)
media-video/ffmpeg (~22 minutes)
net-libs/socket_wrapper (~1 minute)
sys-libs/ldb (~4 minutes)
net-fs/samba (~44 minutes)
Now you need to decide what you want to do with it :)
KODI
At this point, this should yield the following:
Total: 166 packages (164 new, 1 in new slot, 1 reinstall)
I have the following global USE flags set in /etc/portage/make.conf
USE="X systemd -bindist opengl xorg udev"
I also have the following local USE flags set in /etc/portage/package.use/package.use
# Kodi >=sys-libs/tevent-0.9.28 python >=sys-libs/ntdb-1.0-r1 python >=dev-lang/python-2.7.10-r1:2.7 sqlite >=sys-libs/tdb-1.3.8 python media-tv/kodi -gles X alsa bluray css java samba usb opengl media-libs/mesa gles2 media-libs/cogl gles2 opengl
I have the following packages in /etc/portage/package.keywords/package.keywords
# Kodi =media-sound/dcadec-0.2.0 ~arm =dev-libs/tinyxml-2.6.2-r2 ~arm =app-eselect/eselect-java-0.2.0-r1 ~arm #=media-tv/kodi-9999 ** =media-tv/kodi-16.0 ** =dev-java/icedtea-web-1.6.2 ~arm =sys-apps/baselayout-java-0.1.0 ~arm =virtual/jre-1.8.0-r1 ~arm =dev-java/java-config-2.2.0-r3 ~arm =virtual/jdk-1.8.0-r3 ~arm =dev-java/icedtea-bin-3.1.0 ~arm =dev-libs/crossguid-0_pre20150817 ~arm =media-fonts/roboto-2.134 ~arm
I had to limit kodi to 16.0 as 9999 would not compile and repeatedly failed with EGLNative errors. Each time failed costs 30 mins :( A successful build takes ~90mins (not including deps).
MAME
SDLMame yields:
Total: 49 packages (49 new)
Convert HDD to VDI
I've only had to do this a couple of times in my life, but it's useful to know ;) There are 2 ways this can be achieved. I'll state the PROS/CONS at the end of each guide.
Details:
Host system - Gentoo X86_64 with 24Tb storage :), VirtualBox
Guest HDD - Gentoo X86_64 500Gb
I'm going to assume that you have good Linux knowledge using the cli with mounting, tarballing, fdisk and formatting filesystems.
#1 (pure disk dump)
Obviously connect the drive you wish to convert.
Create a drive image using dd:
dd if=/dev/sdX of=/path/to/output/file.dd
Then convert it to a VDI (Virtual Disk Image):
VBoxManage convertfromraw ImageFile.dd OutputFile.vdi
Create a new virtual machine and attach the vdi as the main drive.
Once booted into the VM, you will need to install virtualbox guest addititions and may need to reconfigure/recompile the kernel.
Done!
PROS:
- Not complicated.
- Relatively quick to do.
CONS:
- Uses a lot of space as dd reads every sector of the source drive.
- If you delete any data from the VM, it will not reduce the VDI size. Dynamic disks only grow, not shrink.
#2 (only copy files)
Connect the HDD.
Open a terminal.
su to root
Mount each filesystem. (I did the following)
mkdir -p /mnt/{oldhdd,newhdd}/{bootfs,rootfs,homefs}
mount /dev/sdb1 /mnt/oldhdd/bootfs
mount /dev/sdb2 /mnt/oldhdd/rootfs
mount /dev/sdb3 /mnt/oldhdd/homefs
cd into the place you wish to save the data.
cd /mnt/newhdd/bootfs tar czvf bootfs.tgz /mnt/oldhdd/bootfs/. cd ../rootfs tar czvf rootfs.tgz /mnt/oldhdd/rootfs/. cd ../homefs tar czvf homefs.tgz /mnt/oldhdd/homefs/.
If you don't have an existing Linux VM, create one including an empty HDD and boot it with your favourite Linux CD/DVD. Ensure the tarballs you have just created are available to the VM. hint: setup a shared folder in the VBoxManager GUI and mount it in the VM.
You'll need to partition the empty HDD appropriately to accommodate the data you'll be restoring and format each filesystem. Mount each filesystem you've just created and and the shared directory then unpack the tarballs to the new HDD.
So we'll mount the prepared HDD.
mount /dev/sda2 /mnt/newhdd/rootfs mount /dev/sda1 /mnt/newhdd/rootfs/boot
eg
tar zxvf /path/to/share/rootfs.tgz -C /mnt/newhdd/rootfs tar zxvf /path/to/share/bootfs.tgz -C /mnt/newhdd/rootfs/boot
The fun doesn't end yet :) Now we have restored our data, we now need to reinstall the boot loader. In this case, grub2. But first we need to chroot into our restored data.
cd /mnt/newhdd/rootfs mount -t proc none /mnt/newhdd/rootfs/etc/proc chroot /mnt/newhdd/rootfs /bin/bash env-update source /etc/profile grub-install /dev/sda edit /etc/fstab if required grub-mkconfig -o /boot/grub/grub.cfg exit
Unmount the filesystems and reboot the VM. If everything went to plan, you should now be booted into your VM. You will need to install guest additions and may need to reconfigure and recompile your kernel.
PROS:
The VDI only uses the space required.
CONS:
More involved and takes longer.
#3 (existing VM)
This is a combination of #1 & #2 methods.
Create a drive image using dd:
dd if=/dev/sdX of=/path/to/output/file.dd
Then convert it to a VDI (Virtual Disk Image):
VBoxManage convertfromraw ImageFile.dd OutputFile.vdi
Add the new VDI as an additional HDD to your VM. Also create a new HDD and partition appropriately.
Within the VM,
tar clf - -C /mnt/oldhdd/bootfs .| tar xf - -C /mnt/newhdd/bootfs