Raspberry.PI.4.Full.Disk.Encryption.With.RAID

Burn http://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2020-05-28/2020-05-27-raspios-buster-full-armhf.zip to SD-Card. A 16GB one suffices.

In raspi-config

  • enable SSH access
  • set password for pi user
  • set keyboard to something different than UK
  • disable GUI
  • set timezone

reboot. In your router, find out IP and connect with putty.exe to it using pi user. Get root with sudo su -.

Plug your two SSD hard disks into "ICY BOX Dual SSD und Festplatten Docking Station USB 3.0, SATA Clone Station" (https://www.amazon.de/gp/product/B083ZW86H1/). Other ones won't work. I can't praise this one highly enough. No errors are shown in dmesg, ever, and besides "UAS", it also "supports DPO and FUA" and supports monitoring via SMART. lsusb says it's a "ID 152d:0561 JMicron Technology Corp. / JMicron USA Technology Corp. JMS551 - Sharkoon SATA QuickPort Duo".

Warning: Especially don't use https://www.berrybase.de/computer/externe-festplatten-zubehoer/usb-3.0-adapterkabel/konverter-f-252-r-2-5-sata-festplatten-ssds. This one, no matter if you disable "UAS" with usb-storage.quirks=152d:0578:u in /boot/cmdline.txt or update its firmware with JMS578FwUpdate, always outputs errors on dmesg -w or does come online sporadically.

Warning: Also another duet of USB-To-SATA Adapter cables won't work at the same time. 0.6A power is simply too less to power the hard disk and the adapter.

Warning: All USB to SATA things are active and take power as well. If you max_usb_current=1 in /boot/config.txt to increase combined Ampere on all USB ports from 0.6 to 1.2, one of those adapters could raise the consumption to over 0.6 even if the Samsung QVO 4TB on it only takes 0.44.

Make RAID

Warning: Try not to do following and just start parted with -a optimal and work with 0% 300MB and so on.

Do cat /sys/block/sdb/queue/optimal_io_size and cat /sys/block/sdb/alignment_offset. Add. Divide by 512. Thats the number of increments you have to give your Bytes after parted /dev/sda and mklabel gpt and unit B, starting with 0B. For example mkpart primary 1048576B 16777216B. Then enable the raid label with set 1 raid on for partition 1. print lists all partitions. Mine looked kinda like this:

1
2
3
4
Number  Start   End     Size    File system  Name     Flags
1      33.6MB  250MB   216MB                primary  raid
2      268MB   8221MB  7952MB               primary  raid
3      8254MB  4001GB  3993GB               primary  raid

Do /dev/sdb too. Then

1
2
3
4
5
apt-get install mdadm
mdadm --create /dev/md0 --level=1 --metadata=0.90 --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Monitor progress with cat /proc/mdstat.

Practice some Linux Unified Key Setup (You Can Skip This)

apt-get install cryptsetup

Look at partitions with lsblk -f. cryptsetup also eats UUID=<uuid> instead of a device name e.g. /dev/md0 by resolving /dev/disk/by-uuid. A UUID is given as part of a filesystem information, thus, you see sda1 and sdb1 both have UUIDs, in fact the same, written inside their "RAID superblock". md0 does not yet have a UUID as it's a blank device. We now give md0 a UUID by writing a "LUKS header" on it:

head /dev/random -c 256 > /root/slot-0.key
cryptsetup --type luks2 luksFormat /dev/md0 /root/slot-0.key

LUKS has 8 key slots (0-7) it handles automatically. That means on open if you enter a password (or a key file) all slots are looked at and if one matches it's OK. Now we probably have used slot 0 for /root/slot-0.key.

lsblk -f now shows a UUID for /dev/md0! Look at what we've done - there is the beef

cryptsetup luksDump UUID=2cb98c74-7b95-49ea-90ee-431d86af89d8

Some people use xchacha12,aes-adiantum-plain64 instead of aes-xts-plain64 but as cryptsetup --help shows it's not available and we should compile kernel https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=275542&p=1669351&hilit=cryptsetup#p1669351. We should do a lot of things. We object.

We add an additional key, a passphrase given by keyboard - if you do not want to use the keyboard you can use a file as final argument after the UUID.

cryptsetup --key-file /root/slot-0.key luksAddKey UUID=2cb98c74-7b95-49ea-90ee-431d86af89d8

We authenticated with /root/slot-0.key and added a new key at a free slot.

cryptsetup --key-file /root/slot-0.key open UUID=2cb98c74-7b95-49ea-90ee-431d86af89d8 myname

We can use /dev/mapper/myname now!

1
2
3
4
5
mkfs.ext4 /dev/mapper/myname
mkdir /mnt/myname
mount /dev/mapper/myname /mnt/myname/
echo BLA > /mnt/myname/myfile.txt
ls -la /mnt/myname/

Assume /root/slot-0.key is compromised.

head /dev/urandom -c 256 > /root/slot-0-new.key
cryptsetup --key-file /root/slot-0.key luksChangeKey UUID=2cb98c74-7b95-49ea-90ee-431d86af89d8 /root/slot-0-new.key

If we leave out --key-file /root/slot-0.key we get asked on keyboard for any valid key - which will be wiped - and if we leave out /root/slot-0-new.key we get asked on keyboard for the new key - which replaces the wiped one. You get the idea. With cryptsetup luksDump /dev/md2 | grep Slot you can see the slots and with cryptsetup luksKillSlot /dev/md2 2 you can remove a key in a specific slot, but you need to know at least one key to do this.

1
2
3
umount /mnt/myname
cryptsetup close myname
cryptsetup --key-file /root/slot-0.key open UUID=2cb98c74-7b95-49ea-90ee-431d86af89d8 myname

Brings No key available with this passphrase.. slot-0.key ws replaced by slot-1.key. Success. We're done here.

dd if=/dev/zero of=/dev/md0 bs=200M
rm /root/*.key

Shoot for real

1
2
3
apt-get install screen
screen
mkdir /mnt/usbstick

Insert rinkydink USB stick then write down UUID shown with lsblk -f. Honestly, in my case, rinkydink USB stick partition /dev/sdb1, which, a second ago, the Samsung QVO 4TB had(!).

1
2
3
[91403.414661] md: super_written gets error=10
[91403.415116] md2: detected capacity change from 3992378540032 to 0
[91403.415132] md: md2 stopped.

I had ports plugged like this:

[USB-STICK] [EMPTY     ] [ETH]
[EMPTY    ] [ICYBOX-USB]

Nah dude. Calling /dev/sdb suddenly /dev/sdc is terribad. Rebooting. Plugging like this:

[EMPTY    ] [EMPTY     ] [ETH]
[USB-STICK] [ICYBOX-USB]

left /sda and /sdb on the ICYBOX intact. Booting like this and plugging stick later also worked correctly, putting usb stick at /dev/sdc:

1
2
3
4
5
6
[USB-STICK] [ICYBOX-USB] [ETH]
[EMPTY    ] [EMPTY     ]

mount UUID=7F52-3BB4 /mnt/usbstick
openssl rand -out /mnt/usbstick/slot-0.key 256
cryptsetup --type luks2 luksFormat /dev/md2 /mnt/usbstick/slot-0.key

Note new UUID 96645999-c47b-4b33-a463-feb737d0d101 shown with lsblk -f beneath md2

1
2
3
4
5
6
7
cryptsetup --key-file /mnt/usbstick/slot-0.key open UUID=96645999-c47b-4b33-a463-feb737d0d101 luks-96645999-c47b-4b33-a463-feb737d0d101
mkfs.ext4 /dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101
# Found a atari partition table in /dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101, Proceed anyway? (y,N). Yeah right. Still some random stuff left.
mkdir /mnt/luks-96645999-c47b-4b33-a463-feb737d0d101
mount /dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101 /mnt/luks-96645999-c47b-4b33-a463-feb737d0d101
screen
rsync -aHAXxS --exclude '/proc/*' --exclude '/sys/*' --exclude '/run/*' --exclude '/mnt/*' / /mnt/luks-96645999-c47b-4b33-a463-feb737d0d101/

Enable and build INITRD

apt-get install initramfs-tools. INITRD=Yes in /etc/default/raspberrypi-kernel. Add cryptsetup to INITRD with CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook.

Use lsmod |grep -E '(dm[-_]crypt|aes[-_]arm[-_]bs|cryptd|crypto[-_]simd|raid1|md[-_]mod|dm[-_mod])' to find out what module is needed by what. Find out their names using find /lib/modules/5.4.51-v7l+/|grep -E '(dm[-_]crypt.ko|aes[-_]arm[-_]bs.ko|cryptd.ko|crypto[-_]simd.ko|raid1.ko|md[-_]mod.ko|dm[-_]mod.ko)' and add them in correct dependency order to /etc/initramfs-tools/modules

1
2
3
4
5
6
7
dm-crypt
aes-arm-bs
crypto_simd
cryptd
dm-mod
raid1
md-mod

Create /boot/initrd.img-5.4.51-v7l with update-initramfs -c -k $(uname -r) (from now on you can just use update-initramfs -u instead). lsinitramfs /boot/initrd.img-$(uname -r) |grep -E '(dm[-_]crypt.ko|aes[-_]arm[-_]bs.ko|cryptd.ko|crypto[-_]simd.ko|raid1.ko|md[-_]mod.ko|dm[-_]mod.ko|cryptsetup)' checks out, it has the modules and /usr/sbin/cryptsetup.

Tell the kernel it should use it - initramfs initrd.img-5.4.51-v7l+ followkernel in /boot/config.txt like described on https://www.raspberrypi.org/documentation/configuration/config-txt/boot.md. That means the initrd is not loaded with a kernel command line on RPI4.

Encrypt Hook (You Can Skip This)

Apparently - some say - Raspbian doesn't have "systemd encrypt hook" (rd.luks.uuid/rd.luks.key kernel parameters), but only "encrypt hook" (cryptdevice/cryptkey kernel parameters) so let's try this first.

Look at /boot/cmdline.txt. It has root=PARTUUID=95acd0b5-02. 95acd0b5-02 is the GPT partition table's unique ID for the second partition of device /dev/mmcblk0 - the / you are on right now. Yeah just like file systems which use UUID, partition table also uses ids.

Change that to

cryptkey=UUID=7F52-3BB4:vfat:/slot-0.key cryptdevice=UUID=96645999-c47b-4b33-a463-feb737d0d101:luks-96645999-c47b-4b33-a463-feb737d0d101 root=/dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101 rootdelay=60

and remove splash and quiet.

Open /mnt/luks-96645999-c47b-4b33-a463-feb737d0d101/etc/fstab and change

PARTUUID=95acd0b5-02 / ext4 defaults,noatime 0 1

to

/dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101 / ext4 defaults,noatime,errors=remount-ro 0 1

In theory, on boot, kernel /boot/kernel7.img loads /boot/initrd.img-5.4.51-v7l, spins up raid devices, uses its cryptsetup binaries inside the initrd and the key on USB stick to unlock the LUKS filesystem 96645999-c47b-4b33-a463-feb737d0d101 on /dev/md2, the third raid partition, to the name /dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101, to which the kernel "switches root" - when it's ready - and is able to continue booting because its fstab is showing itself correctly.

It says /dev/mapper/luks.. is not present and Initramfs shell is presented. /dev/disk/by-uuid/7F52-3BB4 exists and when I mount that and do "cryptsetup..." it works, /dev/mapper/luks-.. appears. cryptkey/cryptdevice do nothing - the "encrypt hook" is a lie.

Systemd Encrypt Hook (You Can Skip This)

rd.luks.uuid=96645999-c47b-4b33-a463-feb737d0d101 rd.luks.key=96645999-c47b-4b33-a463-feb737d0d101=/slot-0.key:UUID=7F52-3BB4 root=/dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101 rootdelay=60

strings /lib/systemd/system-generators/systemd-cryptsetup-generator only shows the versions without rd. in them. We could try it but systemd-cryptsetup-generator isn't even in our initrd. According to its official manpage, https://www.freedesktop.org/software/systemd/man/systemd-cryptsetup-generator.html# "reads /etc/crypttab", which, mind you, isn't even able to use a key on an external media. If you man crypttab, it has the debian-only option keyscript, which does god knows what because then rd.luks.key=..external would be redundand hen egg shit.

As some correctly said, using fancy /etc/crypttab does not work for booting into encrypted root partition. This file can be used when the system is already booted, it has LUKS UUID (96645999-c47b-4b33-a463-feb737d0d101), optinal key (/mnt/usbstick/slot-0.key), and mapped name (luks-96645999-c47b-4b33-a463-feb737d0d101) to use cryptsetup open more conveniently.

Cryptroot Script (You Can Skip This)

According to https://www.ullright.org/ullWiki/show/initramfs-tools /usr/share/initramfs-tools/scripts/ have "boot scripts by packages" and local-top is "After these scripts have been executed, the root device node is expected" and we have /usr/share/initramfs-tools/scripts/local-top/cryptroot in it. dpkg -S /usr/share/initramfs-tools/scripts/local-top/cryptroot shows cryptsetup-initramfs and dpkg-query -L cryptsetup-initramfs also shows /usr/share/initramfs-tools/hooks/cryptroot. contrary to its containing directory name it's only for copying stuff to the initrd. /usr/share/initramfs-tools/hooks/cryptroot-unlock is to unlock something remotely. As we've seen /scripts/local-top appear, the cryptroot file is our customer, as the Red Baron would say.

root@raspberrypi:~# cat /usr/share/initramfs-tools/scripts/local-top/cryptroot|grep cryptdevice
root@raspberrypi:~# cat /usr/share/initramfs-tools/scripts/local-top/cryptroot|grep luks.uuid

In Shuttleworth's tradition, and Debian being "the rock of Ubuntu", we have a case of "not invented here" again. As we scan through the file, we see that it uses cryptopts kernel option instead, which does not support key files on external media either! Somebody changed that script to do it http://iutinfo2000.free.fr/Linux/bootkeyscript.

Actually Working AlienMind Hook

/etc/initramfs-tools/scripts/local-top/alienmind

1
2
3
4
5
6
7
8
#!/bin/sh

mkdir /a
mount /dev/sdc1 /a
cryptsetup --key-file /a/slot-0.key open UUID=96645999-c47b-4b33-a463-feb737d0d101 luks-96645999-c47b-4b33-a463-feb737d0d101
umount /a

exit 0

Then update-initramfs -c -k $(uname -r) again.

/boot/cmdline.txt change root=PARTUUID=95acd0b5-02 to root=/dev/mapper/luks-96645999-c47b-4b33-a463-feb737d0d101 rootdelay=60.

Reboot. Works.

Set Up Swap

Deactivate non working Ubuntu crap systemctl disable dphys-swapfile.

1
2
3
openssl rand -out /etc/swap.key 256
chmod 400 /etc/swap.key
cryptsetup --type luks2 luksFormat /dev/md1 /etc/swap.key

Now you can use /etc/crypttab

luks-swap /dev/md1 /etc/swap.key luks

Add to /etc/fstab

/dev/mapper/luks-swap swap swap defaults 0 0

Then

1
2
3
cryptdisks_start luks-swap
mkswap /dev/mapper/luks-swap
swapon -a

Aaaaand we're live. Enjoy 3.4T / and 8GB swap. Don't eat your USB stick ;)

Misc

1
2
3
systemctl disable bluetooth
apt-get install smartmontools
smartctl -d sat -a /dev/sda
Edit
Pub: 29 Aug 2020 21:16 UTC
Views: 374