Debian Stretch, BTRFS, & dm-crypt/LUKS

Goals:

  • Fully encrypted RAID6 BTRFS Array
  • Each drive is encrypted with a unique key file
  • Each keyfile is encrypted with GPG
  • Keyfiles are never on disk in their unencrypted format

Disk Layout:

As shown below, I'm working with 10, 2TB drives and a single 256GB SSD for the OS.

root@server1:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   1.8T  0 disk
sdb      8:16   0   1.8T  0 disk
sdc      8:32   0   1.8T  0 disk
sdd      8:48   0   1.8T  0 disk
sde      8:64   0 238.5G  0 disk
├─sde1   8:65   0 228.8G  0 part /
├─sde2   8:66   0     1K  0 part
└─sde5   8:69   0   9.7G  0 part [SWAP]
sdf      8:80   0   1.8T  0 disk
sdg      8:96   0   1.8T  0 disk
sdh      8:112  0   1.8T  0 disk
sdi      8:128  0   1.8T  0 disk
sdj      8:144  0   1.8T  0 disk
sdk      8:160  0   1.8T  0 disk

Setup the environment

Here is a simple function to set environment variables without having them saved to your shell history.

function prompt { echo -n "$1: "; read -s $2; export $2;}
# prompt GPG GPGPW
GPG: <enter pw here>

Creating the LUKS keys:

This generates 2048 bit keyfiles which will be used to unlock the LUKS containers for each drive. Since we have not formatted the drives, I use the drives 'sdX' name for now. Later we rename the keyfiles to use UUID's in case the drives get mapped differently in the future. I generated these on a tmpfs and then moved them off the system for storage.

Eventually, I plan to redo the encryption to use GPG key pairs of which I'll store & create the private key on a Yubikey 4.

for i in {a,b,c,d,f,g,h,i,j,k}; do openssl rand -base64 2048 | gpg --passphrase $GPGPW --symmetric --cipher-algo aes --armor >./server1-luks-sd$i.key; done

Creating the LUKS containers:

for i in {a,b,c,d,f,g,h,i,j,k}; do gpg --passphrase $GPGPW -q -d server1-luks-sd$i.key 2>/dev/null | cryptsetup -v --key-file=- --cipher aes-xts-plain64 --key-size 512 --hash sha
512 --iter-time 5000 --use-urandom luksFormat /dev/sd$i; done

Opening the LUKS containers:

for i in {a,b,c,d,f,g,h,i,j}; do gpg --passphrase $GPGPW -q -d server1-luks-sd$i.key 2>/dev/null | cryptsetup -v --key-file=- luksOpen /dev/sd$i luks_sd$i; done

Creating the array:

This will create the BTRFS array with both the data and metadata in RIAD6. You could switch to -m raid1 to have the metadata stored as in the RAID1 format, but I didn't really see a point.

Warning: As of this writing, BTRFS RAID5/6 support is only recommended for use in testing environments. For more information please see https://btrfs.wiki.kernel.org/index.php/Status and https://btrfs.wiki.kernel.org/index.php/RAID56

root@server1:~# mkfs.btrfs -f -d raid6 -m raid6 -L mainarray /dev/mapper/luks_sd[a,b,c,d,f,g,h,i,j,k]
btrfs-progs v4.4
See http://btrfs.wiki.kernel.org for more information.

Label:              mainarray
UUID:               12345678-1234-1234-1234-012345678901
Node size:          16384
Sector size:        4096
Filesystem size:    18.19TiB
Block group profiles:
  Data:             RAID6             8.01GiB
  Metadata:         RAID6             3.21GiB
  System:           RAID6            16.50MiB
SSD detected:       no
Incompat features:  extref, raid56, skinny-metadata
Number of devices:  10
Devices:
   ID        SIZE  PATH
    1     1.82TiB  /dev/mapper/luks_sda
    2     1.82TiB  /dev/mapper/luks_sdb
    3     1.82TiB  /dev/mapper/luks_sdc
    4     1.82TiB  /dev/mapper/luks_sdd
    5     1.82TiB  /dev/mapper/luks_sdf
    6     1.82TiB  /dev/mapper/luks_sdg
    7     1.82TiB  /dev/mapper/luks_sdh
    8     1.82TiB  /dev/mapper/luks_sdi
    9     1.82TiB  /dev/mapper/luks_sdj
   10     1.82TiB  /dev/mapper/luks_sdk

Rename keyfiles with UUIDs:

root@server1:~/temp# for i in {a,b,c,d,f,g,h,i,j,k}; do mv server1-luks-sd$i.key server1-luks-$(cryptsetup luksUUID /dev/sd$i).key; done

Decrypt LUKS container w/ UUID mappings:

This is horrible and gross but, gets the job done.
Alternatively, you can use a script like, https://github.com/jaredledvina/array-scripts/blob/master/array-unlock.sh to handle this for you in a more clean way.

for i in $(blkid | grep crypto_LUKS | awk '{ print $2 }' | cut -d '"' -f2); do gpg --passphrase $GPGPW -q -d server1-luks-$i.key 2>/dev/null | cryptsetup -v --key-file=- open /dev/disk/by-uuid/$i luks_sd$(blkid | grep $i | awk {' print $1 '} | sed -e 's/^.*\(.\).$/\1/' ); done

Conclusion

There you have it! A fully encrypted BTRFS RAID6 array. Since we have the keyfiles always encrypted with GPG (minus the moment we pass them to cryptsetup) you can shuffle them around and store them elsewhere and be reasonably confident they are secure. This setup even always separate passwords for each keyfile. The downside with this configuration is you can't easily automatically mount the array at boot. That's something I was willing to give up to increase the size of my tin foil hat.