Understanding the PureOS default disk encryption (luks, cryptsetup) / possible security issue

[ChriChri]

When I received my Librem 13 I followed the initial setup after switching it the first time on.

After setting everything up (including a password for the disk encryption) I started to look around and accustom myself to the new enviroment. After these steps I had the impression I’d have a savely encrypted notebook.

I will try to explain why I had to change my mind about this feeling, how I resolved the issue on my Librem 13 and what could be done to improve the situation - if I didn’t get thinks wrong and all of it is only my misunderstanding.

I bought a Librem 13 because I very much like the philosophy behind the products of Purism, but I generally do not trust anyone.

Remembering that with luks and cryptsetup one can store more then one password I read a bit into luks and cryptsetup and found on my Librem 13 that there are two passwords set for the encryption of my /root device. Searching for an explenation I found the discussion here Hidden luks password automatically enabled in default PureOS? .

After reading through it I became interested in how the same data could be decrypted with several different passwords.

I found this explanation: https://crypto.stackexchange.com/questions/24022/luks-multiple-key-slots-whats-the-intuition . It describes that the data on the disk is encrypted with one masterkey. For each password used on the device this masterkey is encrypted with the passphrase and stored in one of the key slots of the header of the encrypted device.

When decrypting the device the given password is tried agains each of the used key slots and if it can decrypt the masterkey in one of the key slots the device can be decrypted using the masterkey.

Thinking some more about it I realized that the time I had to wait for the gnome-initial-setup after I first switched on my new Librem 13 wouldn’t have been long enough to encrypt the whole volume of ~450GB with a new masterkey and I suspected that the volume had already been encrypted when I received the device and I only set the passphrase to encrypt the same masterkey that had been generated on the notebook at Purism when PureOS had been put on the disk.

I looked at the source of gnome-initial-setup and found the place where the password I entered is used to setup encryption: https://source.puri.sm/pureos/core/gnome-initial-setup/blob/master/debian/patches/04_new-luks-page.patch#L132 . gnome-initial-setup calls the method SetInitialDiskPassword via dbus.

In package pureos-init-disk-crypto I found that the dbus call in the end calls _set_initial_disk_password which then sets the password to encrypt the existing masterkey (https://source.puri.sm/pureos/core/pureos-init-disk-crypto/blob/master/daemon/crypto_setup.py#L142).

From this I assumed that I was right and the masterkey the data I’d put onto my Librem 13 would be encrypted with had already been on the Librem 13 since Purism put the software onto its disk. Since I didn’t need any further information like an initial setup password provided by Purism to start the initial setup the masterkey for encryption and all information to read (and if necessary to decrypt it) had been on the disk during processing and delivery.

If this would be true all people who handled my notebook at Purism, while shipping, at customs and in my company before I put my hands on it could have copied my masterkey for disk encryption without my knowledge.

Wouldn’t be to bad one might think, because the only information on the disk had been the bare operating system PureOS. And in the end I entered my individual password to protect the disk.

But remember: My individual password is only used to finally encrypt that masterkey used to protect my disk - exactly that masterkey that sat unencrypted or together with all information to decrypt it on my disk on its way from Purism to my desk.

So after setting an encryption password on my possibly already compromised disk encryption masterkey I’d have put my data I wished to protect on the disk and the person who had her/his hands on my disk encryption masterkey even before my notebook reached my desk could any time in the future use this information to decrypt my disk or a copy of my disk.

I wrote to the original thread to get a confirmation of this problem or to get information where I assumed something wrong and waited for an answer.

Meanwhile I read a bit more about luks and cryptsetup found found cryptsetup-reencrypt https://manpages.debian.org/testing/cryptsetup-bin/cryptsetup-reencrypt.8.en.html which is able to change the masterkey of the disks encryption while the disk is not being used/mounted.

Since I didn’t have data to loose I used it successfully after booting a rescue system from an usb stick.

Finally I chatted via matrix https://matrix.to/#/!bGtSETqkYFOebMtlfH:talk.puri.sm?via=talk.puri.sm&via=matrix.org&via=librem.one with Jeremiah Foster and Kyle Rankin who confirmed my thoughts.

I asked about the way they put the initial PureOS installation onto the disk of the Librems - whether they do it individually or from an image. As they do claimed to do it individually I assume that each Librems disk has its own unique masterkey.

If the disks would be created from an image (like I did it several times for mass installations of unencrypted systems using PXE and a small script to automate the process) the initial masterkey for disk encryption would be the same on multiple devices.

One way to check this would be to publish the hash of the masterkey of several devices. If these hashes would be the same for different devices each owner of one of those devices with the same masterkey could probably decrypt the disk of any other device without knowing the encryption password the owner of that device set.

If I didn’t get thinks wrong (I’d love an explanation how I came to a wrong conclusion) I suggest

• that information about the risk of using the factory set disk encryption masterkey is explained to the user during initial setup to give the user the oportunity to take further measure (like a new installation or re-encryption of the disk)
• that latest when online reencryption (cryptsetup 2.2.0 as Jonas wrote on the chat) is available in PureOS the process of reencryption with a newly created masterkey is automated as part of the initial setup

[Xx5]

I have to say I agree that this is something new users should be aware of. I didn’t think of this possibility and it does seem as though it could be a serious issue in many situations for those uninformed of the risks.

For instance say a couple where the husband/wife receives delivery of the laptop and is able to obtain the master key before the other takes ownership. This could compromise someone in certain situations that could become very serious if one had the belief that encryption is only set up after the password is in place.

While having physical access would certainly already be a problem in these situations, the ease of which someone could access encrypted data after this without any kind of rootkit/etc. or any noticeable change to the device whatsoever makes it scary in my opinion. Especially for someone who believes they set up encryption themselves.

[ChriChri]

Howto re-encrypt your disk

As I already wrote I re-encrypted my disk using a new encryption master key to get rid of the risk that the original master key that had been generated during installation of the harddrive at Purism already has been compromised at Purism, while shipping my Librem or while my Librem passed through our company.

I compiled the steps I took so that people who’d like to re-encrypt their devices, too, could hopefully follow them.

If re-encryption does not work as expected for you all of the data on your disk might get lost. Don’t try this without a backup.

During the process you’ll generate a sha256 checksum for the old and new master keys for the disk encryption. You can compare them and make sure that the masterkey for your disk encryption really changed.

Also, I’d like to ask you to publish the checksum of your old master key in this thread: If we’d find that different people received the Librems with the same checksum of their master keys (aka the same master keys) it could be easily explained by Purism putting disk images on the new Librems (which to my knowledge is the fastest way of installing a number of devices if you grow afterwards the device size to the final size).

If this would be true the encryption of these devices would be useless in any case, because there’d be a good chance that someone could get their hands on those master keys used multiple times and just try them on Librems.

Here’s a short overview of the steps I took:

• find boot and root devices
• show information about encryption
• boot rescue system
• mount unencrypted boot and encrypted root devices of PureOS
• save sha256 checksum of encryption master key
• umount root device of PureOS
• re-encrypt root device with new master key
• show information about (new) encryption
• mount encrypted root device
• show sha256 checksum of new encryption master key
• re-add crypto_keyfile.bin to LUKS keys
• umount devices in rescue system, reboot

find boot and root devices

To find the devices containing your root and your boot filesystem you can use the following commands:

root device:

root@grml:# grep -v '^#' /etc/fstab | col | egrep '[[:space:]]+\/[[:space:]]+' 
/dev/nvme0n1p2

boot device:

root@grml:# grep -v '^#' /etc/fstab | col | egrep '[[:space:]]+\/[[:space:]]+' 
/dev/nvme0n1p1

These commands are taken from a script by Kyle Rankin.

So my devices are:
/ (root) = /dev/nvme0n1p2
/boot = /dev/nvme0n1p1

If your system uses different devices you’ll have to substitute my devices with your devices in all of the commands shown below.

E.g.:
If I write

ls /dev/nvme0n1p2

and you found that your root device is /dev/sda2 then you need to change it to

ls /dev/sda2

to make it work on your system.

show information about encryption

Below is how you can look at the non-secret information in your encrypted device that describes how your device is encrypted.

root@grml:# cryptsetup luksDump /dev/nvme0n1p2
LUKS header information for /dev/nvme0n1p2

Version:       	1
Cipher name:   	aes
Cipher mode:   	xts-plain64
Hash spec:     	sha256
Payload offset:	4096
MK bits:       	512
MK digest:     	89 3b b1 57 52 87 3a b7 5b a6 6c 26 84 2b ed 84 fa d0 47 1c 
MK salt:       	72 c4 af cb 2a 5b a6 5c 0f 24 b9 e4 22 54 fa 62 
               	be f6 65 78 22 65 13 f1 d6 09 e5 23 f5 97 4a 25 
MK iterations: 	104356
UUID:          	64e871c7-a683-6de5-9a8d-2f7828012db0

Key Slot 0: ENABLED
	Iterations:         	1624562
	Salt:               	91 c4 d6 80 25 6a 29 7c 5e 80 df 44 70 48 1d 81 
	                      	ff 99 63 99 0b 64 b1 97 28 6d 02 be 66 32 3e 62 
	Key material offset:	185
	AF stripes:            	4000
Key Slot 1: ENABLED
	Iterations:         	1834526
	Salt:               	e2 3a f5 72 d5 06 50 30 d5 68 d5 b1 5a 1e 54 e8 
	                      	68 e3 d7 68 ac 18 fe 67 46 b5 b6 42 e7 c1 48 66 
	Key material offset:	512
	AF stripes:            	4000
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

To find out more about cryptsetup and its functions you’ll need to read the man page of the program by typing

man cryptsetup

In my example above you can see that two key slots are used - each of it containing the same master key for that device, but encrypted with two different passwords. One of the passwords is the one I choose setting up my Librem the first time and the other can be decrypted with the content of the file /crpto_keyfile.bin .

To understand why is that you can read the explanation here.

You don’t need to take note of these informations. They are not needed for the following steps. It’s just to make you aware how things work and to understand later that after re-encryption one key slot less is used and we have to fill up the second one manually again.

boot rescue system

I used the boot image from https://grml.org/ and put it on an usb stick. Generally you can use any rescue image you like that contains the program cryptsetup-reencrypt and preferable is based on Debian Buster. I choose to use GRML, because it offers to choose my keyboard layout in its boot menu.

To put the rescue image on an usb stick I followed “2. install using dd” in GRMLs wiki.

mount unencrypted boot and encrypted root devices of PureOS

Create mountpoints:

root@grml:# mkdir /mnt/boot
root@grml:# mkdir /mnt/root

Mount boot device:

root@grml:# mount /dev/nvme0n1p1 /mnt/boot

unencrypt root device and make it available as /dev/mapper/root-dev:

root@grml:# cryptsetup luksOpen --readonly /dev/nvme0n1p2 root-dev
Enter passphrase for lukstest.img: 

mount the newly created decrypted device:

root@grml:# mount -oro /dev/mapper/root-dev /mnt/root

check whether everything worked:

root@grml:# mount | grep mnt
/dev/nvme0n1p1       on /mnt/boot type ext3 (rw,relatime,data=ordered)
/dev/mapper/root-dev on /mnt/root type ext3 (ro,relatime,data=ordered)

root@grml:# ls /mnt/root/crypto_keyfile.bin
/mnt/root/crypto_keyfile.bin

You now can write things to /mnt/boot for further reference and you can read the crypto_keyfile.bin of your root device. The crypto_keyfile.bin contains a password to unlock one of the key slots of your luks device.

save sha256 checksum of encryption master key

Now we will use the password in crypto_keyfile.bin to dump the encryption master key. I chose to do it this way because I didn’t want anybody to print her/his master key without need in clear text to the screen. This way we can cut of the part of the output of cryptsetup containing the master key and put it through sha256sum which will print its checksum in a hopefully reproducible way.

Since it is extremly difficult to find all the inputs that would create the same hash and then find in all of these endless variations the only one that is your master key, in my opinion it is safe to print this hash to your screen and safe it for further reference into a file.

Furthermore hopefully the re-encryption of your device with a new master key will work and the hash of your now still active and then old master key will only serve to compare and make sure the master key really changed.

Also I’d like to ask you to publish the hash we’ll now write into /mnt/boot/old-masterkey.sha256.txt (which in your running PureOS will be /boot/old-masterkey.sha256.txt) in this thread. This helps to make sure that there are no Librems that have for any reason been delivered with the same master key (in which case I’d strongly suggest that the owners need to be informed personally).

root@grml:# cryptsetup luksDump -q --dump-master-key --key-file /mnt/root/crypto_keyfile.bin /dev/nvme0n1p2  | grep -A4 '^MK dump:' | sha256sum | tee /mnt/boot/old-masterkey.sha256.txt
a9f3b6bb0389747d9914b2635ab0285cc572e703aabfd02c116e8f115f12c94b  -

umount root device of PureOS

For re-encryption of the device I closed and unmounted it. I’m not sure if this is necessary since the man page states:

Cryptsetup-reencrypt reencrypts data on LUKS device in-place. During reencryption process the LUKS device is marked unavailable.

root@grml:# umount /mnt/root
root@grml:# cryptsetup luksClose root-dev

re-encrypt root device with new master key

With the reencryption part I had a problem: Either I entered several times a bad password or the version of cryptsetup-reencrypt I used didn’t try all the key slots with the password I provided. I ended up giving a parameter to define which key slot should be used.

If you’ll need to do the same, here’s how to find the right key slot:

root@grml:# cryptsetup --verbose luksOpen --test-passphrase /dev/nvme0n1p2
Enter passphrase for /dev/nvme0n1p2: 
Key slot 1 unlocked.
Command successful.

In my case the key slot I could decrypt with my password is key slot 1.

To re-encrypt the device we use the following command. The parameter -S1 defines that we want to use key slot 1. You could also try to run the command without the -Sx option - in theory cryptsetup-reencrypt should choose the right key slot.

Before running the re-encryption command make sure that you have a backup and that your notebook is connected to power and your battery is full. The manpage of cryptsetup-reencrypt states the following warning:

WARNING: The cryptsetup-reencrypt program is not resistant to hardware or kernel failures during reencryption (you can lose you data in this case).

ALWAYS BE SURE YOU HAVE RELIABLE BACKUP BEFORE USING THIS TOOL.
THIS TOOL IS EXPERIMENTAL.

Here we go:

root@grml:# cryptsetup-reencrypt -S1 /dev/nvme0n1p2
Enter passphrase for key slot 1:
Finished, time 46:15:192, 469674 MiB written, speed 169,2 MiB/s

As you can see the re-encryption of my 500GB NVMe took about three quarters of an hour.

show information about (new) encryption

If you look now at the content of the luks header (like above):

root@grml:# cryptsetup luksDump /dev/nvme0n1p2

You’ll note that only one key slot is used: It contains the password you typed. We’ll re-add your crypto_keyfile.bin later.

mount encrypted root device

To do so we need to unlock and mount the device again:

root@grml:# cryptsetup luksOpen --readonly /dev/nvme0n1p2 root-dev
Enter passphrase for lukstest.img: 
root@grml:# mount -oro /dev/mapper/root-dev /mnt/root

re-add crypto_keyfile.bin to LUKS keys

Now we can add the password in crypto_keyfile.bin again:

root@grml:# cryptsetup luksAddKey /dev/nvme0n1p2 /mnt/root/crypto_keyfile.bin
Enter any existing passphrase:
cryptsetup luksAddKey /dev/nvme0n1p2 /mnt/root/crypto_keyfile.bin 6,74s user 0,02s system 76% cpu 8,841 total

show sha256 checksum of new encryption master key

Let’s see whether we changed the master key for disk encryption successfully:

root@grml:# cryptsetup luksDump -q --dump-master-key --key-file /mnt/root/crypto_keyfile.bin /dev/nvme0n1p2  | grep -A4 '^MK dump:' | sha256sum; cat /mnt/boot/old-masterkey.sha256.txt
f2ca1bb6c7e907d06dafe4687e579fce76b37e4e93b7605022da52e6ccc26fd2  -
a9f3b6bb0389747d9914b2635ab0285cc572e703aabfd02c116e8f115f12c94b  -

In my example the re-encryption worked and the sha256 checksum for the masterkey changed.

umount devices in rescue system, reboot

root@grml:# umount /mnt/*
root@grml:# cryptsetup luksClose root-dev
root@grml:# reboot

After you started your PureOS again (hopefully succesfull) you could do to us Librem and PureOS users two favors:

• Write a short reply that this howto worked for you / didn’t work for you / you didn’t understand some things to help us improve it
• If you’re feeling save with it include the content of your /boot/old-masterkey.sha256.txt to help to find out whether this possible security issue might be more severe than we thought until now

[2x5]

So, what does this mean?

Can somebody confirm that the https://pureos.net pitch may be leaving out some details?:

PureOS allows you to easily encrypt your whole operating system and data, with your own encryption keys [in addition to the master key?], whether you download and install it yourself or receive it [with a purism master key?] preloaded.

???

[jeremiah]

We encrypt the laptops that we ship from our secure facility. For many of our customers, this is added value, because the encryption setup is complex and can easily brick your device. We do this because it makes life a lot easier for many people. Those who feel that the risk of someone possibly tampering with your device during shipping or before it reaches you, can go through the process to re-encrypt the device.

[2x5]

Many thanks, that is perfectly fine. Please, just add the missing master key bits to the pureos website (or remove that part there and add it at least to the order and shipping info, and the user’s manual).

[ChriChri]

In my opinion this is an issue that needs to be taken care of. I don’t say I don’t like the default encryption - far from it, I love it.

But for the goals of Purism to give back the decision about their data to the users the user needs to be informed where a risk is taken. It all is just a matter of information.

An additional text dialog during setup could easily give the information to the user and could easily point to a resource which explains the background.

[kieran]

If this would be true the encryption of these devices would be useless in any case

it could be easily explained by Purism putting disk images on the new Librems (which to my knowledge is the fastest way of installing a number of devices if you grow afterwards the device size to the final size)

That would certainly be less than “best practice”.

Could Purism urgently clarify whether this is the way it works with Librem laptops?

Maybe the same issue arises with the soon-to-ship Librem 5 (where it might be much harder to resolve)?

[kieran]

WARNING: The cryptsetup-reencrypt program is not resistant to hardware or kernel failures during reencryption (you can lose you data in this case).

ALWAYS BE SURE YOU HAVE RELIABLE BACKUP BEFORE USING THIS TOOL.
THIS TOOL IS EXPERIMENTAL.

This might be a silly question but why wouldn’t you just install from scratch? i.e. blow everything away and start again.

Isn’t that ultimately more robust if you are concerned about problems in having an encrypted system already arrive encrypted?

[moe]

Great posts, thanks @ChriChri

I tried following those steps but got a strange error when trying to run cryptsetup-reencrypt:

data offset is not multiple of 4096
failed to set data offset

Can anybody offer any insight into what this means? Everything went fine up until that step, I confirmed it was the right volume and key slot, etc.

Thanks in advance!

[ChriChri]

I tried following those steps but got a strange error when trying to run cryptsetup-reencrypt:

data offset is not multiple of 4096
failed to set data offset

Didn’t run into that. Maybe some more information could shed some light on the cause of the problem. Could you post your cryptsetup luksDump? From the message I’d expect that Payload offset: in your configuration is different.

Which Librem are you using and when did you get it (the encryption might be generated with some older version of cryptsetup with some other defaults)?

Which linux boot image did you use and which version of cryptsetup-reencrypt did it contain?

[jeremiah]

To follow up with ChriChri, he’s likely referring to this full command;

$ sudo cryptsetup luksDump /dev/nvme0n1p2

Of course on your machine the device might be different or on a different partition. Look for “Payload offset”.

[moe]

I have Payload offset: 4102. I did a bit of searching but I’m not entirely sure what this means!

I was using a latest Debian live image with cryptsetup-reencrypt version 2.1.0 and btw I have a librem 15v3.

Thanks again!!

[jeremiah]

This may be a bug or issue with the Debian version of cryptsetup.

[ChriChri]

@moe: cryptsetup changed its api between version v2.0.6 and v2.1.0. The error message you see has been introduced with version v2.1.0. In the older version v2.0.6 and before the way to find the beginning of the data in an cryptsetup encrypted device had been different.

I’m not a programer and I didn’t read much of the code, but maybe you’d have a chance to reencrypt if you’d try with a debian strech based boot iso containing cryptsetup v2.0.6. If this succeeded I’d say it might be worth to open an issue for cryptsetup v2.1.0 . As far as I understood the api changes should have been 100% compatible to older versions…

Just a guess, though.

Edit: I found some more interesting information regarding the Payload offset here. As I understand it could also be that your device uses some cipher/keylength/whatever combination that led to that strange offset. But still, there is no reason, why it should’nt work with an actual version of cryptsetup-reencrypt.

Maybe you could provide the information on the used encryption like this:

user@system:~$ cryptsetup luksDump cryptsetup-test.img | head -n8
LUKS header information for cryptsetup-test.img

Version:       	1
Cipher name:   	aes
Cipher mode:   	xts-plain64
Hash spec:     	sha1
Payload offset:	4096
MK bits:       	256

[2disbetter]

First off thank you for discussing this. Secondly thank you for the instructions to reencrypt. For those who trust Purism this is a fantastic guide. Lastly, I think the issue is not as severe as you are portraying.

By your own admission, you don’t trust anyone. For this to be true, you should have wiped your drives and started from scratch using software you were completely confident with.

This issue isn’t something I think most people have a problem with as they want to trust someone and Purism has given them more reason than any other company to trust them.

As was mentioned, I think the way things are done currently are because it was one of the safer ways to permit an encrypted system with as small of a margin of error from user mistakes.

Overriding all of this discussion though and underpinning your own fears is that the means to completely reinstall the system and encrypt it from scratch are made available to you by Purism. On top of this as was suggested by @kieran if you truly don’t trust anyone and feel that your system may have been compromised then how are you suggesting that reencrypting a compromised drive is a good solution?

[ChriChri]

By your own admission, you don’t trust anyone. For this to be true, you should have wiped your drives and started from scratch using software you were completely confident with.

Yes, would be a better start. Even better would be to build my own hardware, to be sure to understand what it does and be safe. Or I could just stop using stuff I do not fully understand. But this all for me is not practical.

Also I do not write my own software and I have to trust Microsoft, Google, Debian or Purism. I decided to try PureOS.

And I think from the way Purism presents themselfs I owe them my positive friendly mistrust.

As was mentioned, I think the way things are done currently are because it was one of the safer ways to permit an encrypted system with as small of a margin of error from user mistakes.

As I wrote: The information about how that default encryption works and which compromises are taken should be given to the user.

In my opinion this is an issue that needs to be taken care of. I don’t say I don’t like the default encryption - far from it, I love it.

But for the goals of Purism to give back the decision about their data to the users the user needs to be informed where a risk is taken. It all is just a matter of information.

On top of this as was suggested by @kieran if you truly don’t trust anyone and feel that your system may have been compromised then how are you suggesting that reencrypting a compromised drive is a good solution?

Let’s say you’d want to make sure that you’d have access to the data on disk drives of Purism customers at some point in the future if needed. What would you do? Alter some software on the notebook being delivered and taking the risk of that being uncovered? Would you build some hardware device into those notebooks and take the risk that being uncovered?

The way you’d do it now you would simply copy the master key for disk encryption without anybody ever being able to prove that you did so. You couldn’t be uncovered or detected taking this step by checking on the hardware or software of those notebooks.

Later, if you’d get into the need to read users data, you could decide which measures and risks to take to get your hands on the raw data on that harddrive.

If the user changed the master key for disk encryption this wouldn’t work.

In the end you’re right. This is just one little step improving one aspect of security. But knowing this everybody can make their own decisions.

[moe]

Thanks again @ChriChri
Your advice worked. I tried again using an Ubuntu live image which uses cryptsetup-reencrypt 2.0.2 and it just worked!

For the record, the previous key: MK digest: 23 98 08 be 62 c8 b4 9c 2e 6b b4 4e 3c f5 c7 40 c0 89 cc 6b

It’s an interesting debate about whether or not it’s really necessary to reenncrypt disks. I agree that it’s a quite niche risk that’s being addressed… Even if somebody were retaining the factory master key, wouldn’t they need physical access to the disk to do anything about it? If they have that you’re probably in trouble anyway.

Since it’s a one of job I think it’s worth doing but yeah, it’s more about users being aware so they can decide for themselves.

[ChriChri]

@moe To help to improve the world you could now file an issue on gitlab to give the developers feedback and the opportunity to improve on this problem or just document it.

Still it’s an interesting question why your Payload Offset is different.

[Skalman]

@ChriChri thanks for explaining all that so thoroughly! It was a few years ago, do you know if anything important has changed since then regarding these things?