PureBoot issue - boot stuck on "Checking verified default boot hash file"

Hello everyone,

I am stuck with a PureBoot issue, got a new Purism server and the plan is to re-use existing SSD with their data. Those 2 SSD are formated using BTRFS in a RAID1 volume.

I checked on heads-community matrix channel, and got feedback that @MrChromebox wasn’t sure if BTRFS would be supported by heads or not, anyhow, I decided to add (for now) a third disk to put the /boot partition on it.

After all those changes, of course PureBoot was unhappy and I also wanted to use my own GPG key with my own pin codes instead of factory ones.
So I went ahead and did a reset of TPM/TOTP/HOTP and generate a new GPG key, re-flash to add the GPG key in the bios permanently, reset the HOTP once more (since bios changed)…

Eventually, HOTP is happy and GPG files signature seems ok (boot hashes) but every time I try to boot, I am stuck in the boot process on this step “Checking verified default boot hash file” … I have once waited 20 minutes there, will wait a bit more now and see, but I guess there is something wrong :frowning:

Any idea ? Maybe @MrChromebox or @Kyle_Rankin ? Thanks :slight_smile:

More details on the boot steps:

so the way PureBoot (Heads) boot verification works is that all files in /boot are hashed and then verified prior to each boot. If your /boot partition contains more than just your kernels and grub config files, it’s going to take a lot longer. I’ve not seen it take 20 minutes though, but I have no idea how many files are on your /boot partition.

However, it looks like it’s past that and stuck checking the default boot target, which is simply comparing the saved default grub entry/index to the current list of grub entries. That should be instantaneous, so I’d presume something has hung.

edit: also, pinged you on Matrix. Have an updated Pureboot server build for you to try

Thanks - the problem on matrix is that my homeserver is on the system I am migrating to this new chassis so I didn’t saw your message :frowning: (could you repost here or in PM as you see fit?)

From the message heads display, it seems to be stuck on this line: https://github.com/osresearch/heads/blob/3c551cc249e4cba4207409c735e43f07aa1f0daa/initrd/bin/kexec-select-boot#L236

I can double check but I doubt I have anything else that kernel / grub stuff on this /boot.
Anyway, it is 1g size so there can’t be much data indeed.

sent you a PM

Just in case someone would run into the same issue (for symptoms see screenshot above):

The root cause was a “bad” /boot/grub/grub.cfg file.
That files contains your boot entries and is auto-generated by update-grub. Those boot entries set, among other things, the path to the kernel ELF (vmlinuz file) and the initrd file.
In my case, those file paths were absolute, like : /boot/vmlinuz-version

Apparently this can confuse PureBoot and make it generate an incorrect /boot/kexec_default.*.txt file which then result in a boot process being stuck as per above (stuck on step “Checking verified default boot hash file”).

How to fix:

  • With the PureBoot rescue shell: just edit the /boot/grub/grub.cfg file and make sure all records of kernel paths or initrd paths do not include /boot. They should look like /vmlinuz-version or /initrd-version
    You will need to resign /boot and then choose a new default boot option.
    Side note: /boot is mounted read-only in the rescue shell, so you need to mount -o remount,rw /boot
  • When the OS is booted: it seems that update-grub generates the grub.cfg file correctly if there are symlinks at root level pointing to the /boot files. Ie: /vmlinuz-version -> /boot/vmlinuz-version. I didn’t test this but I think that in absence of those symlink, update-grub then generates a grub.cfg file with full paths that PureBoot will have trouble to work with.

Voila, now everything works perfectly fine on my side, should anyone encouter the same issue, hope this can help.

Thanks for your time in helping me troubleshoot this @MrChromebox

4 Likes

thanks for following up with the issue/workaround :slight_smile:

I’ve reproduced the issue with absolute paths for the kernel/initrd, and patched Pureboot to strip the boot dir from the entries. Will be included in next release.

1 Like