How to investigate and fix a Librem 5 installation that hangs at boot

Lliure · September 8, 2023, 3:52pm

If your Librem 5 hangs somewhere during boot, chances are – if you’re like me – that you prefer looking at the filesystem and making some changes in the hopes of fixing the issue, so that you can avoid reflashing and restoring everything from backup.

This guide explains, step by step, how to boot the Librem 5 into Jumpdrive and access the filesystem of the Librem 5.

Note that this guide has been compiled from several comments I’ve found on Purism’s forums (special thanks to @dos, @guru, @irvinewade, @mladen and others), plus my own trial-and-error work, all rolled into one guide.
It’s meant for people who feel grumpy and agitated because their phone doesn’t boot, and feel they just don’t have the nerves for looking through a billion threads right now.

This is how I successfully got my Librem 5 to boot again (after misconfiguring my /etc/crypttab and /etc/fstab yesterday).

Requirements

A Librem 5 that shows some signs of boot activity, displays the Librem 5 boot logo, maybe even asks you for your encryption passphrase.
But somehow, it still hangs during boot and never reaches the graphical session where you can unlock the screen and see the apps.
A working x86_64 host computer which runs Linux and has an USB port.
A USB-C data cable, for example the one that came with the phone.
The cryptsetup, telnet, udisksctl, and uuu tools installed on the host. How to get those depends on the specific Linux distribution you’re running on your host.
For example, on Arch Linux, the udisks2 package contains the udisksctl executable, while the mfgtools-git package on the AUR provides uuu.
(It’s ok to use another similar tool instead of telnet if you don’t like it; but telnet is what this guide uses.)
You should be somewhat comfortable with the Linux command line.
You should have a rough idea of what you’re actually going to do to the filesystem – once it’s mounted – in order to make the phone bootable again.
This guide will only walk you through mounting the Librem’s (root) filesystem.
For everything after that, you’ll be on your own.
You should have a backup of your data ready in case things go wrong.
This guide may contain errors, which might cause you to lose all data on your phone and on your computer.
If you’re reading this, it’s probably too late to make a backup given that your phone hangs at boot.
If you do have a backup though, one might argue that you don’t really need this guide (because you can wipe your phone and restore it from your backup.)
However, this guide is made for people like me, who prefer trying to fix an unbootable system first before wiping and restoring.
Needless to say, you’re using this guide at your own risk. Lliure takes no responsibility for any damage or data loss.

The steps

Download the latest Jumpdrive release for the Librem 5.
On the host computer, extract the Jumpdrive release package that you’ve downloaded to some (temporary) directory.
Turn off your Librem 5.
Make sure there’s some battery charge left to get you through
recovery.
Optionally, follow additional steps recommended by Purism support.
However, for me they’ve never been necessary; the following steps show an alternate way that works for me on Linux without having to remove the battery.
Use the USB cable to connect the Librem 5 to the host.
On the host, run sudo dmesg -w.
This step is not strictly necessary but useful for quick feedback.
On the phone, press and hold the volume-up button.

With the volume-up button still held down, push the power button for at least one second.
You should see something similar to this in the dmesg output:

usb 1-6: new high-speed USB device number 89 using xhci_hcd
usb 1-6: New USB device found, idVendor=1fc9, idProduct=012b, bcdDevice= 0.01
usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-6: Product: SE Blank M850 
usb 1-6: Manufacturer: NXP       SemiConductor Inc 
hid-generic 0003:1FC9:012B.00E3: hiddev98,hidraw4: USB HID v1.00 Device [NXP       SemiConductor Inc  SE Blank M850 ] on usb-0000:00:14.0-6/input0

Release the power button.
Release the volume-up button.

Optionally, double-check via lsusb on the host that it has recognized the serial device properly:

$ lsusb -d 1fc9:012b  # works for Evergreen; if unsure, run just `lsusb` without parameters
Bus […] Device […]: ID 1fc9:012b NXP Semiconductors i.MX 8M Dual/8M QuadLite/8M Quad Serial Downloader

On the host, go to the directory where you unpacked Jumpdrive. Then run:
```
./boot-purism-librem5.sh
```
After a few seconds, you should see the Librem 5 screen displaying a picture and a message that says Jumpdrive is running.
Make a decision on which pieces of the system you want to look at first:
a) an unencrypted partition, e.g. the boot partition; or
b) one of the encrypted partitions such as the root partition.
If you want to access the unencrypted partition, run the following command line on the host in order to connect to the Jumpdrive shell via Telnet:
```
telnet 172.16.42.1
```
This gives you an emergency shell with Busybox.
To mount the boot partition, run:
```
mkdir -p /boot
mount /dev/mmcblk0boot0 /boot
```
What you can actually do in /boot is quite limited.
For example, you may want to revert the initramfs to a known-good state, or clean up unneeded files to make space.
If you’re done inspecting and making your changes, run umount /boot and skip most of the rest of this guide, all the way to the step that says Turn off the Librem 5.

If you want to access one of the encrypted partitions such as the root partition on the eMMC card or maybe an encrypted partition on your SD card, the Telnet session won’t help you. This is because Jumpdrive doesn’t provide a cryptsetup executable that you’d need to decrypt those partitions.
Instead, we’re going to access those partitions on the host via USB.
First, identify the block device you want:

$ udisksctl status
MODEL                     REVISION  SERIAL               DEVICE
--------------------------------------------------------------------------
[…]
JumpDriv e eMMC           0509      Jumpdrive            sdc
JumpDriv e microSD        0509      Jumpdrive            sdd

Then identify the partition on sdc:

$ lsblk /dev/sdc
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sdc      8:32   1 29,1G  0 disk
├─sdc1   8:33   1  465M  0 part
└─sdc2   8:34   1 28,7G  0 part

Decrypt and mount the root partition (in this example it’s sdc2 but mind that yours may vary):

sudo mkdir -p /mnt/jumpdrive/emmc
sudo cryptsetup luksOpen /dev/sdc2 jumpdrive-emmc  # replace sdc2 with your individual device name
sudo mount /dev/mapper/jumpdrive-emmc /mnt/jumpdrive/emmc

Make all relevant changes to make the system bootable again
(e.g. fix /etc/crypttab, fix /etc/fstab etc.)

Unmount the eMMC partition and close the dm-crypt container:

sudo umount /dev/mapper/jumpdrive-emmc
sudo cryptsetup luksClose jumpdrive-emmc

Turn off the Librem 5.
Remove the cable.
Turn on the Librem 5.
Check if everything works now. If not, repeat to re-mount the filesystem and to try other fixes.
Alternatively, give up and re-flash your phone, then restore from backup.

This guide is licensed under CC0.
To the extent possible under law, Lliure has waived all copyright and related or neighboring rights to this guide.

Sources (selection)

Building uuu and Jumpdrive by @irvinewade on the Librem 5 Community Wiki
Comment on New Post: Reflashing the Librem 5 by @dos
Comment on Can’t use L5 after update! by @mladen and @dos
Comment on Phone is not booting anymore (after update?) by @guru

Sarcasmo220 · September 8, 2023, 9:35pm

Very helpful to post a guide like this. Just a suggestion, but it could be helpful to have “guide” in the title somehow like “Guide to investigate and fix…”

Lliure · September 8, 2023, 9:36pm

Good point. Done.

ChriChri · September 9, 2023, 12:26pm

To the wiki…?