Can't turn on after latest update (kernel), no response to touch input, USB keyboard not working

Hardware Librem 5
4

I tried that, renaming the files in several different ways (renaming all the 6.6 kernel and dtb files to .broken, creating new unversioned symlinks to the versioned files, renaming the 6.5 files to 6.6) but none of them worked, all resulted in a black screen with a green led.

I already backed up the root filesystem with tar, I’m trying to flash a new image now, after struggling for a while with librem5-flash-image not finding a matching image I found I need to add --stable for it to find it. Now I have another problem, the image is downloaded but the image fails checksum verification.

/Downloads/librem5-flash-image-v0.0.4$ ./scripts/librem5r4-flash-image --stable
2024-04-04 03:32:26 INFO Looking for librem5r4 plain byzantium image
2024-04-04 03:32:27 INFO Found disk image Build "stable" 'Last stable librem5r4 build' from Fri Jun 23 22:43:18 2023
2024-04-04 03:32:28 INFO Found uboot Build 85 from Thu Aug 25 15:22:41 2022
2024-04-04 03:32:28 INFO Downloading to ./tmp_librem5r4-flash-image_zflholae
2024-04-04 03:32:29 INFO Downloading image from https://storage.puri.sm/librem5/images/byzantium/latest/librem5r4/plain/artifact/librem5r4.img.xz
                                                                                                                                                                                                                  2024-04-04 03:34:57 INFO Calculating sha256sum of ./tmp_librem5r4-flash-image_zflholae/librem5r4.img                                                                                                                
2024-04-04 03:35:25 ERROR Checksum of image 27107f77b443a65e0446aec11c79754fdc623e3a40328d898b5fb0c0188fed70 does not match 6cfc1c2671a76baf3fc72ca611856143e3b74c969239ac8a695ce881a1467672                       
2024-04-04 03:35:25 INFO Cleaning up.

If I knew where to download and check the hash of the image and uboot I’d do that myself. I know where the image is, at that URL, not sure where the u-boot images are, looks like a jenkins job?
I see that the image checksum is in a file meta.yml next to the image.

1 Like
5

To see what is going on use UART

1 Like
6

I wasn’t really intending to spend so much time fixing my phone today, so I think reflashing will be faster. I know some debug information will be lost, but someone with time can hopefully reproduce the issue. I did find that the 400-something MB filesystem where the kernel images are stored was 95% full, with 4 or 5 kernel versions in it. It wasn’t 100% full though, and it seemed like all the 6.6 kernel files where there. Maybe some were missing or incomplete, I don’t know.

It’s odd that if I download the image myself and compute its decompressed checksum, I get a match. Why librem5-flash-image doesn’t, I don’t know, must be a bug.

xzcat  librem5r4.img.xz |sha256sum -b
6cfc1c2671a76baf3fc72ca611856143e3b74c969239ac8a695ce881a1467672 *-

cat meta.yml 

gitrev: afaabdbe43650b83744922353f86027cd555c474
image:
  size: 4500000256
  sha256sum: 6cfc1c2671a76baf3fc72ca611856143e3b74c969239ac8a695ce881a1467672
2 Likes
7

Ok so looks like ur Librem 5 not booting because there are not space to boooting. So becareful next time to have only one Gnu Core version.

So maybe Jumpdrive or UART will help to perform a: ‘apt autoremove’. However i never used Jumpdrive but UART.

1 Like
8

Does it need to write to the filesystem during booting? I don’t think so, usually failed kernel installations on other systems I’ve seen due to full /boot were not terminal, the boot configuration (grub) was not updated if the kernel failed to install. It’s weird that here it would. I’m suspecting more that something is wrong with the new kernel. It’s not good that there’s no way to recover or select an older kernel to boot with though, what’s the use of having multiple kernels installed? Okay, maybe with the serial console you can somehow, I’ve never tried it as I don’t have the breakout cable to connect to it.

1 Like
9

I managed to successfully reflash, now I just need to restore stuff like contacts, call logs, files from the backup I made. The trick was to download the image, decompress it, then run the update script like this:

./scripts/librem5-flash-image --dir . --stable
2 Likes
10

I think the incantation that you would previously have used (too late now) is:

sudo flash-kernel --force xyz

e.g. as here: Convergence Issues - #22 by ASwyD2

1 Like
11

I would have needed to boot into the OS first to do that. Maybe chrooting using Jumpdrive is possible, I’ve read about it in another thread.
I’ve installed updates and it’s still working, so it must’ve been the full filesystem that caused a failed kernel installation. I’m surprised that the kernel update procedure doesn’t have any failsafe to prevent this.

1 Like
12

I haven’t faced the situation but I would think: telnet in after Jumpdrive boots and then, yes, there is the possibility that you would have to chroot as well (would need to check what pathnames flash-kernel uses and what that looks like in the Jumpdrive environment).

1 Like
13

For what it’s worth, essentially the same thing happened to me yesterday; perhaps it’s a common problem?

All of the above happened exactly as jjakob said. Then I attached a keyboard using a USB hub (that I bought from Purism). It was a powered USB hub connected to power, so that’s not the problem. And it, and the keyboard, had worked fine with my Librem 5 a few days before.

1 Like
14

Did you find a solution?

1 Like
15

I found this Rescue system (jumpdrive), chroot, luks and kernel updates (#306) · Issues · Librem5 / OS-issues · GitLab
saying that Jumpdrive can’t open LUKS volumes.

https://people.debian.org/~praveen/librem5-rescue.txt
This is a rough idea of how to chroot with qemu-aarch64-static. Setting up resolv.conf isn’t required in this case as you only need to use flash-kernel. First you need to free space on the boot filesystem, I don’t know what the best way to do this is, maybe removing the files by hand (vmlinuz, initrd.img, System.map, config, dtb symlink and subdirectory under dtbs/). I also think this warning in the flash-kernel manpage needs more investigation:

WARNING: Take great care when running flash-kernel in a chroot, since the choice of machine may cause host filesystem partitions to be mounted and modified.

1 Like
16

I’m not 100% sure if my story below is related, but I experienced a quite similar symptoms as mentioned in this thread. Unfortunately, these symptoms were not very reproducible, so I will write here from my memory.

I experienced since the update to the 6.6 kernel series, a couple of weeks ago regular hangs, and created this issue for that: Regular hangs of L5 (#495) · Issues · Librem5 / linux · GitLab. Also I had often issues in booting up the Librem 5. For example the screen sometimes stayed black, sometimes I could enter the LUKS password, but the Librem 5 did not continue booting. They green led was sometimes on when this happened.

I even created this forum thread to try to get some info from other Librem 5 users regarding the instability: Stability, oopses, and traces of the Librem 5 - statistics and details

After the latest kernel updates, a few days ago (the same one as you referred to), I could not boot anymore at all as well.

To investigate the issue further, I booted with Jumpdrive, see also: Trying to use Jumpdrive: Possible eMMC defect

After I mounted the Librem 5 with Jumpdrive (on Fedora Linux) I could unlock the LUKS partition without any problems. I ran fsck on the file systems and noted that the boot partition had quite some errors. I let fsck fix them, but the Librem 5 still did not boot after this.

Then I decided to do a full reflash, following roughly these instructions:

After that, the Librem 5 booted perfectly.

But, I still have sometimes the half hangs, and difficulty booting the Librem 5. It seems to be in a weird state then. My guess is also that the file system corruption may have been caused by forcefully powering down the Librem 5, by long pressing the power button, or removing the battery.

1 Like
17

Those boot problems are being discussed in https://forums.puri.sm/t/screen-doesnt-come-on-after-power-on/

3 Likes
18

No, all I’ve done is come to this forum, and here I find that the methods suggested are all things I don’t know (Jumpdrive, UART). I’m sure I can learn them eventually, and I can get local help where I am, but it’s a little discouraging.

1 Like
19

@edrehasivar Please create a new thread and ask there for support, if you are looking for that. I’m totally willing to help you, as far as I can. I was also confused by what Jumpdrive is etc. I have detailed notes on what I did to get my Librem 5 working again. But I think we should not use this thread for writing that all.

3 Likes
20

I suspect it’s a bad update. I too have this issue and will need to use Jumpdrive it looks like. Make it a +1 for whomever is tracking these things. I’ve let support know too.

2 Likes
21

When I cannot turn on (only sometimes), i remove the battery, then put it back and it always works until now. It is frustrating but it is fortunately not often…

1 Like
22

please help me with this issue, I’m a complete noob.

1 Like
23

I don’t remember if I mentioned it but I didn’t have any success using jumpdrive to fix the issue, I’ve had to clean reflash (losing data). I’ve since set up backups with borgbackup (emborg), which is good since I’ve had to reflash once or twice again due to the LUKS volume somehow corrupting or the initramfs not being able to decrypt it. I could enter the password but it said it was incorrect even though I’m sure it was correct. I tried Jumpdrive, I can’t remember for sure but I think even there it didn’t accept my password. I had to reflash and restore from backup (call history, chatty db, address book). But since I’ve had to do this 2 or 3 times now, I can restore from backup pretty quickly, so it’s not that much of an issue. I just have to remember to run emborg once a week or so.