Hard resets triggered by installing Flatpak app on microSD

This is clearly a continuation of this thread:

Since then I have got rid of the questionable practice of storing swap on microSD on Librem 5, which made life better. I have also upgraded to Crimson. But the problem is still there: high IO on microSD crashes the phone.

In particular, now I need to install a Flatpak app onto an microSD installation. To be more precise, I have a LUKS inside LVM on a microSD. I have tried different tricks to limit IO, but each time the phone reboots with a short loud vibration.

I also tried limiting IO bandwidth with cgroups as following, but the phone still crashes.

systemd-run --pty \
            -p 'IOWriteBandwidthMax=/dev/dm-2 2M' \
            -p 'IOReadBandwidthMax=/dev/dm-2 2M' \
            -p 'IOWriteBandwidthMax=/dev/dm-0 2M' \
            -p 'IOReadBandwidthMax=/dev/dm-0 2M' \
            -p 'IOWriteBandwidthMax=/dev/sda 2M' \
            -p 'IOReadBandwidthMax=/dev/sda 2M' \
            \
            -p 'IOWriteIOPSMax=/dev/dm-2 2K' \
            -p 'IOReadIOPSMax=/dev/dm-2 2K' \
            -p 'IOWriteIOPSMax=/dev/dm-0 2K' \
            -p 'IOReadIOPSMax=/dev/dm-0 2K' \
            -p 'IOWriteIOPSMax=/dev/sda 2K' \
            -p 'IOReadIOPSMax=/dev/sda 2K' \
            flatpak --installation=sdcard install "$APP_NAME"

I also tried to manually pause installation at intervals and resume with the following script only after dstat -tdD total,sda,mmcblk0 5 reported almost no IO activity, but I have not succeeded:

pkill -CONT -e -f /bin/flatpak
sleep 20
pkill -STOP -e -f /bin/flatpak

I have repeatedly tried to install app this way more than a dozen of times in a row with no success. This is frustrating.

UPD: Another symptom is that the phone’s screen often freezes when there is such high IO. When it freezes like that, it has a chance to reset with a buzz, as described above.

I also noticed, that when I write to the same microSD from a laptop using a USB adapter, there is a USB traffic (as reported by usbmon) for a long time after block device IO stops being reported. The block device can not be unmounted until the USB traffic stops. Sadly, I have not found a tool to similarly monitor bus IO on microSD reader in Librem 5.

For fault isolation purposes, is it possible to install the app on the exact same µSD card but without LUKS and LVM i.e. vanilla file system? Or keep the volume arrangements as they are but try a different µSD card?

Presumably just some host-side caching?

Yes, but this is strange. We have two caches between a userspace program and hardware:

  • Delayed block device writing. A “write” system call returns before actual write finishes. The data is cached by kernel and gradually written to a block device. This is best illustrated by the iotop program, which displays separately IO between userspace and kernel and between kernel and block device.
  • A strange bus cache. Kernel’s block device engine thinks that data is already written, but there is another cache somewhere between the block device abstraction and the bus.

I focus on these caching peculiarities because it seems like phone crashes as soon as one of these caches fills. I have just managed to finally install the app after God knows how many attempts by manually pausing the process and waiting for scheduled writes to finish. Sadly, this is not a solution, but an ugly workaround.

Good idea, I’ll try it once I find my spare microSD. The same problem happens when copying large amount of files with rsync without bandwidth limit (i.e. the --bwlimit argument), so it will be easy to reproduce.

I’ve been running my flatpaks (and waydroid) from a 1TB microSD and I’ve never had any issues. The microSD is encrypted with LUKS. I’ve ran rsync backups from it, plus the native pureos backup and also rsync copies between internal storage and the microSD both ways. I’ve done 100GB+ copies via wired ethernet to and from the microSD card into my nas as well.

I’d suggest checking io usage real time (maybe via serial) and, yes, as already suggested, trying another card.

I remember changing the microSD card at least once. Maybe, it depends on manufacturer, its declared IO speed, or something like that. :thinking:

The first thing to do when this happens is to check the files in /var/lib/systemd/pstore to see what exactly made the kernel unhappy.

Librem 5’s microSD reader is a USB device, so the exact same tools apply.

I see /var/lib/systemd/pstore/dmesg-ramoops-0 file with today’s date in modification timestamp which ends with this:

dmesg-ramoops-0
<3>[ 4108.814536] INFO: task phoc:1120 blocked for more than 120 seconds.
<3>[ 4108.821051]       Not tainted 6.12.0-1-librem5 #2
<3>[ 4108.825961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[ 4108.834006] task:phoc            state:D stack:0     pid:1120  tgid:1120  ppid:1      flags:0x00000004
<6>[ 4108.843444] Call trace:
<6>[ 4108.846069]  __switch_to+0xf0/0x150
<6>[ 4108.849687]  __schedule+0x2f0/0xc60
<6>[ 4108.853380]  schedule+0x3c/0xd8
<6>[ 4108.856642]  io_schedule+0x44/0x68
<6>[ 4108.860106]  folio_wait_bit_common+0x174/0x3d0
<6>[ 4108.864649]  folio_wait_bit+0x20/0x38
<6>[ 4108.868414]  folio_wait_writeback+0x54/0xc8
<6>[ 4108.872673]  migrate_pages_batch+0x248/0x1080
<6>[ 4108.877095]  migrate_pages_sync+0x158/0x230
<6>[ 4108.881395]  migrate_pages+0xac0/0xb80
<6>[ 4108.885207]  __alloc_contig_migrate_range+0xfc/0x368
<6>[ 4108.890249]  alloc_contig_range_noprof+0x190/0x580
<6>[ 4108.895144]  __cma_alloc+0x140/0x640
<6>[ 4108.898794]  cma_alloc+0x28/0x40
<6>[ 4108.902084]  cma_alloc_aligned+0x48/0x78
<6>[ 4108.906111]  dma_alloc_contiguous+0x38/0x58
<6>[ 4108.910377]  __dma_direct_alloc_pages.constprop.0+0xb4/0x298
<6>[ 4108.916142]  dma_direct_alloc+0x74/0x3a0
<6>[ 4108.920126]  dma_alloc_attrs+0x94/0x240
<6>[ 4108.924039]  drm_gem_dma_create+0xb0/0x168
<6>[ 4108.928239]  drm_gem_dma_create_with_handle+0x30/0xd8
<6>[ 4108.933366]  drm_gem_dma_dumb_create+0x40/0x58
<6>[ 4108.937912]  drm_mode_create_dumb_ioctl+0x98/0xc0
<6>[ 4108.942696]  drm_ioctl_kernel+0xcc/0x148
<6>[ 4108.946684]  drm_ioctl+0x268/0x560
<6>[ 4108.950202]  __arm64_sys_ioctl+0xbc/0x100
<6>[ 4108.954276]  invoke_syscall+0x4c/0xf8
<6>[ 4108.958001]  el0_svc_common.constprop.0+0x48/0xf0
<6>[ 4108.962809]  do_el0_svc+0x24/0x38
<6>[ 4108.966186]  el0_svc+0x2c/0x80
<6>[ 4108.969292]  el0t_64_sync_handler+0xc8/0xe0
<6>[ 4108.973577]  el0t_64_sync+0x190/0x198
<0>[ 4108.977512] Kernel panic - not syncing: hung_task: blocked tasks
<4>[ 4108.983580] CPU: 2 UID: 0 PID: 45 Comm: khungtaskd Not tainted 6.12.0-1-librem5 #2
<4>[ 4108.991250] Hardware name: Purism Librem 5r4 (DT)
<4>[ 4108.996045] Call trace:
<4>[ 4108.998541]  dump_backtrace+0xa0/0x128
<4>[ 4109.002348]  show_stack+0x20/0x38
<4>[ 4109.005714]  dump_stack_lvl+0x38/0x90
<4>[ 4109.009467]  dump_stack+0x18/0x28
<4>[ 4109.012833]  panic+0x3b8/0x3d8
<4>[ 4109.015938]  watchdog+0x2c0/0x580
<4>[ 4109.019347]  kthread+0x11c/0x128
<4>[ 4109.022626]  ret_from_fork+0x10/0x20
<2>[ 4109.026252] SMP: stopping secondary CPUs
<0>[ 4109.030771] Kernel Offset: disabled
<0>[ 4109.034308] CPU features: 0x00,00000080,00200000,4200421b
<0>[ 4109.039755] Memory Limit: none

ECC: 36 Corrected bytes, 0 unrecoverable blocks

We configure the kernel to force a panic when there are hung tasks detected, as in my experience these usually signify a problem that’s unlikely to resolve on its own and leaves the device in a state where it’s unable to function properly. Looks like in your case SD I/O is so slow that combined with the aggressive caching behavior your seem to be seeing, perhaps due to all the layering on top of the block device, it causes some very long delays with tasks waiting for file cache pages to be migrated onto the card before being able to allocate themselves, so the hung task detector kicks in. You can disable panics on hung tasks with sysctl.

That said, you are clearly running an out-of-date system. Recent wlroots package wouldn’t make phoc reallocate its swapchains on display unblank anymore, so this particular case wouldn’t trigger a CMA allocation at all, which is what hung for two minutes there in this trace you pasted. You’re also running a 6.12 kernel while we’re already on 6.18, and I know that there’s been a lot of work on folio that recently happened upstream, so I wouldn’t be surprised if this was already improved/fixed. I believe some of the kernel config changes we did recently could also help there.

Yea, listen to dos and keep your phone updated.

This is a promise Purism has been keeping consistently. The phone is way better now than 3 years ago, when I started daily driving it. =)

The last time I updated was indeed several months ago :thinking:

The kernel is indeed old, but I don’t see updates for wlroots. The currently installed version is 0.16.2-2pureos4:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name               Version         Architecture Description
+++-==================-===============-============-===================================================
ii  libwlroots11:arm64 0.16.2-2pureos4 arm64        Modular wayland compositor library - shared library

I probably need to buy a faster microSD. But I have read on this forum that read speed of the card reader itself in Librem 5 is limited. :thinking:

By the way, the microSD is reported by udevadm as following:

udevadm info -a -n /dev/sda
  looking at parent device '/devices/platform/soc@0/38200000.usb/xhci-hcd.4.auto/usb1/1-1/1-1.1/1-1.1:1.0/host0/target0:0:0/0:0:0:0':
    KERNELS=="0:0:0:0"
    SUBSYSTEMS=="scsi"
    DRIVERS=="sd"
    ATTRS{blacklist}=="INQUIRY_36 IGN_MEDIA_CHANGE"
    ATTRS{cdl_enable}=="0"
    ATTRS{cdl_supported}=="0"
    ATTRS{delete}=="(not readable)"
    ATTRS{device_blocked}=="0"
    ATTRS{device_busy}=="0"
    ATTRS{eh_timeout}=="10"
    ATTRS{evt_capacity_change_reported}=="0"
    ATTRS{evt_inquiry_change_reported}=="0"
    ATTRS{evt_lun_change_reported}=="0"
    ATTRS{evt_media_change}=="0"
    ATTRS{evt_mode_parameter_change_reported}=="0"
    ATTRS{evt_soft_threshold_reached}=="0"
    ATTRS{inquiry}==""
    ATTRS{iocounterbits}=="32"
    ATTRS{iodone_cnt}=="0x54c47"
    ATTRS{ioerr_cnt}=="0x4"
    ATTRS{iorequest_cnt}=="0x54c47"
    ATTRS{iotmo_cnt}=="0x0"
    ATTRS{max_sectors}=="240"
    ATTRS{model}=="Ultra HS-SD/MMC "
    ATTRS{power/autosuspend_delay_ms}=="500"
    ATTRS{power/control}=="auto"
    ATTRS{power/runtime_active_time}=="9641290"
    ATTRS{power/runtime_status}=="suspended"
    ATTRS{power/runtime_suspended_time}=="15743937"
    ATTRS{queue_depth}=="1"
    ATTRS{queue_type}=="none"
    ATTRS{rescan}=="(not readable)"
    ATTRS{rev}=="2.09"
    ATTRS{scsi_level}=="0"
    ATTRS{state}=="running"
    ATTRS{timeout}=="30"
    ATTRS{type}=="0"
    ATTRS{vendor}=="Generic "

In future, please don’t report issues that happen on severely outdated software - or at least not without clearly mentioning it. It just wastes everyone’s time, regardless of whether the reported things have been already fixed or not.

This is pretty solid advice on any device regardless of operating system, nature of device, …

Maybe the use of LVM is a bit niche in this context though. How many Librem 5 users would be using LVM on the µSD card?

I haven’t personally experienced this problem but my bulk updates to the content on the µSD card tend to be via the local network (sshfs), so perhaps too slow to get enough backlog to see this problem or maybe just not enough total MB being moved at once. In my case neither LVM nor LUKS is in use (not bothering with LUKS because 99.9% of the content is public content anyway).

Is it possible to lengthen the timeout? (while still keeping the panic if the timeout is reached)

Updated everything. Hard resets are now much more rare. There are still UI freezes for 2-20 seconds when on high IO load on microSD.

This time it is Waydroid processes blocked, but it again happened while installing another Flatpak app.

dmesg-ramoops-0
<3>[61560.082639] INFO: task main:7011 blocked for more than 122 seconds.
<3>[61560.082746]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.082856] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.082919] task:main            state:D stack:0     pid:7011  tgid:7011  ppid:6886   task_flags:0x440100 flags:0x04000810
<6>[61560.083027] Call trace:
<6>[61560.083130]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.083201]  __schedule+0x374/0xbb8
<6>[61560.083259]  schedule+0x3c/0xe8
<6>[61560.083366]  io_schedule+0x44/0x68
<6>[61560.083430]  blk_mq_get_tag+0x180/0x330
<6>[61560.083496]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.083608]  blk_mq_submit_bio+0x414/0x710
<6>[61560.083666]  __submit_bio+0x8c/0x238
<6>[61560.083729]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.083834]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.083891]  submit_bio+0xb0/0x228
<6>[61560.083951]  ext4_mpage_readpages+0x184/0x760
<6>[61560.084061]  ext4_readahead+0x40/0x58
<6>[61560.084124]  read_pages+0x7c/0x2d8
<6>[61560.084188]  page_cache_ra_order+0x288/0x4b0
<6>[61560.084291]  filemap_fault+0x588/0xb50
<6>[61560.084350]  __do_fault+0x44/0x160
<6>[61560.084412]  __handle_mm_fault+0x840/0x13c8
<6>[61560.084525]  handle_mm_fault+0x194/0x2e0
<6>[61560.084587]  do_page_fault+0x100/0x6f0
<6>[61560.084652]  do_translation_fault+0x4c/0x70
<6>[61560.084763]  do_mem_abort+0x4c/0xa8
<6>[61560.086438]  el0_ia+0x50/0xd0
<6>[61560.086522]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.086628]  el0t_64_sync+0x198/0x1a0
<3>[61560.086765] INFO: task storaged:7075 blocked for more than 122 seconds.
<3>[61560.086865]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.086918] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.087014] task:storaged        state:D stack:0     pid:7075  tgid:7054  ppid:6886   task_flags:0x440040 flags:0x04000810
<6>[61560.087117] Call trace:
<6>[61560.087175]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.087235]  __schedule+0x374/0xbb8
<6>[61560.087337]  schedule+0x3c/0xe8
<6>[61560.087398]  io_schedule+0x44/0x68
<6>[61560.087454]  blk_mq_get_tag+0x180/0x330
<6>[61560.087516]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.087619]  blk_mq_submit_bio+0x414/0x710
<6>[61560.087675]  __submit_bio+0x8c/0x238
<6>[61560.087733]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.087791]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.087846]  submit_bio+0xb0/0x228
<6>[61560.087946]  ext4_mpage_readpages+0x184/0x760
<6>[61560.088005]  ext4_readahead+0x40/0x58
<6>[61560.088064]  read_pages+0x7c/0x2d8
<6>[61560.088228]  page_cache_ra_order+0x288/0x4b0
<6>[61560.088293]  filemap_fault+0x588/0xb50
<6>[61560.088354]  __do_fault+0x44/0x160
<6>[61560.088464]  __handle_mm_fault+0x840/0x13c8
<6>[61560.088527]  handle_mm_fault+0x194/0x2e0
<6>[61560.088590]  do_page_fault+0x100/0x6f0
<6>[61560.088701]  do_translation_fault+0x4c/0x70
<6>[61560.088763]  do_mem_abort+0x4c/0xa8
<6>[61560.088828]  el0_da+0x2c/0x88
<6>[61560.088938]  el0t_64_sync_handler+0xd0/0xe8
<6>[61560.088999]  el0t_64_sync+0x198/0x1a0
<3>[61560.089081] INFO: task system_server:7257 blocked for more than 122 seconds.
<3>[61560.089192]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.089246] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.089343] task:system_server   state:D stack:0     pid:7257  tgid:7257  ppid:7011   task_flags:0x440140 flags:0x04000810
<6>[61560.089450] Call trace:
<6>[61560.089504]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.089563]  __schedule+0x374/0xbb8
<6>[61560.089672]  schedule+0x3c/0xe8
<6>[61560.089738]  io_schedule+0x44/0x68
<6>[61560.089803]  blk_mq_get_tag+0x180/0x330
<6>[61560.089906]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.089967]  blk_mq_submit_bio+0x414/0x710
<6>[61560.090026]  __submit_bio+0x8c/0x238
<6>[61560.090129]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.090193]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.090302]  submit_bio+0xb0/0x228
<6>[61560.090364]  ext4_mpage_readpages+0x184/0x760
<6>[61560.090429]  ext4_readahead+0x40/0x58
<6>[61560.090538]  read_pages+0x7c/0x2d8
<6>[61560.090598]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.090655]  page_cache_ra_order+0x420/0x4b0
<6>[61560.090762]  filemap_fault+0x588/0xb50
<6>[61560.090826]  __do_fault+0x44/0x160
<6>[61560.090888]  __handle_mm_fault+0x840/0x13c8
<6>[61560.091000]  handle_mm_fault+0x194/0x2e0
<6>[61560.091063]  do_page_fault+0x100/0x6f0
<6>[61560.091128]  do_translation_fault+0x4c/0x70
<6>[61560.091228]  do_mem_abort+0x4c/0xa8
<6>[61560.091289]  el0_ia+0x50/0xd0
<6>[61560.091344]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.091448]  el0t_64_sync+0x198/0x1a0
<3>[61560.091564] INFO: task android.fg:7273 blocked for more than 122 seconds.
<3>[61560.091667]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.091722] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.091815] task:android.fg      state:D stack:0     pid:7273  tgid:7257  ppid:7011   task_flags:0x440140 flags:0x04000818
<6>[61560.091923] Call trace:
<6>[61560.091980]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.092043]  __schedule+0x374/0xbb8
<6>[61560.092101]  schedule+0x3c/0xe8
<6>[61560.092198]  io_schedule+0x44/0x68
<6>[61560.092258]  blk_mq_get_tag+0x180/0x330
<6>[61560.092316]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.092431]  blk_mq_submit_bio+0x414/0x710
<6>[61560.092490]  __submit_bio+0x8c/0x238
<6>[61560.092551]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.092652]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.092715]  submit_bio+0xb0/0x228
<6>[61560.092811]  ext4_mpage_readpages+0x184/0x760
<6>[61560.092871]  ext4_readahead+0x40/0x58
<6>[61560.092932]  read_pages+0x7c/0x2d8
<6>[61560.093032]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.093089]  page_cache_ra_order+0x420/0x4b0
<6>[61560.093150]  filemap_fault+0x588/0xb50
<6>[61560.093258]  __do_fault+0x44/0x160
<6>[61560.093316]  __handle_mm_fault+0x840/0x13c8
<6>[61560.093374]  handle_mm_fault+0x194/0x2e0
<6>[61560.093474]  do_page_fault+0x100/0x6f0
<6>[61560.093532]  do_translation_fault+0x4c/0x70
<6>[61560.093590]  do_mem_abort+0x4c/0xa8
<6>[61560.093696]  el0_ia+0x50/0xd0
<6>[61560.093757]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.093815]  el0t_64_sync+0x198/0x1a0
<3>[61560.093925] INFO: task ActivityManager:7283 blocked for more than 122 seconds.
<3>[61560.093985]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.094081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.094135] task:ActivityManager state:D stack:0     pid:7283  tgid:7257  ppid:7011   task_flags:0x440140 flags:0x04000810
<6>[61560.094247] Call trace:
<6>[61560.094350]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.094414]  __schedule+0x374/0xbb8
<6>[61560.094481]  schedule+0x3c/0xe8
<6>[61560.094547]  io_schedule+0x44/0x68
<6>[61560.094650]  blk_mq_get_tag+0x180/0x330
<6>[61560.094713]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.094772]  blk_mq_submit_bio+0x414/0x710
<6>[61560.095141]  __submit_bio+0x8c/0x238
<6>[61560.095214]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.095317]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.095380]  submit_bio+0xb0/0x228
<6>[61560.095441]  ext4_mpage_readpages+0x184/0x760
<6>[61560.095547]  ext4_readahead+0x40/0x58
<6>[61560.095605]  read_pages+0x7c/0x2d8
<6>[61560.095666]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.095766]  page_cache_ra_order+0x420/0x4b0
<6>[61560.095827]  filemap_fault+0x588/0xb50
<6>[61560.095888]  __do_fault+0x44/0x160
<6>[61560.095988]  __handle_mm_fault+0x840/0x13c8
<6>[61560.096050]  handle_mm_fault+0x194/0x2e0
<6>[61560.096107]  do_page_fault+0x100/0x6f0
<6>[61560.096215]  do_translation_fault+0x4c/0x70
<6>[61560.096278]  do_mem_abort+0x4c/0xa8
<6>[61560.096344]  el0_ia+0x50/0xd0
<6>[61560.096450]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.096513]  el0t_64_sync+0x198/0x1a0
<3>[61560.096575] INFO: task ActivityManager:7285 blocked for more than 122 seconds.
<3>[61560.096675]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.096730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.096824] task:ActivityManager state:D stack:0     pid:7285  tgid:7257  ppid:7011   task_flags:0x440040 flags:0x04000800
<6>[61560.096927] Call trace:
<6>[61560.096985]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.097088]  __schedule+0x374/0xbb8
<6>[61560.097150]  schedule+0x3c/0xe8
<6>[61560.097213]  io_schedule+0x44/0x68
<6>[61560.097270]  blk_mq_get_tag+0x180/0x330
<6>[61560.097377]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.097442]  blk_mq_submit_bio+0x414/0x710
<6>[61560.097550]  __submit_bio+0x8c/0x238
<6>[61560.097606]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.097672]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.097776]  submit_bio+0xb0/0x228
<6>[61560.097838]  ext4_mpage_readpages+0x184/0x760
<6>[61560.097907]  ext4_readahead+0x40/0x58
<6>[61560.098014]  read_pages+0x7c/0x2d8
<6>[61560.098072]  page_cache_ra_order+0x288/0x4b0
<6>[61560.098134]  filemap_fault+0x588/0xb50
<6>[61560.098260]  __do_fault+0x44/0x160
<6>[61560.098323]  __handle_mm_fault+0x840/0x13c8
<6>[61560.098381]  handle_mm_fault+0x194/0x2e0
<6>[61560.098483]  do_page_fault+0x100/0x6f0
<6>[61560.098547]  do_translation_fault+0x4c/0x70
<6>[61560.098606]  do_mem_abort+0x4c/0xa8
<6>[61560.098706]  el0_ia+0x50/0xd0
<6>[61560.098760]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.098822]  el0t_64_sync+0x198/0x1a0
<3>[61560.098931] INFO: task AlarmManager:7327 blocked for more than 122 seconds.
<3>[61560.098987]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.099084] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.099136] task:AlarmManager    state:D stack:0     pid:7327  tgid:7257  ppid:7011   task_flags:0x440040 flags:0x04000810
<6>[61560.099243] Call trace:
<6>[61560.099295]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.099354]  __schedule+0x374/0xbb8
<6>[61560.099415]  schedule+0x3c/0xe8
<6>[61560.099471]  io_schedule+0x44/0x68
<6>[61560.099567]  blk_mq_get_tag+0x180/0x330
<6>[61560.099630]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.099690]  blk_mq_submit_bio+0x414/0x710
<6>[61560.099787]  __submit_bio+0x8c/0x238
<6>[61560.099847]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.099943]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.100000]  submit_bio+0xb0/0x228
<6>[61560.100057]  ext4_mpage_readpages+0x184/0x760
<6>[61560.100162]  ext4_readahead+0x40/0x58
<6>[61560.100220]  read_pages+0x7c/0x2d8
<6>[61560.100278]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.100380]  page_cache_ra_order+0x420/0x4b0
<6>[61560.100437]  filemap_fault+0x588/0xb50
<6>[61560.100501]  __do_fault+0x44/0x160
<6>[61560.100598]  __handle_mm_fault+0x840/0x13c8
<6>[61560.100655]  handle_mm_fault+0x194/0x2e0
<6>[61560.100717]  do_page_fault+0x100/0x6f0
<6>[61560.100818]  do_translation_fault+0x4c/0x70
<6>[61560.100874]  do_mem_abort+0x4c/0xa8
<6>[61560.100931]  el0_ia+0x50/0xd0
<6>[61560.101033]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.101088]  el0t_64_sync+0x198/0x1a0
<3>[61560.101165] INFO: task binder:294_4:7449 blocked for more than 122 seconds.
<3>[61560.101265]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.101317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.101414] task:binder:294_4    state:D stack:0     pid:7449  tgid:7257  ppid:7011   task_flags:0x440140 flags:0x04000800
<6>[61560.101540] Call trace:
<6>[61560.101596]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.101701]  __schedule+0x374/0xbb8
<6>[61560.101763]  schedule+0x3c/0xe8
<6>[61560.101825]  io_schedule+0x44/0x68
<6>[61560.101882]  blk_mq_get_tag+0x180/0x330
<6>[61560.101983]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.102047]  blk_mq_submit_bio+0x414/0x710
<6>[61560.102103]  __submit_bio+0x8c/0x238
<6>[61560.102204]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.102265]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.102364]  submit_bio+0xb0/0x228
<6>[61560.102422]  ext4_mpage_readpages+0x184/0x760
<6>[61560.102482]  ext4_readahead+0x40/0x58
<6>[61560.102585]  read_pages+0x7c/0x2d8
<6>[61560.102644]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.102706]  page_cache_ra_order+0x420/0x4b0
<6>[61560.102810]  filemap_fault+0x588/0xb50
<6>[61560.102868]  __do_fault+0x44/0x160
<6>[61560.102925]  __handle_mm_fault+0x840/0x13c8
<6>[61560.103028]  handle_mm_fault+0x194/0x2e0
<6>[61560.103085]  do_page_fault+0x100/0x6f0
<6>[61560.103142]  do_translation_fault+0x4c/0x70
<6>[61560.103243]  do_mem_abort+0x4c/0xa8
<6>[61560.103304]  el0_da+0x2c/0x88
<6>[61560.103363]  el0t_64_sync_handler+0xd0/0xe8
<6>[61560.103466]  el0t_64_sync+0x198/0x1a0
<3>[61560.103540] INFO: task pool-8-thread-1:8103 blocked for more than 122 seconds.
<3>[61560.103644]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.103697] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.103794] task:pool-8-thread-1 state:D stack:0     pid:8103  tgid:7257  ppid:7011   task_flags:0x440040 flags:0x04000810
<6>[61560.103901] Call trace:
<6>[61560.103956]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.104018]  __schedule+0x374/0xbb8
<6>[61560.104120]  schedule+0x3c/0xe8
<6>[61560.104179]  io_schedule+0x44/0x68
<6>[61560.104239]  blk_mq_get_tag+0x180/0x330
<6>[61560.104301]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.104403]  blk_mq_submit_bio+0x414/0x710
<6>[61560.104459]  __submit_bio+0x8c/0x238
<6>[61560.104560]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.104615]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.104671]  submit_bio+0xb0/0x228
<6>[61560.104772]  ext4_mpage_readpages+0x184/0x760
<6>[61560.104936]  ext4_readahead+0x40/0x58
<6>[61560.105007]  read_pages+0x7c/0x2d8
<6>[61560.105106]  page_cache_ra_order+0x288/0x4b0
<6>[61560.105163]  filemap_fault+0x588/0xb50
<6>[61560.105227]  __do_fault+0x44/0x160
<6>[61560.105323]  __handle_mm_fault+0x840/0x13c8
<6>[61560.105385]  handle_mm_fault+0x194/0x2e0
<6>[61560.105442]  do_page_fault+0x100/0x6f0
<6>[61560.105539]  do_translation_fault+0x4c/0x70
<6>[61560.105600]  do_mem_abort+0x4c/0xa8
<6>[61560.105657]  el0_ia+0x50/0xd0
<6>[61560.105758]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.105815]  el0t_64_sync+0x198/0x1a0
<3>[61560.105885] INFO: task binder:294_1D:17987 blocked for more than 122 seconds.
<3>[61560.105980]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.106037] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.106130] task:binder:294_1D   state:D stack:0     pid:17987 tgid:7257  ppid:7011   task_flags:0x440040 flags:0x04000800
<6>[61560.106237] Call trace:
<6>[61560.106289]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.106388]  __schedule+0x374/0xbb8
<6>[61560.106450]  schedule+0x3c/0xe8
<6>[61560.106507]  io_schedule+0x44/0x68
<6>[61560.106563]  blk_mq_get_tag+0x180/0x330
<6>[61560.106666]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.106725]  blk_mq_submit_bio+0x414/0x710
<6>[61560.106780]  __submit_bio+0x8c/0x238
<6>[61560.106880]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.106936]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.107036]  submit_bio+0xb0/0x228
<6>[61560.107092]  ext4_mpage_readpages+0x184/0x760
<6>[61560.107151]  ext4_readahead+0x40/0x58
<6>[61560.107250]  read_pages+0x7c/0x2d8
<6>[61560.107307]  page_cache_ra_order+0x288/0x4b0
<6>[61560.107369]  filemap_fault+0x588/0xb50
<6>[61560.107469]  __do_fault+0x44/0x160
<6>[61560.107525]  __handle_mm_fault+0x840/0x13c8
<6>[61560.107587]  handle_mm_fault+0x194/0x2e0
<6>[61560.107689]  do_page_fault+0x100/0x6f0
<6>[61560.107744]  do_translation_fault+0x4c/0x70
<6>[61560.107802]  do_mem_abort+0x4c/0xa8
<6>[61560.107900]  el0_da+0x2c/0x88
<6>[61560.107963]  el0t_64_sync_handler+0xd0/0xe8
<6>[61560.108021]  el0t_64_sync+0x198/0x1a0
<6>[61560.108123] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
<3>[61560.108209] INFO: task BroadcastRunnin:7706 blocked for more than 122 seconds.
<3>[61560.108313]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.108367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.108463] task:BroadcastRunnin state:D stack:0     pid:7706  tgid:7542  ppid:7011   task_flags:0x440040 flags:0x04000810
<6>[61560.108570] Call trace:
<6>[61560.108625]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.108724]  __schedule+0x374/0xbb8
<6>[61560.108785]  schedule+0x3c/0xe8
<6>[61560.108841]  io_schedule+0x44/0x68
<6>[61560.108900]  blk_mq_get_tag+0x180/0x330
<6>[61560.108999]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.109060]  blk_mq_submit_bio+0x414/0x710
<6>[61560.109156]  __submit_bio+0x8c/0x238
<6>[61560.109214]  submit_bio_noacct_nocheck+0x254/0x328
<6>[61560.109274]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.109375]  submit_bio+0xb0/0x228
<6>[61560.109430]  ext4_mpage_readpages+0x184/0x760
<6>[61560.109490]  ext4_readahead+0x40/0x58
<6>[61560.109592]  read_pages+0x7c/0x2d8
<6>[61560.109649]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.109706]  page_cache_ra_order+0x420/0x4b0
<6>[61560.109807]  filemap_fault+0x588/0xb50
<6>[61560.109865]  __do_fault+0x44/0x160
<6>[61560.109923]  __handle_mm_fault+0x840/0x13c8
<6>[61560.110021]  handle_mm_fault+0x194/0x2e0
<6>[61560.110080]  do_page_fault+0x100/0x6f0
<6>[61560.110140]  do_translation_fault+0x4c/0x70
<6>[61560.110236]  do_mem_abort+0x4c/0xa8
<6>[61560.110297]  el0_ia+0x50/0xd0
<6>[61560.110354]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.110448]  el0t_64_sync+0x198/0x1a0
<6>[61560.110509] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
<3>[61560.110733] INFO: task droid.launcher3:21113 blocked for more than 122 seconds.
<3>[61560.110788]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.110885] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.110936] task:droid.launcher3 state:D stack:0     pid:21113 tgid:21113 ppid:7011   task_flags:0x440140 flags:0x04000800
<6>[61560.111047] Call trace:
<6>[61560.111143]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.111205]  __schedule+0x374/0xbb8
<6>[61560.111264]  schedule+0x3c/0xe8
<6>[61560.111364]  io_schedule+0x44/0x68
<6>[61560.111425]  blk_mq_get_tag+0x180/0x330
<6>[61560.111487]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.111615]  blk_mq_submit_bio+0x414/0x710
<6>[61560.111672]  __submit_bio+0x8c/0x238
<6>[61560.111731]  submit_bio_noacct_nocheck+0x278/0x328
<6>[61560.111828]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.111884]  submit_bio+0xb0/0x228
<6>[61560.111943]  ext4_mpage_readpages+0x184/0x760
<6>[61560.112047]  ext4_readahead+0x40/0x58
<6>[61560.112105]  read_pages+0x7c/0x2d8
<6>[61560.112163]  page_cache_ra_unbounded+0x1ec/0x308
<6>[61560.112261]  page_cache_ra_order+0x420/0x4b0
<6>[61560.112321]  filemap_fault+0x588/0xb50
<6>[61560.112379]  __do_fault+0x44/0x160
<6>[61560.112480]  __handle_mm_fault+0x840/0x13c8
<6>[61560.112542]  handle_mm_fault+0x194/0x2e0
<6>[61560.112599]  do_page_fault+0x100/0x6f0
<6>[61560.112700]  do_translation_fault+0x4c/0x70
<6>[61560.112756]  do_mem_abort+0x4c/0xa8
<6>[61560.112819]  el0_ia+0x50/0xd0
<6>[61560.112921]  el0t_64_sync_handler+0xe0/0xe8
<6>[61560.112977]  el0t_64_sync+0x198/0x1a0
<6>[61560.113039] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
<3>[61560.113201] INFO: task [edited]:21689 blocked for more than 122 seconds.
<3>[61560.113302]       Not tainted 6.18.0-1-librem5 #2
<3>[61560.113354] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
<6>[61560.113451] task:kgram.messenger state:D stack:0     pid:21689 tgid:21689 ppid:7011   task_flags:0x440140 flags:0x04000800
<6>[61560.113551] Call trace:
<6>[61560.113606]  __switch_to+0xf8/0x1a8 (T)
<6>[61560.113663]  __schedule+0x374/0xbb8
<6>[61560.113762]  schedule+0x3c/0xe8
<6>[61560.113819]  io_schedule+0x44/0x68
<6>[61560.114071]  blk_mq_get_tag+0x180/0x330
<6>[61560.114205]  __blk_mq_alloc_requests+0x1c8/0x3d0
<6>[61560.114272]  blk_mq_submit_bio+0x414/0x710
<6>[61560.114332]  __submit_bio+0x8c/0x238
<6>[61560.114440]  submit_bio_noacct_nocheck+0x278/0x328
<6>[61560.114500]  submit_bio_noacct+0xd4/0x4d8
<6>[61560.114558]  submit_bio+0xb0/0x228
<6>[61560.114664]  ext4_mpage_readpages+0x184/0x760
<6>[61560.114729]  ext4_readahead+0x40/0x58
<6>[61560.114787]  read_pages+0x7c/0x2d8
<6>[61560.114909]  page_cache_ra_order+0x288/0x4b0
<6>[61560.114969]  filemap_fault+0x588/0xb50
<6>[61560.115028]  __do_fault+0x44/0x160
<6>[61560.115133]  __handle_mm_fault+0x840/0x13c8
<6>[61560.115191]  handle_mm_fault+0x194/0x2e0
<6>[61560.115250]  do_page_fault+0x100/0x6f0
<6>[61560.115355]  do_translation_fault+0x4c/0x70
<6>[61560.115411]  do_mem_abort+0x4c/0xa8
<6>[61560.115469]  el0_da+0x2c/0x88
<6>[61560.115574]  el0t_64_sync_handler+0xd0/0xe8
<6>[61560.115630]  el0t_64_sync+0x198/0x1a0
<6>[61560.115700] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
<0>[61560.115871] Kernel panic - not syncing: hung_task: blocked tasks
<4>[61561.829215] CPU: 1 UID: 0 PID: 45 Comm: khungtaskd Not tainted 6.18.0-1-librem5 #2 PREEMPT 
<4>[61561.837577] Hardware name: Purism Librem 5r4 (DT)
<4>[61561.842285] Call trace:
<4>[61561.844736]  show_stack+0x20/0x38 (C)
<4>[61561.848419]  dump_stack_lvl+0x38/0x90
<4>[61561.852092]  dump_stack+0x18/0x28
<4>[61561.855412]  vpanic+0x2d8/0x300
<4>[61561.858559]  panic+0x68/0x70
<4>[61561.861444]  watchdog+0x290/0x718
<4>[61561.864762]  kthread+0x150/0x210
<4>[61561.867995]  ret_from_fork+0x10/0x20
<2>[61561.871576] SMP: stopping secondary CPUs
<0>[61561.876288] Kernel Offset: disabled
<0>[61561.879827] CPU features: 0x000000,00100000,40004001,0400421b
<0>[61561.885664] Memory Limit: none

ECC: 27 Corrected bytes, 0 unrecoverable blocks

And also, sorry for not updating from the very beginning.

Well, it depends. After being stuck without being able to use a mission-critical application because the new version not only requires an account but otherwise phones home aggressively (I had to restore the pre-login-required version), unless you have a test device to update first, I sometimes tell people to wait a couple of days.

I always check the forums first before updating my Librem 5 and wait for a few days anyway. It is my only phone, so I can’t afford it dying on me. Also, I can’t just go to any shop, get a phone on a whim, and restore my backups onto it. =)

Yes, there are currently very few large projects which make an effort to ensure that an update will not break something and will not inconvenience users in any other way. Debian and PureOS are miraculous in terms of update reliability.

I once was affected by broken sound in Waydroid after its Android image update, and was forced to cross my fingers and manually downgrade the image back. The new Android image started working properly only after upgrading to Crimson, when some Waydroid’s hardcoded values started to match the host OS once again.

It really depends on software developer’s track record whether you should install updates as soon as they are available, or update only when you have at leas a week of free time to deal with any unfortunate surprises.

Are you using the microSD only for storage and flatpak installs or are you booting from it? I boot Crimson from microSD, have many flatpaks installed and do not experience crashes. It does run much slower than booting from the eMMC, but no crashes to speak of.

Those were very annoying. I figured that if I would survive the jittery cracking noise for 30-40 seconds the sound would start working fine and then I could use Signal. (but now I use signal desktop)