OK then - so install this deb: https://source.puri.sm/angus.ainslie/uboot-imx/-/jobs/387599/artifacts/raw/debian/output/u-boot-librem5_2022.01-librem5.1_arm64.deb, then run:
sudo u-boot-install-librem5 /dev/mmcblk0
and then try booting without battery again (and make sure to try the attached charger as well, this one definitely works for me).