The following seems to work well (using some parts of the script from @dcz above) to cross-compile. Do the following on the build computer, in my case an x86 laptop:
First get the kernel source code:
git clone https://source.puri.sm/Librem5/linux-next.git
Then checkout the branch you want and/or make your own changes.
Install build dependencies by running this command inside the linux-next directory:
sudo apt-get build-dep .
Configure and build like this:
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- librem5_defconfig
make -j8 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bindeb-pkg
That should result in a .deb file created in the parent directory, one step outside the linux-next directory, the deb file is called something like linux-image-5.9.16-librem5-00002-gefe0e25dccf0_5.9.16-librem5-00002-gefe0e25dccf0-1_arm64.deb. Copy that .deb file to the Librem 5 using scp.
On the Librem 5, install it like this:
sudo apt-get reinstall -y --allow-downgrades ./linux-image-something.deb
where linux-image-something.deb is the deb file that was just copied over.
Then reboot the Librem 5 and it should be running your newly built kernel.
This procedure is much faster than running dpkg-buildpackage on the Librem 5 itself, both because the x86 laptop does the job faster and because doing make like this means that it only rebuilds the parts needed so after small changes building again it is very quick, in contrast to the dpkg-buildpackage approach which seems to always rebuild everything from scratch.
So I’m happy with this, now it just remains to learn more about how the kernel actually works. 