I think the source code for the kernel used is supposed to be in this repo: Librem5 / linux · GitLab (note that there are different branches there, probably the “byzantium” and “crimson” branches are the most interesting ones.
See also this, a few years ago but I think much of it still applies: