Update: A possible workaround
add a .rules file under /etc/udev/rules.d, for me, I named it /etc/udev/rules.d/60-librem-key.rules
It should look like this:
# my librem key
SUBSYSTEM=="usb", ATTR{idVendor}=="316d", ATTR{idProduct}=="4c4b", ENV{ID_SMARTCARD_READER}="1", ENV{ID_SMARTCARD_READER_DRIVER}="gnupg"
It will work after reboot, or should work after remove and plug in the librem key (not tested).
like this:
yw662@yw662p:~$ gpg --card-status
Reader ...........: 316D:4C4B:...:0
Application ID ...: ...
Version ..........: 3.3
Manufacturer .....: ZeitControl
...