Adduid / change email address

I’d like to change the email address on my gpg key pair of which the secret key sits on my LibremKey.

I read that the usual procedure would be to use gpg --key-edit current@email.ad, add a new email address using adduid, making that the primary and deleting the old email address using deluid - assuming that the key has never been published to a key server.

Works fine on a test key I generated. But when I try to do this on my actual key (private key on the LibremKey) I fail in the first step:

gpg> adduid
Need the secret key to do this.

user@system:~$ gpg --list-secret-keys
/home/cvogel/.gnupg/pubring.kbx
-------------------------------
sec#  rsa4096 2019-07-15 [SC]

The man page of gpg says about the ‘#’ sign:

A # after the initial tags sec or ssb means that the secret key or subkey is currently not usable. We also say that this key has been taken offline (for example, a primary key can be taken offline by exporting the key using the command --export-secret-subkeys). A > after these tags indicate that the key is stored on a smartcard. See also --list-keys.

The ‘>’ sign is missing, probably because I generated my key on a different (offline) computer and didn’t copy the keyring over to my notebook, but imported my public key?

How do I proceed here? It seems (and maybe this is wanted this way) that it is not possible to add or delete uids when the secret key is stored on the LibremKey.