What you are seeing is representative of the current state of Librem 5’s audio hardware and handling.
I think this is quite common and presents an issue, if most people who try calling give up on it due to call quality then it’s unlikely to see much if any developer time and will remain in an unusable state for quite some time. It also means that if there are not many people using Calls there could be a number of other underlying usability and stability issues not being picked up on or addressed, it’s a bit of a catch 22.
There has been some recent updates to some open issues regarding external microphone/headset support so may be you’ll see some improvements in that area sooner rather than later. There was a comment regarding a recent merge that suggests selecting the external mic from the drop down will now enable it (albeit with a low volume and no auto sensing so the input won’t auto switch back when the headset is disconnected). I don’t think that merge has made it to a release as yet but may be available in a week or so.
On the positive side of things, with a little workaround you can use headsets for Calls, and the audio quality is decent (quality of headset/hardware dependent of course) so it seems that for audio call quality the current root cause is all down to the source.