I’m no expert but 
… I would expect more like 3.8V. That’s what it says on the back of the battery (right?) and I just measured my spare battery with a multimeter and that’s what the multimeter said (give or take a few hundredths of a Volt).
Maybe @dos can comment on that.
Review also: Bought broken batteries for Librem 5