Exactly.
Last time I tried one of these with a fully free (libre) kernel, it worked as advertised, with no need for any additional drivers.
The CPU usage increased while using it, but not very much. This was on a machine much less powerful than the Librems. As long as the adapter was not being used with content that needed large areas of the display to be completely redrawn frequently (3D games or video editing, for example), I found the performance to be perfectly adequate.