Thanks, figured it as well on my own that the USB-qubes needs more than the RAM initially allocated on install. Also upping VCPUs might help people and, if you are really into it, you can the weight of the USB qube via xl sched-credit command on Qubes. As I understand it, it means that the USB qubes is being favoured for CPU time.
See also: https://wiki.xenproject.org/wiki/Tuning_Xen_for_Performance
I’d also like to point to How to reduce annoying fan noise
where it is recommended to minimize the videochat-window as much as possible to reduce CPU usage.