There’s https://usbguard.github.io, but admittedly it needs a lot of love to become stable and user-friendly.
With such a mechanism in place, a lot of attacks can be mitigated.
In general, Linux used to trust devices. Perhaps it’s time for a change - and refuse devices by default, unless explicitly configured to do otherwise.
In the L5 case I can see the beginning of this approach - no closed source on the main CPU and memory, any firmware blobs run on devices themselves and are behind well defined interface, that provides them only those bits that we consider necessary.
Than again, since USB is a shared bus, a malicious device can go sniffing on the bus and get read-only access to the data. This is quite a different can of worms, and USB guard cannot protect against such a sniffer.