Will there be a blogpost, if the discussion with the FSF and the upstreaming of your patch are done?
Sure, absolutely, even before that, once we have something to report. ATM everything is pretty much “in the air” but as soon as we start have a plan or path we will for sure talk about it.
Any news on free bluetooth drivers for librem 13?
You announced almost a year ago that it was “in the air”. But since then, nothing …
In addition, I see a tremendous development of the product range at Purism, but think of your customers who bought the first versions and who are still waiting for a 100% release of their equipment (I am also thinking of bios) .
Thank you for your answers.
Are there any news on this topic, @nicole.faerber? Is there some public thread somewhere public to follow?
I try to be patient… but after two years now I have to ask: do you have any news on this topic?
Yeah, hmm… very unsatisfactory, totally get it. The only kind of solution I can offer is to install the binary firmware for the ar3k Bluetooth part, I’m sorry. The problem is with the FSF and the PureOS endorsement which we have not been able to straighten out. We must not include any blobs in PureOS and we must not endorse any sources that would carry such blobs.
While I understand the goal of it and also support that this puts us into a very VERY difficult situation - and this Bluetooth issue is just the tip of the iceberg! Pretty soon we will run out of WiFi chips that will work without binary blobs! And then?
I think this binary blob firmware issue needs to be thought over. It does not fit into the current hardware world anymore. Most hardware is full of proprietary firmware but this stays unnoticed because you do not see it - starting from the touchpad to the DP-HDMI transceiver to the battery charge controller, the SD card reader etc. etc. - there are at least a dozen firmwares hidden in the laptop that are not free. But these are assumed to be “OK” since they come with the hardware, in some form. But actually by hiding them from the user this also keeps them out of control - no way to see them, analyze them or update them. For some of these firmwares this can even impose a security risk as we have seen in the past (not with Purism hardware luckily) with WiFi exploits that were only possible to fix in firmware. So hiding the firmware in some ROM can actually be harmful.
So I am wondering how we can move this forward in a reasonable way?
Of course we do not want to give up our goal to free everything! As soon as and as good as we can.
But our corridor for hardware chips not requiring firmware blobs is getting more and more narrow and for some it is close to impossible to develop free firmware. Soon we will run out of ATH9k WiFi cards and there are no chips on the market than can be used without the blob download at runtime. None, seriously, I am looking literally for years now.
With the FSF’s super strict policy we are hurting ourselves and also users by cutting us off from a lot of hardware.
So the question is what can we do to move forward?
I do not at all want to preclude anything here, there is no decision made, the following is just a very personal train of thoughts (and especially NOT a Purism policy change, OK?).
I would be in favor of rethinking the binary blob firmware situation. What is the real goal? What do we want to achieve with a blob policy? What are certain risks we want to mitigate with it? What are longer term goals? Are there possible security threats? Etc. So first bring all that on the table and then discuss possible new solutions that can address these concerns and reach these goals.
My personal view on this is…
Soon there will be no other choice than to support binary blob firmwares - or we will loose a substantial part of our functionality (like WiFi). Binary blobs in themselves are not harmful - it is the same proprietary binary if you include it in some chip or on your harddisk, does not make a difference (concerning the content and its purpose). Having the blob on storage can have security implications. It could get replaced without the user’s notice. But that could get addressed by signatures etc. Blobs in general could include code that could access main memory - think of peripherals like an integrated graphics chip which has access to main memory. We may not want to allow that, so a new rule or barrier for firmware blobs could be where it is executed and which kind of access and control the main operating system can have over that. And finally of course if such blob would need to be executed by the main CPU that also runs the operating system and applications. Here I would say is a clear red line, this would not be acceptable. Having the main CPU take the binary and shove it into some chip, maybe OK, but not execute mystery code, no way.
Something along these lines. I could see something like Debian does, having a non-free repo that contains such blobs. Plus maybe implementing a system so that a user can sign these once they land on the system, with his/her own key and that only properly signed firmware can get loaded (I think we could implement that along with a necessary chain of trust using the LibremKey). And to only provide firmware of course that are vetted against rules like “not being executed on same CPU running user software” etc.
I would be very interested in more opinions and especially the reasoning behind these!
Like to share?
Maybe with an open discussion we can develop a new set of rules that properly address concerns while keeping access to current hardware.
Non-free firmware is a fact of life. It is just not possible to implement all functionality in hardware alone.
For me, the important thing is freedom. Freedom to be able to do with my hard- and software as I please.
If hard- and software is opensource, I do not have to fear vendor-lockin or vendor-control.
Hardware should be open so that the community can write drivers and software. If that requires non-free firmware (binary blobs) on the hardware to protect certain IP, so be it.
Thanks for sharing!
Right, I tend to agree and you bring up a good point I forgot to add to “my” set of rules:
The firmware blob must come with a perpetual license to copy, redistribute and bundle it. This is essential for the freedom you pointed out and we all want to maintain.
For me it is a fact that it needs to be possible to update firmware where the hardware vendor expects us to do so - even if an initial version of the firmware is in a rom inside the hardware component.
If security issues arise in a specific firmware we have to be able to follow up with patches/upgrades.
This statement is completely independent of the “open, free or no source” question.
If we can agree on the above statement and the point of view that security and thereby protection of user data is more important than ideals of the FSF we’d have a starting point.
Furthermore I’d say that there are enough arguments and reasons to always try everything to comply with the strict rules of the FSF as far as possible - even if it means some compromise regarding convenience and performance.
Purism could offer various levels of ‘free’ to let the user choose between convenience and willingness to support development by using less performant or convenient products that open the pass to more freedom and openness in the future.
Some projects to look at and some unsorted thoughts:
- MNT Reform - whole platform as open and free as possible - best documentation ever?
- development for BL602 Bluetooth/Wifi Chip
- development of a libre RISC-V + 3D GPU chip for mobile devices
- better documentation of every part that can’t yet be freed including potential risks and information about possible alternatives now or in the future
- stronger coorperation between forces working for free and open solutions
- IMX8 and LS1028A for notebooks (like MNT Reform, but there is still a market segment wide open for solutions with worse repairability and more convenient/stylish design)
- Official support of Debian on Librem notebooks as an option to FSF certificated PureOS
- How to build better support for libre developments? The evolving ecosystem around Librem5 and PinePhone might give us some ideas.
sorry for my late reply. I don’t know how to answer that… I’m just feeling sad. I knew that the marketing of Purism especially at the beginning was way too optimistic, sure. But still…
I know that there are a lot of binary blobs in chips, but I thought ‘no binary blobs as part of the OS’ is only the first step. I thought it might be possible to free everything – so let’s start with the OS.
I’m really getting old. This is what I feel when I read your posting. You’ve got much more insight and I have no reason to doubt your assessment.
I don’t know… I can still hope for big surprises. Maybe that is what a lot of old people do.
If enough people are interested in hardware with open-sourced firmware, will a supplier eventually be incentivized to provide such hardware?
The FSF policy on non-free firmware seems very appropriate for user freedom, and I don’t interpret the role of FSF as “realistic.” Instead, FSF has always striven to be idealistic instead of realistic. This has been the unique and vital role of FSF and RMS for a long time, to the point of many questioning the relevance of the organization and its founder because they are not realistic enough.
I don’t like the approach toward the FSF policy of, “this might be impossible right now or in the near future, so we should change the policy.”
No, FSF puts forward ideals, not “goals that are achievable with the current offerings of suppliers.” Purism has made, and continues to make, more progress toward such ideals than any other company that I know about—that progress even seems closely linked with Purism’s social purpose. Don’t give up! If not Purism, what company is better situated to pursue these ideals?
I am not trying to be critical @nicole.faerber or Purism! You have done far more for free technology than I could dream of. And you have far more knowledge about the impossibility of the situation. Even if I am naive, I only want to offer encouragement to not give up on a good ideal that will make society better and people more free
thank you very much for sharing your views!
And yes, I totally share the sentiment that the FSF is absolutely needed as this idealistic party that represents a strong force everything can gravitate towards! Totally get it and agree. Actually this is also the very exact same point how I found myself defending the FSF and sometimes even RMS in some discussions - it is this uncompromising idealistic force that is needed to be present in the world.
But the reality also is that it makes it close to impossible to make products based only on these idealistic rules. There must be ways to grant exceptions somehow. These may come with penalties or such, totally fine. But they need to be there or else we will shoot ourselves in the knee with a tremendously large gun! We will cut ourselves off from current hardware.
Matter of fact is that more and more silicon’s function these days is determined by software not by silicon hardware design. The software defining it is the firmware. There are some that can be rebuilt in the open, like how we did it with the smartcard reader firmware in the Librem5. In the dev kit we used a proprietary reader chip with its proprietary firmware (though this firmware was in ROM inside the chip), in the final L5 we are using a programmable microcontroller and our free firmware.
But there are also tons of chips where this is either a lot harder or even close to impossible, for several reasons.
Let’s take Bluetooth as an example, since it is also this thread’s topic. As an early Bluetooth adopter I got in touch with pretty much the very first generation of Bluetooth hardware, at that time made by CSR - Cambridge Silicon Radio. These modules were self contained, they included the MCU driving the logic, the DSP for the radio and, last but not least, storage for the firmware. From the outside you had the Bluetooth defined host controller interface (HCI), USB and serial UART at that time. Firmware update on these was a highly proprietary thing, required special tools etc. But you did not really need it, the modules worked in HCI mode and you could develop your stack against it. But also Bluetooth became more and more complex, firmware become more complex and bugs were found, so firmware updates became necessary. Also regulations became fragmented, the 2.4GHz ISM can not be used in all regions in the same way so you need to accommodate for that in your firmware (the Bluetooth stack can not do this). Having all that fixed in silicon quickly becomes a problem, you end up with tons of SKUs of one product. No one wants this. Over the years also flash memory became much more expensive than RAM - expensive in terms of chip size. Mask programmed ROM was not an option, you need to be able to fix bugs, eventually even in the field.
So the quite natural choice for many chip makers was to drop all or most of the built-in-silicon software, add more RAM instead (saves money) and upload what is needed in its most current version right at the start of the system. This solved a lot of the pain points - you can update them easily, just replace the uploaded bits, you can configure them easily in the same way and you save a lot of cost by reducing the silicon die size and the number of SKUs. A strong incentive for the chip makers.
The same is happening in pretty every peripheral chip these days. Only very small and very well defined ones still have their firmware on chip (either masked ROM or flash) like a touchpad controller. But more and more only have a tiny bootloader in ROM that will wait for a firmware and eventually parameter data set to be handed over through the HCI, put in RAM and then being executed.
So this is what’s happening on the hardware side - on the chip level.
The internals of these specialized chips, like Bluetooth or WiFi or even these touchpad controllers, are pretty much proprietary, you do not know how they are made, what kind of CPU core they have, how they work and how they attach to the outside world. Most of them are actually protected against third party hacking by “secure bootloaders”, i.e. they will only accept signed firmware images etc. Which also kind of makes sense when thinking about radios and regulations. But this of course makes it awfully hard to hack such a chip and to put something free in there.
I don’t want to say it is impossible, but it is very very hard and takes a LOT of effort, person years of development time. If you think about Bluetooth or WiFi the radio part alone is already really really complex - modulations, decoding, traffic handling etc.
And now assume for a minute we, i.e. Purism, would heavily invest in such an effort. What would we do? We for sure are not in a position to design our own radio silicon. So we have to take something, reverse engineer it (since the original maker will for sure not give us the internal information), write that firmware with all its bits and pieces (radio handling, protocol, host interface, drivers etc.) and then what?
As soon as we publish and start to sell such a product we are basically legal bait for a hole bunch of lawsuits. First of all for reverse engineering which is not legal in all regions. Second for publishing proprietary information. Third for infringing on a metric ton of patents (by the chip maker and very likely also from hundreds of other stake holders of that technology). And fourth we would very likely by forced to use the same secure bootloader stuff to lock down the firmware loaded into the chip as the original maker due to regulation compliance, for which we, as a company, will be held liable and accountable. We must take precautions that the hardware we sell can not be used in non compliant ways (at least for radios).
This is at least my line of thinking based on what I know right now. The big problem here is that Purism is a company selling products. We are not a free software project. As a company we have additional regulations and rules to follow than a free software project.
So as long as no chip maker makes new chips with fully embedded firmware, I am so sorry, I am afraid we are out of luck. We, i.e. Purism, will not be able to fill this gap with a product. We can not, no matter how much we wished to. It is way beyond our scope - and honestly also beyond our current abilities. I am not saying it’s impossible, but you would need a huge seven figure budget just to create one of these free radio chips. And even then once you are done with that you can start all over again because customers will expect the next generation. We simply do not (yet) sell enough devices to be able to pay for that. Not even close.
So what’s left?
The only way we can continue to advance all our causes and make products is to use chips we can buy on the open market. And here, as I explained above, the definitive trend in the chip industry is not to embed firmware in these chips. Even worse all the chips I looked at, and these were a lot over the years, do no allow to load that firmware through any other channel than the host interface. There used to be some that could load it from an attached serial flash chip with which we could have built our own cards, we would have done that! But these do not exist anymore.
So here we are. There are chips for which there are also free kernel drivers but which need a firmware download through the host interface at startup. And there are no alternatives to these chips. None.
So what shall we do?
Stop selling products?
Cobble together as many (really) old parts as we can find and build outdated hardware? If you want to get an idea what this “outdated” looks like in terms of radios, look at the FSF RYF hardware page for the WiFi radios. Pretty much no 802.11ac radio there:
The latest is the Atheros9k we are also using in our products, which is quite OK but 802.11abgn only, no 802.11ac. There are no 802.11ac chip without firmware requirement - at least as far as I know. And the Ath9k is being phased out, we already have a lot of difficulties sourcing the QCNFA222 M.2 cards. And this is just the WiFi radio.
We must find a way that
a) does not totally give up the ideals that we all want to reach some glory day and
b) gives us enough breathing room to create actual products today.
And let’s be totally clear about this too, if we do not sell quite a number of products we can not make enough money to advance anything anymore. Compared to a lot of other consumer hardware product makers we already squeeze a lot out of our margin into R&D. And even that is not even close to enough to create what would be necessary to be fully free.
The silicon industry is clearly against us. Not because they don’t like freedom as much as we do but simply because of driving their cost down and because of regulations.
Can we incentivize a maker to make something that is free enough for us? And I am not even talking about us as in Purism but the us as in “freedom lovers”. No, we can’t. Was there, tried that and failed. I talked to many of them and our market and audience is just a very tiny blip on their radar. We do not matter. A company like Qualcomm now owning Atheros owns thousands of patents on all of that and proud themselves for it. They have no intention and also no reason to make any of this “free to use”. The chips they make are made and sold in the millions. Let’s be generous and say we are a strong community of some ten thousand, oh no, be more generous and say hundreds of thousands. And let’s also very unrealistically assume all of them would commit to buy the very same chip. Even then, maybe, but only maybe, you would probably get invited to a meeting.
So not, sorry, not going to happen in the chip industry.
I don’t know how to solve this. And to be very honest with you, I am currently pretty frightened about this future to come. If we stick with the hard “no non-free firmware blobs on OS storage” requirement then we will soon run into a serious problem for which I have no solution to offer. None. The first to fall is WiFi/BT. Next could be the GPU (a reason why we can not use Ryzen and who knows how long Intel GPUs will remain blob free).
I am totally open for suggestions!
But I think we will need some form of blob enclave in userspace. I am just not sure how to make it. Right now we can not create it in any way within PureOS since this would violate the endorsement. We also do not want users to have to enter things manually to enable such a source. And if we create something like it, it needs to be compatible with other OSes too, like Qubes. A conundrum.
I used to be very excited about RYF certification, but after looking into the details, I realized that there are many problems with it. RYF does call attention to the problem and makes people think about the blobs in their devices, but RYF certification actually discourages hardware makers from producing new RYF devices and discourages the community from working on freeing hardware.
There are certain components like CPU, GPU, WiFi, Bluetooth and cellular modem, where it simply isnt safe to have firmware which isn’t upgradable, and the FSF is being ridiculous by using ambiguous language in its RYF criteria and not answering my pointed questions as to whether proprietary firmware updates are allowed or not. I repeatedly asked the FSF to clarify its position on this point and I never got a response. The FSF can’t expect any hardware company to design and produce new RYF hardware if it probits firmware updates, because that is a security risk; but at the very least the FSF needs to clarify its position as to whether RYF certification allows updates to proprietary firmware or not. If updates are allowed, under what conditions are they allowed?
Right. And the FSF operating system “endorsement” is even worse since you must not in form or fashion even mention non-free things in it. So the OS can neither include nor point to such upgrades and it can only in a very limited fashion include the tools to do the upgrade (maybe include the tool but must not point to a source where to get the actual non-free upgrade).
Issues with e.g. WPA vulnerabilities were already public and many of these could only be fixed by a firmware upgrade, so the non-upgrade indeed left users in a vulnerable situation. So we are not talking about theoretical possibilities, this has already happened and endangered users.
I totally get that this is a problematic fine line and one has to be careful not to open the floodgates to binary-only hell. But we have to come up with something.
@nicole.faerber I have some questions, hopefully not too stupid ones
What do you think about the BL602 @ChriChri mentioned?
Apart from it “only” being b/g/n, has it any problems or drawbacks?
From this comment I gather that it is specifically low-power oriented, which makes sense as it is a new chip, but skipping the 5 Ghz band.
And while WiFi 5 or 6 might be nice to have, honestly, I couldn’t care less about more (theoretical) bandwidth. I care about stable connectivity, preferably even two rooms away.
For huge transfers I’d probably use a wired connection anyway.
What about USB-WiFi-BT-dongles?
No, hear me out I gather we are looking for mid-term viable solutions, something between “give up” and “spend millions on our own silicon”.
So, a quick search turns up a WiFi-5+BT-Thumbstick, tiny, cheap …
So, there should be a pinky-nail sized PCB with USB connectivity in this thing. Yes?
How could we use that (best first)?
- source the PCB, attach own antennae and USB (laptop + phone!)
- just… put the stick inside the laptop (does that work for the radio?)
- have something like a “bay” for dongles on the laptop, so they don’t stick out
- just plug it in, sticking out
At least for that I can say this is based on RTL8821CU and this again needs a runtime firmware, pretty sure. I have sourced and tested a bunch of them and did grind through the kernel drivers looking for the firmware load - and everyone I looked at did load a firmware at startup
The GIT driver you link to for example does it too, if you look into: hal/rtl8821c/rtl8821c_mac.c
Around line 112 is the firmware load (at least one of the spots). I would need to analyze the code deeper to figure out where the firmware actually comes from, i.e. if they load it from the filesystem via the Linux kernel firmware subsystem or if they buried it in hex in some header file (which IMHO would not be acceptable and would also never be accepted upstream (anymore)).
Of course I can not tell if there is really absolutely no chip that does it. But over the years I looked at a lot and so far all tries came up empty.
As for the BL602 this is for sure an interesting project, but not usable for us I’m afraid for reasons I also tried to explain before. Even if we would use it with a freed firmware, once this exists some fine day, we get into regulatory trouble. We could of course try to make use of it as-is (as we could with regular ESP chips as well) and develop a firmware that gets flashed into it using the proprietary bits, but these would raise an eyebrow or two, especially in our PC based products where, and that’s a fact, customers expect at least almost up to date performance - and quite some customers already complain at us about the poor performance of the Atheros. This will then be even a lot worse.
Something like the BL602 or the ESP chips for that matter can be acceptable for something like a phone where huge bandwidth, at least to me, is rather secondary, but where power saving is more important. But in a PC? Meh, no, don’t think this would not end well. And it would still be a significant effort needed to go into such a project since these chips usually can not be used as Linux WFi/BT cards but usually run the whole stack on their own. There once was a SDIO station firmware and even a Linux driver for the ESP8266 but that is now unmaintained and wasn’t really stable, as far as I know.
I work in R&D for a large silicon chip manufacturer. I’ve been there long enough that I can pretty much talk to almost anyone either at the lunch tables in the company cafeteria, or through inquiring e-mails that go outside of my usual area of expertise, to people I know in other areas of the company. Nicole’s assessment of the situation is quite accurate.
A high level marketing person who I get along well with and who I see at the lunch table most days (pre covid), even said that he could arrange a meeting with the company president if we could put together a compelling presentation for my fully-free cell phone SOC idea, while also showing some skepticism by saying that the cell phone chip market is extremely competitive. But he started an email thread that went quite high up the chain in the marketing department anyway, to see what the feedback from others would be. Nothing we could discuss or come up with was compelling from the chip manufacturer perspective. Unless tens or hundreds of millions of dollars in profits are obviously on the table, the free (as in open) cell phone niche market is far too small now. I spoke to and exchanged emails with several people individually also. The consensus is that the Chinese companies have driven the prices down so far that there aren’t any profits to go after in the cell phone market, especially when needing to make large investments. One other marketing person I spoke to used the word “bloody” (a term that probably has a different meaning in the US than it does in the UK. In the US in this context, it’s more like likely to cause financial bleeding and financial injury) to describe the cell phone market when it comes to silicon chips. Also, any investment dollars are competing with other big ROI projects in the company.
I think a lot of them humored me by engaging the topic after they realized that we were talking about a cell phone SOC. The response was generally that their knowing the market, no executive would approve anything to do with cell phone chips. After seeing responses from Directors and Marketing executives and others, I tend to agree with their logic as it relates to the best interests of the company and the careers of those involved.
Also, most of those who I communicated with about this were totally on-board with the personal and societal benefits of free (as in opensource - Todd’s vision) hardware. They really get it. In fact, I suspect that’s why they entertained the idea while knowing as professionals that there was no money in it before the conversation even got very far. But then it was back to reality and the bottom line. We’re all there to make profits for the company.
But if someone thinks there that a closed-source proprietary firmware upgrade is needed to secure their system, then that person has adopted a world view that says their security relies on proprietary software. Is that a reasonable position?
Here is how I look at my system communicating with the world, where the dashed box is my computer (like my Librem 5 or my laptop):
Now, suppose my use case is that I communicate by email and I use GPG to encrypt and sign emails as described in the Email Self-Defense guide – then, I can achieve secure communication despite the fact that the Wi-Fi chip cannot be trusted. I use it, I communicate through it, just like my communication passes throuugh lots of routers and other networking devices before reaching the recipient who will decrypt my messages. Any actors that look at my messages in transit, including the Wi-Fi chip, will not be able to decrypt or distort my messages because they do not have my private key.
If someone says “there is a vulnerability in your Wi-Fi chip, you should install this proprietary firmware upgrade” then my answer would be that the Wi-Fi chip is anyway fundamentally insecure, with or without that upgrade it can still not be trusted. So is that upgrade really critical for me?
I guess my point is that instead of accepting the common view that proprietary firmware upgrades are strictly necessary for security, we can focus on using free software tools in ways that allow us to stay secure without being dependent on proprietary software for security (remembering that the proprietary software may still be spying on us no matter how many proprietary firmware upgrades we install).
Oh, I was in no way suggesting that this would be “strictly necessary” or the solution to security and privacy. But I think we can agree that being able to additionally apply these fixes to known problems would be beneficial? These kind of fixes not necessarily always have to do with security sometime it is also about usability and stability. To stay with WiFi there are known vulnerabilities in some implementations that also allow third parties to either crash the firmware from remote, even taking control over it. Or take the deauther as another example. These problems do not necessarily have to do with security (privacy) of the data sent over the link but also with overall system safety and access to a service. Some other of these things could introduce new features to existing hardware too. Or take the Intel microcode as another example. Just recently we have seen a number of microcode updates, which are of course blobs, which fix CPU vulnerabilities which could even compromise what you point out as your “FOSS core system”. Right now we go around this problem by providing these microcode fixes as part of Coreboot but not all users update their BIOS on a regular bases. But the OS can load microcode updates at startup and all bits and pieces for that are there, just that we can not use them with current PureOS and the FSF endorsement.
Like I said before, I am not at all advocating opening the floodgates for blobs. But I still think that a more flexible and accessible approach than we have now would be beneficial while still keeping our goal of striving for as much free software as we possibly can. I think that in some areas we simply have to accept the fact that it is close to impossible to free them - like the microcode. So instead of limping along with some crutch like hiding blobs somewhere (like microcode in Coreboot) better pull them out in the open so that users get better control over them? Then adding firmware signing (by the user) also gives some piece of mind that these cannot be replaced without the user knowing.
I’m still curious about the internal or semi-interal USB path. That wouldn’t cause trouble for the RYF certification, as I understand it, and would enable using the latest and greatest wireless networking technology.
My non-expert way of doing this would probably involve a cable connected to the existing USB interface making a sharp 180* turn within the space of the connector (so it does not stick out), and then going to a USB wifi/bt board stuck (literally probably glued) somewhere within the laptop case. We’d have to find a board with appropriate connectors for the antennas already in the case in a retrofit scenario, or source adapters.