curious is there server monitoring software that can be deployed without third party access- like machine AI type autonomous monitoring of traffic without any logging, that detects and alerts on these attacks in the early phases? that way mitigation could be more effective. Or is it just normal traffic after a minute non normal traffic so nothing that can be alerted on?
1 Like
I appreciate you and the rest of the Purism team having our best interest at heart.
It may be worth considering deploying a Tor hidden service to mirror the Purism websites.
5 Likes
Thanks for doing that, honestly.
I haven’t heard or seen many (any?) bad things about Cloudflare in the past. But knowing how much of the internet relies on Cloudflare has always made me skeptical of it. I think once a business/entity/system hits a certain critical mass I am hard-wired to start distrusting it.
Also, can I add how uncool it is for people to DDOS a small business? There are so many more deserving targets.
4 Likes
@francois-techene, I hope Purism reported the attack to the appropriate authorities, even if you can’t comment publicly on that at the moment.
2 Likes
You’re buying into the idea that an intermediate services provider is a man in the middle?
Maybe if I’d put /s at the end you’d have realized the description of a legitimate service provider as a man in the middle was sarcasm.
Isn’t think true of other service providers that you rely on, and even those you don’t rely on.
For example it is true that any CA (or rogue actor inside) could issue a certificate to allow decryption and re-encryption. - you “just trust” them.
You chose to use the cheapest SSL provider (zeroSSL) for protecting your site.
You have a hosting provider in Germany, that has access to your severs and the disks (or virtual disks) and data inside said servers - you “just trust” them.
You have factories in China assembling phones, you “just trust” them to use the right components, in the right places without substitution.
In all these cases you’re choosing to just trust companies that are based on Germany and China (and would be incredibly difficult to bring any enforcement action against) whilst at the same time saying you cannot trust a company (with a proven track record of good service longer than any of your other service providers) not just in the same country, but literally the same state/city, meaning it would be incredibly easy for you to seek legal recourse against the provider if that trust was breached.
You understand that this makes very little sense right?
I buy into the idea that trust should not be given to other entities unless it is necessary to function. Reducing our attack surface requires reducing and limiting trust as much as possible, regardless of threat models.
This is indeed true, and one of the reasons why I suggested for Purism to consider deploying a Tor hidden service to mirror their websites earlier in this thread. They could also just self-sign their certificate, but that usually gives off security warnings in browsers now.
Cheapest or not does not matter, considering a large amount of websites use Let’s Encrypt, which is free, enough that the EFF is already on its way to sunsetting HTTPS Everywhere due to how well Let’s Encrypt has been deployed across the Internet.
Purism uses Digital Ocean for their web hosting provider, which is based in the USA, at least for their main website.
Correct, which is why Purism also assemble their products themselves in the USA for their Librem 14, Liberty Phone, and Librem Key.
Ignoring Germany since it is not applicable, this situation with China is largely because Purism had to start somewhere over 4 years ago. Clearly they have been making enormous efforts recently to bring their manufacturing process into the USA. My speculation is that they intend to bootstrap to this instead, since it gives them a significantly more secure supply chain.
Trusting another entity only to deal with legal recourse against them later for your brand and clients could have been avoided in the first place if good judgement was exercised to begin with. I can at least appreciate that the Purism team has a firm stance on this matter, enough that they understand placing liability on another entity goes against their social purpose.
1 Like
Just wanted to clarify that unlike Liberty Phone and Librem Key, Librem 14 isn’t assembled in USA (aside of the final component assembly like RAM and battery, but that applies to all other products as well).
3 Likes
Thank you for correcting me.
I would myself prefer they used a better jurisdiction - like Switzerland or another country outside of any alliance with strong protecting laws. For many, U.S. is now considered unsafe and to be avoided for data storage and hosting. But at least, Purism can and does publish canaries at regular intervals - a privilege only a tiny number of companies can afford!
Just think of Cloudflare or M$ ever publishing a canary…
2 Likes
That depends on Purism’s threat model. They are at least aware of the advantages of using warrant canaries legally in the USA, which may not be available as an option elsewhere. That can be a crucial indicator in determining if their operations have been compromised and whether or not customers implement a plan for it ahead of time.
Rejoice not when thine enemy falleth, and let not thine heart be glad when he stumbleth:
1 Like
OK, and that’s great, but, they managed to get the main site working again pretty fast. - note that this thread is talking about the forum…
Non-authoritative answer:
Name: forums.puri.sm
Address: 128.140.118.223
inetnum: 128.140.0.0 - 128.140.127.255
netname: DE-HETZNER-20111010
country: DE
Which appears to be hosted by a German company.
You have completely misunderstood what I am saying,
When you reach the point where you realize that you cannot secure a service yourself there are only two questions that you need to ask.
Can I trust the party I contract with to provide a secure service?
How can I get recourse if that service breaks down?
You could simplify this to can I trust them? What if I’m wrong about that?
What I’m saying it Purism are poor at thinking about the “what next?” side of this, they didn’t think they’d be attacked, so had no plan, or skills. they don’t expect to be let down, so have left themselves with no recourse if they are let down.
I agree.
I agree with this too.
Purism have proven that they cannot deal with a small “attack” on their services, - they were down for days, and they lost data.
So they cannot function on their own, they do not have the capabilities. (much like they didn’t have the capabilities to start building phones in the USA)
I agree with what you are saying, there is a capability necessary to function they they don’t have, - they should buy that in.
They need to trust someone else because they cannot do it themselves.
On the contrary, this was the entire Purism infrastructure. The community forums were just one of many parts that were taken down during the attack.
I know about Hetzner very well and have personally used their services before for my own web projects, so at least for me, I already highly trust them. However, I appreciate you disclosing this information for others to consider in their threat model. Clearly there are multiple web hosting providers involved, not just Digital Ocean, but how many to be exact across the Purism infrastructure is not certain.
They definitely did not plan ahead for this DDoS attack, but they did swiftly responded to it all by themselves - the community forums were the last to be restored though. They now have experience they can carry over for next time, if such an occurrance happens again in the future.
Purism has proven that this was their first experience, that it took some time to deal with it in an effective manner, and that they handled the situation all by themselves. Some data was indeed lost from the community forums, but in the end, Purism is still here, and the DDoS attack failed to take them down.
The Purism team do have the skills necessary to handle a DDoS attack. They have no need to consider trusting someone else if this is the extent of available resources from a malicious actor or party, and that any further DDoS attacks do not continue growing in frequency and intensity.
1 Like
that’s funny, because, I checked if the forums and mainsite were down, consistently over a few days found the mainsite working ok, and forums not working,
even in this thread Purism employees have explained that they got the mainsite working…
Nobody said that only the forums were attacked, but many people including Purism have said that ways they attempted to handle the attack on the main site was effective, there but not in other places.
you are diverting the conversation and arguing against points that have not been raised.
you realize this is like me saying I know cloudflare very well and I trust them? - it means nothing.
Exactly… thankyou for supporting the point I raised.
when Purism say “we don’t want to make it so you have to trust other providers” that’s a bit of a non-argument since they already do force people to trust other providers…
And the outcome was multiple days of downtime, and apparently lost data…
It boggles my mind that you think this is a good outcome. - it’s not, it’s a fucking disaster.
We’ll have to agree to disagree on this.
But they could field a better (potentially faster/cheaper) solution to doing it inhouse badly.
And we know that even though Purism are able to do things themselves, they have chosen to trust outsourced providers.
for example despite literally making computers themselves, they are renting servers from an outside company for hosting. sometimes it’s not a question of if you can do it, sometimes it is a question of whether you can buy a service better than you can do it.
- and in the case of DDOS protection the answer seems obvious.
Those are pretty big “ifs” to stake the future of a business on.
That is not the exact quote I referenced earlier.
The key word is large. Cloudflare has a significantly large presence so they are in a very strong position to cause major influentual changes across the Internet, similar to Google.
I never explicitly stated whether or not this was a good outcome, but I will at least generously divulge that this outcome is better than I expected from a company with no prior history of handling DDoS attacks.
Right, but at the cost of trusting another entity. It is not easy to revoke trust once given, and the potential consequences outweigh the benefits, which is why the Purism team has already made their stance clear earlier in this thread.
It is not clear if they are renting servers. Hetzner does offer colocation in their datacentres, so you can bring your own hardware, slot it in a rack or two, and manage it both physically and remotely.
Regardless of whether they are renting Hetzner’s servers or not though, they have to trust Hetzner with managing the datacentre itself and all of its other clients within them, because owning datacentres are not economically feasible for most businesses and corporations.
Hardly, considering that there has been no further disruptions since the community forums came back online. About the only unusual traffic around these parts is our lively discourse, of which I am grateful for.
Then pick a small vendor of DDOS prevention, Talk to Hetzner see what they can provide.
I’m not saying go get cloud flare (that was 1 example of a company) I said go get someone to act as a shield.
Who said anything about Datacenters? I’ve seen companies host sites in their offices, it’s not hard to get dedicated high speed and divergent lines to an office.
But, I do take, and I do agree with your point, that some things are better left to professionals and specialists… - that’s exactly what I’m saying… I don’t think it is economically viable for Purism to suddenly get these skills, capabilities or equipment
There are mitigation techniques to prevent attacks as they describe them (random generated web requests) and blocking IP addresses (manually) as they appear in your logs is not what I would call a competent response!
Don’t misunderstand me, I understand why they did that, and why this worked for them in the situation they were in, but you can’t rely on ineffective attackers…
The above argument seems to be mainly about technical issues. We also need to look at the financial issue. Effective mitigation of a DDoS that is able to wield hundreds of thousands of source computers will cost dollars. You have to ask what is the cost in not mitigating the attack and what ongoing spend is justified?
In particular, there probably isn’t much direct cost in having the forum down (may even be a saving LOL) but there is a much more readily quantifiable direct cost in having the shop down.
There is reputational damage in having the forum down but the greater part of that, as per topic title, is “no comms” and you don’t need much technology to turn that around. It’s not a technology problem.
2 Likes
Speaking of comms, during the down time, email was working, at least. Support replied during the outage. I think the intent of “no comms” on the Subject line. refers back to here. Since most of don’t know each others private emails, this is what most of us use to talk to each other.
Although I agree with reputational damage. Marketing and PR has not been puri.sm’s strong suit.
3 Likes
I believe most businesses like Expedia that clearly depend on their website being up since they have nothing else have had to figure out how to mitigate those attacks, and have experienced them taking down, or slowing down their websites. Its a learning curve for every company.
It is a badge of honor, fought back attack - won, add a badge to the website - similar to how many airplanes you took down in the fighter jet, since it is a battle (just partially exaggerating).
2 Likes
No comms also means nothing outside of here, either. No announcement on Mastodon, Reddit (i know, but there is a page for Purism there), Lemmy, or any other social media. And if email was working, they are bound to have email addresses of customers that they could have sent an email blast and they in turn could spread the word.
1 Like