Please
-
what happened?
-
how to be in touch in the meanwhile domain (website, forum,..) is off? Element/Matrix? I entered into that room but it seems death.. Mastodon? No posts since 542 days!!
Thank you
3 Likes
I noticed that too. It seemed to die about 12 hours ago but is back now (came back overnight for me and hence I don’t know exactly when it came back).
DNS lookups were giving SERVFAIL when addressed indirectly and timing out when addressed directly to any one of the three name servers.
This (puri.sm domain outage) has happened multiple times before. An earlier discussion gives some alternative domains but I didn’t test any of them during the outage.
I don’t know the answers to either of your questions.
3 Likes
If you want to stay in touch with me regardless of intermittent and unaddressed Purism infrastructure issues, you can use the VanLUG Forum:
2 Likes
- Purism received some abnormally-high cloud hosting charges recently, which their cloud hosting provider is actively investigating.
- Purism’s finance manager contacted the cloud hosting sales representative on Friday 10/10 and again on Monday 10/13.
- Purism’s finance manager requested a payment extension while the cloud hosting provider determines either an invoice forgiveness plan for the anomaly, or establish a payment plan.
- Purism was informed on Monday or Tuesday that they have a new account representative.
- Purism was guaranteed a payment extension on Monday by the old account representative. Meanwhile, Purism’s finance manager was focused on benefits- and tax-related hard deadlines due yesterday.
- Purism was aware of the outage approximately 0530PST.
- During the outage, Purism’s finance manager contacted the cloud hosting support manager to regain service via voicemail and email. They responded approximately 30 minutes later via email. It took a while longer for service to resume, but was much faster than other cloud hosting providers which have quoted next-business-day fixes.
- Following the outage when Purism’s finance manager reached out to the cloud hosting provider, the extended payment deadline was deferred for 10 days.
- The cloud hosting provider is setting up a meeting between their account representative and support representative next week.
- The finance manager is coordinating a follow-up meeting with the cloud hosting representative following their meeting next week.
- It took a while longer to get https://puri.sm operational due to some server-side services that failed to start following the unexpected power-off event. Purism is investigating ways to improve failsafe measures.
Purism forums and GitLab instances are hosted on a separate cloud hosting provider, so they were supposed to work in this case. However, due to nameserver glue records referencing an old IP address, DNS resolution was failing. This was fixed; updates take ~1 day to propogate. Now, the forums and GitLab should be accessible even if the cloud hosting provider (that went down today) goes down again. We are also considering other solutions like migrating communication infrastructure so that this does not occur again.
I apologize for the outage, but thank you for your patience and understanding.
10 Likes
To add to point 11, I noticed that RSS was still down even after the forum was back.
NB: RSS appears to be using puri.sm rather than a subdomain and the two domains (puri.sm and forums.puri.sm) appear to be on different servers (but I would imagine that the main Purism web site would be on the same server and quite possibly the same server process as the RSS service).
RSS has recovered its good humour now though.
3 Likes
Ugh. I can be blamed in part for the current site outage, but have been ensuring that Purism is not the blocking party at all stages. Here’s an update:
- 24 October – Purism receives a message from the hosting provider stating that service will be interrupted on 27 October due to an outstanding invoice. Purism emails the hosting provider, asking for confirmation that service will not be interrupted on 27 October until we can work toward resolving the (much higher than normal) cloud hosting charges, as mentioned previously. Promises to send a payment plan later that day.
- 24 October – The hosting provider account rep promises that service will not be shut down on 27 October, saying that the email was a “normal system notification.”
- 24 October – I devise a payment plan with the Purism finance manager and this proposal is sent to the hosting provider account rep.
- 27 October – Hosting provider account rep:
“I will submit your proposal, and our billing team will review it for approval.”
- 27 October – Purism finance manager to hosting provider account rep:
“Please keep JC and I posted with updates and/or questions.”
- 29 October – I send an email to our hosting provider account rep:
“I would just like to confirm with you that service will not be interrupted on 11/03 as indicated by the automated responses. Can you please confirm that service will not be interrupted at least until we come to an agreement regarding the payment plan?”
- Crickets ever since.
- 3 November – I planned to follow up this morning, but service was interrupted this morning before my day started.
I called our account rep directly and was greeted with voicemail. I left a voicemail. After I type this update, I’ll continue working with the hosting provider (whomever I can reach) to restore services. I apologize for this frustrating and seemingly completely preventable outage, everyone.
Update from the hosting provider:
My apologies for the service interruption. I have gone ahead and made the necessary adjustments. You should not be facing any interruptions now. I will also reach out to your account manager and they’ll get back to you. Please feel free to reach out if you experience any issues. We’re here to help.
6 Likes
About time to start self-hosting.
Only half-joking really.
2 Likes
I agree with a couple implementation-specific caveats. On-prem servers and failsafes have already been underway from 2 outages ago and their importance was stressed again during the last outage.
2 Likes
Good luck.
Maybe it’s time to consider a backup .tld or two in case this happens again?
Can we visit the forum and get the site to load using the IP address 128.140.118.223 when the DNS is down?
1 Like
Already considered:
Sure, for now.
It is readily the case that that will not work (rejected by the web server, depending on the config).
However the right approach in that case is to stuff that IP address into /etc/hosts and you could even do that now but commented out - because if the forum disappears due to a DNS problem, you are unlikely to remember the IP address.
3 Likes
That’s not a bad idea. You could also grab the historical DNS data from someplace like Security Trails
2 Likes
Thanks for those thread links.
I tried both forums.purismspc.com and forums.puri.st and only the latter one worked. Interestingly it resolves to a different IP
1 Like
The different IP address is likely attributed to the main domain:
I receive a 301 when attempting to access forums.puri.st, so it is not technically a redundant mirror.
Strange, because that hostname redirects to forums.puri.sm for me.
1 Like
A 301 is a redirect. And redirecting to a URL that contains a hostname does not provide any protection against domain fail on the domain that is the target of the redirection.
Redundancy (for domain fail) would be provided by configuring both domains to resolve to the same IP address and configuring the web server on that IP address simply to accept requests targeted at either domain and to respond identically regardless of which of the two domains the client uses. (That could end up being painful though if any absolute links ended up embedded in content in the forum where the link identifies content within the forum itself. That it, the forum software would need to replace those absolute links with relative links.)
That would not provide as much redundancy as a full mirror - but a full mirror would be harder work.
1 Like
It is basically a limitation of discourse. If discourse can support aliases, this would be easier.
2 Likes