As announced earlier in #community-librem-one-staging:talk.puri.sm matrix synapse upgrade is in progress. Since this needs a newer version of postgresql and the database is huge, the postgresql upgrade is taking some time to complete. Maintenance like this are posted on https://status.librem.one/ usually.
This update is now complete. matrix synapse was updated to 1.134.0 (from 1.118.0) and database server was updated from Debian Buster to Debian Bullseye (PostgreSQL 11 to PostgreSQL 13). Normal postgres upgrade (using dump/restore internally) was taking forever so I had to cancel and do an in place upgrade with -m link
option.
Awesome! That’s good news.
Unfortunately something broke during the upgrade and users are not able to login. So I’m going to restore the old version from backup.
I had tested the upgrade on two instances already (talk-staging.puri.sm and talk.puri.sm - both worked fine after the upgrade - though the database of chat.librem.one is much larger).
Restored matrix synapse 1.118.0 and postgres 11 on debian buster (same versions before the update).
I will now try to do just the postgresql/OS upgrade independently on 2nd August at 6am UTC.
PostgreSQL 11 to 13 update (Debian buster to bullseye) completed in 3 hours - most of the time was for taking snapshot of the vm.
Update: it ended up having the same issues after a while - clients remained offline. So PostgreSQL upgrade was reverted, but OS upgrade was kept. So running PostgreSQL 11 on Debian Bullseye now.