As announced earlier in #community-librem-one-staging:talk.puri.sm matrix synapse upgrade is in progress. Since this needs a newer version of postgresql and the database is huge, the postgresql upgrade is taking some time to complete. Maintenance like this are posted on https://status.librem.one/ usually.
5 Likes
This update is now complete. matrix synapse was updated to 1.134.0 (from 1.118.0) and database server was updated from Debian Buster to Debian Bullseye (PostgreSQL 11 to PostgreSQL 13). Normal postgres upgrade (using dump/restore internally) was taking forever so I had to cancel and do an in place upgrade with -m link option.
2 Likes
Awesome! That’s good news.
2 Likes
Unfortunately something broke during the upgrade and users are not able to login. So I’m going to restore the old version from backup.
I had tested the upgrade on two instances already (talk-staging.puri.sm and talk.puri.sm - both worked fine after the upgrade - though the database of chat.librem.one is much larger).
2 Likes
Restored matrix synapse 1.118.0 and postgres 11 on debian buster (same versions before the update).
2 Likes
I will now try to do just the postgresql/OS upgrade independently on 2nd August at 6am UTC.
2 Likes
PostgreSQL 11 to 13 update (Debian buster to bullseye) completed in 3 hours - most of the time was for taking snapshot of the vm.
Update: it ended up having the same issues after a while - clients remained offline. So PostgreSQL upgrade was reverted, but OS upgrade was kept. So running PostgreSQL 11 on Debian Bullseye now.
2 Likes
I have cloned the instance was able to reproduce the issue on the clone. This turned out to be a behavior change in how postgresql 13 handles statement timeouts[1]. The fix was to set 2m for statement timeout in postgresql configuration. I will do the production update on 5th Aug at 6am UTC.
[1] PostgreSQL: Documentation: 17: 19.11. Client Connection Defaults
2 Likes
Unfortunately, the fix that worked on the clone seems to be not enough on production. It is better than before - login works on element, but sync is still broken. I will try to troubleshoot a bit more, but plan to revert back to old version if I can’t figure this out today.
2 Likes
Sync as of right now is rather slow still. And seemingly unable to connect. There was an error regarding unable to connect to an IP (I think 169.x.x.x. But I only saw it for a second so I am unsure.