Handling glibc v2.28 breaking changes

2022-04-24 Thread Pradeep Chhetri
Hello everyone,

I am sure this has been discussed multiple times in the past but I would
like to initiate this discussion again. I have 3 nodes cluster of Postgres
v9.6. They all are currently running on Debian 9 (with glibc v2.24)
and need to upgrade them to Debian 10 (with glibc v2.28) without downtime.
In order to bypass the glibc issue, I am trying to evaluate whether I can
compile glibc v2.24 on Debian 10, pin postgres to use this manually
compiled glibc and upgrade the linux distribution in rolling fashion. I
would like to know how others have achieved such distro upgrades without
downtime. I am new to Postgres so please pardon my ignorance.

Thank you for your help.
Best regards,
Pradeep


Re: Handling glibc v2.28 breaking changes

2022-04-24 Thread Pradeep Chhetri
Hi Adrian,

Thank you for your quick response.

By zero downtime, I meant at least one of the three nodes is up at any time
to handle the writes and reads.

> Define how the 3 node cluster works?
These 3 nodes are configured as 1 primary, 1 sync replica and 1 async
replica. These are managed via stolon.

> What is the locale for the Postgres instances?
We are using en_US.UTF-8 collation.

> What is acceptable downtime in the process?
We want to minimize as little as possible since these will be customer
facing clusters.

> Are you using ICU collation?
As far as I know, ICU collation is supported from Postgres v10 but we are
still running v9.6 so I guess that is not an option unless we upgrade our
cluster first.

I am open to ways including changing architecture or upgrading cluster
first or evaluating logical replication or any other option but our primary
goal is to achieve it with minimal downtime.

Thank you for your help.
Best regards,
Pradeep


On Sun, Apr 24, 2022 at 11:43 PM Adrian Klaver 
wrote:

> On 4/24/22 08:31, Pradeep Chhetri wrote:
> > Hello everyone,
> >
> > I am sure this has been discussed multiple times in the past but I would
> > like to initiate this discussion again. I have 3 nodes cluster of
> > Postgres v9.6. They all are currently running on Debian 9 (with glibc
> > v2.24) and need to upgrade them to Debian 10 (with glibc v2.28) without
> > downtime. In order to bypass the glibc issue, I am trying to evaluate
> > whether I can compile glibc v2.24 on Debian 10, pin postgres to use this
> > manually compiled glibc and upgrade the linux distribution in rolling
> > fashion. I would like to know how others have achieved such distro
> > upgrades without downtime. I am new to Postgres so please pardon my
> > ignorance.
>
> You are going to have to be more specific as upgrading a distro involves
> downtime. I'm guessing you mean downtime for Postgres, still at least
> one of the instances is going to be down while it's OS is being
> upgraded. So:
>
> 1) Define how the 3 node cluster works?
>
> 2) What is the locale for the Postgres instances?
>
> 3) What is acceptable downtime in the process?
>
> 4) Are you using ICU collation?
>
> Also you might want to look at:
>
> https://wiki.postgresql.org/wiki/Locale_data_changes
>
> >
> > Thank you for your help.
> > Best regards,
> > Pradeep
>
>
> --
> Adrian Klaver
> adrian.kla...@aklaver.com
>


Re: Handling glibc v2.28 breaking changes

2022-04-25 Thread Pradeep Chhetri
Thank you Laurenz and Nick. That sounds like a good plan to me.

Best Regards,
Pradeep

On Mon, Apr 25, 2022 at 9:44 PM Nick Cleaton  wrote:

> On Mon, 25 Apr 2022 at 12:45, Laurenz Albe 
> wrote:
>
>>
>> You could consider upgrade in several steps:
>>
>> - pg_upgrade to v14 on the current operating system
>> - use replication, than switchover to move to a current operating system
>> on a different
>>   machine
>> - REINDEX CONCURRENTLY all indexes on string expressions
>>
>> You could get data corruption and bad query results between the second
>> and the third steps,
>> so keep that interval short.
>>
>
> We did something like this, with the addition of a step where we used a
> new-OS replica to run amcheck's bt_index_check() over all of the btree
> indexes to find those actually corrupted by the libc upgrade in practice
> with our data. It was a small fraction of them, and we were able to fit an
> offline reindex of those btrees and all texty non-btree indexes into an
> acceptable downtime window, with REINDEX CONCURRENTLY of everything else as
> a lower priority after the upgrade.
>
>