Re: CentOS 7 yum package systemd bug?

[Laurenz Albe]

On Tue, 2020-11-03 at 09:42 -0600, Doug Whitfield wrote:
> Unclear to me if this is a systemd bug or a Postgresql 12 bug, so I figured I 
> would get some thoughts here before reporting in detail.
> 
> It is pretty simple to reproduce. If you start your standby server with 
> incorrect username or password
>  using ’systemctl start postgresql-12’ then systemctl just “hangs”. The 
> replication issues get logged,
>  and it isn’t hard to fix, but it doesn’t seem like the appropriate outcome. 
> If you make a syntax error,
>  the systemctl knows that you failed. Of course, this is better than having a 
> normal exit status and
>  moving on with life only to find out your replication isn’t working, but it 
> doesn’t seem right.
> 
> To be clear, I already fixed the issue. I am just wondering if people think 
> this is a systemd bug
>  or a PostgresQL bug or it is just a garbage in, garbage out sort of 
> situation not worth filing anywhere.

That must be the "Type=notify" from the service file.

PostgreSQL notifies systemd as soon as it has started up, which didn't happen 
in your case.

The idea is that later services that depend on PostgreSQL can rely on it being 
available.
I think that is a good thing to have.
I am no systemd expert, but as far as I know services are started in parallel, 
so it
shouldn't block your boot process for other services that don't depend on 
PostgreSQL.

The best place to discuss this would be the "pgsql-pkg-yum" list.

Yours,
Laurenz Albe
-- 
Cybertec | https://www.cybertec-postgresql.com

Re: CentOS 7 yum package systemd bug?

[Magnus Hagander]

On Wed, Nov 4, 2020 at 9:45 AM Laurenz Albe  wrote:
>
> On Tue, 2020-11-03 at 09:42 -0600, Doug Whitfield wrote:
> > Unclear to me if this is a systemd bug or a Postgresql 12 bug, so I figured 
> > I would get some thoughts here before reporting in detail.
> >
> > It is pretty simple to reproduce. If you start your standby server with 
> > incorrect username or password
> >  using ’systemctl start postgresql-12’ then systemctl just “hangs”. The 
> > replication issues get logged,
> >  and it isn’t hard to fix, but it doesn’t seem like the appropriate 
> > outcome. If you make a syntax error,
> >  the systemctl knows that you failed. Of course, this is better than having 
> > a normal exit status and
> >  moving on with life only to find out your replication isn’t working, but 
> > it doesn’t seem right.
> >
> > To be clear, I already fixed the issue. I am just wondering if people think 
> > this is a systemd bug
> >  or a PostgresQL bug or it is just a garbage in, garbage out sort of 
> > situation not worth filing anywhere.
>
> That must be the "Type=notify" from the service file.
>
> PostgreSQL notifies systemd as soon as it has started up, which didn't happen 
> in your case.
>
> The idea is that later services that depend on PostgreSQL can rely on it 
> being available.
> I think that is a good thing to have.
> I am no systemd expert, but as far as I know services are started in 
> parallel, so it
> shouldn't block your boot process for other services that don't depend on 
> PostgreSQL.
>

I believe in a hot standby system, we will send the notification to
systemd once we have reached a consistent state and are "ready for
read only connections". If the system is not in that state, and cannot
connect to the primary, then it seems like indeed it gets stuck there
"forever".

I think one can argue whether that's good or bad :)

The PostgreSQL documentation at
https://www.postgresql.org/docs/13/server-start.html talks about this
behaviour, but only notes that crash recovery might be a reason to hit
this timeout. Maybe it needs to also mention replication (and probably
archive recovery)?


> The best place to discuss this would be the "pgsql-pkg-yum" list.

I don't think this is a packaging issue, all the RPMs did was enable
the functionality that's in core postgresql.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

PANIC: could not write to log file {} at offset {}, length {}: Invalid argument

[Shani Israeli]

Hi all,

We are running PostgreSQL v9.5.19 over Windows Server 2012 R2, 16GB RAM.
Lately, postgres started to crash (happened already 3 times ~once a month)
and before its crashes I found this message in Event Log:

PANIC:  could not write to log file {} at offset {}, length {}: Invalid
argument

(so I assumed it is related).

attached is our configuration.

Any ideas about what is the problem? or anything else I need to check?

Thanks is advance,
Shani Israeli - Software Developer

+972 54 6689920
sisra...@illusivenetworks.com
www.illusivenetworks.com


postgres.conf
Description: Binary data

Re: PANIC: could not write to log file {} at offset {}, length {}: Invalid argument

[David Day]

I Could not read your config file but
What is the size of the Postgres log file?
Do you have a log rotation policy on it?
Perhaps your Postgres log level is to high or your connections are
generating a lot of errors that need investigating.

Dave

On Wed, Nov 4, 2020 at 5:24 AM Shani Israeli 
wrote:

> Hi all,
>
> We are running PostgreSQL v9.5.19 over Windows Server 2012 R2, 16GB RAM.
> Lately, postgres started to crash (happened already 3 times ~once a month)
> and before its crashes I found this message in Event Log:
>
> PANIC:  could not write to log file {} at offset {}, length {}: Invalid
> argument
>
> (so I assumed it is related).
>
> attached is our configuration.
>
> Any ideas about what is the problem? or anything else I need to check?
>
> Thanks is advance,
> Shani Israeli - Software Developer
>
> +972 54 6689920
> sisra...@illusivenetworks.com
> www.illusivenetworks.com
>
>
>

Re: PANIC: could not write to log file {} at offset {}, length {}: Invalid argument

[Adrian Klaver]

On 11/4/20 2:24 AM, Shani Israeli wrote:

Hi all,

We are running PostgreSQL v9.5.19 over Windows Server 2012 R2, 16GB RAM.
Lately, postgres started to crash (happened already 3 times ~once a 
month) and before its crashes I found this message in Event Log:


PANIC:  could not write to log file {} at offset {}, length {}:
Invalid argument

(so I assumed it is related).

attached is our configuration.

Any ideas about what is the problem? or anything else I need to check?


Any time I see seemingly random crashes involving file corruption on 
Windows I think anti-virus software. Has someone turned an AV program 
loose on this machine?




Thanks is advance,
Shani Israeli - Software Developer
+972 54 6689920

sisra...@illusivenetworks.com

www.illusivenetworks.com 




--
Adrian Klaver
adrian.kla...@aklaver.com

Re: PANIC: could not write to log file {} at offset {}, length {}: Invalid argument

[Andreas Kretschmer]

On 4 November 2020 11:24:03 CET, Shani Israeli  
wrote:
>Hi all,
>
>We are running PostgreSQL v9.5.19 over Windows Server 2012 R2, 16GB
>RAM.
>Lately, postgres started to crash (happened already 3 times ~once a
>month)
>and before its crashes I found this message in Event Log:
>
>PANIC:  could not write to log file {} at offset {}, length {}: Invalid
>argument
>
>(so I assumed it is related).
>
>attached is our configuration.
>
>Any ideas about what is the problem? or anything else I need to check?


wild guess: Antivirus Software?



>
>Thanks is advance,
>Shani Israeli - Software Developer
>
>+972 54 6689920
>sisra...@illusivenetworks.com
>www.illusivenetworks.com


-- 
2ndQuadrant - The PostgreSQL Support Company

Race condition with restore_command on streaming replica

[Brad Nicholson]

Hi,

I've recently been seeing a race condition with the restore_command on
replicas using streaming replication.

On the primary, we are archiving wal files to s3 compatible storage via
pgBackRest. In the recovery.conf section of the postgresql.conf file on the
replicas, we define the restore command as follows:

restore_command = '/usr/bin/pgbackrest
--config /conf/postgresql/pgbackrest_restore.conf --stanza=formation
archive-get %f "%p"'

We have a three member setup - m-0, m-1, m-2.  Consider the case where m-0
is the Primary and m-1 and m-2 are replicas connected to the m-0.

When issuing a switchover (via Patroni) from m-0 to m-1, the connection
from m-2 to m-0 is terminated.  The restore_command on m-2 is run, and it
looks for the .history file for the new timeline.  If this happens before
the history file is created and pushed to the archive, m-2 will look for
the next wal file on the existing timeline in the archive.  It will never
be created as the source has moved on, so this m-2 hangs waiting on that
file.  The restore_command on the replica looking for this non-existent
file is only run once. This seems like an odd state to be in.  The replica
is waiting on a new file, but it's not actually looking for it.  Is this
expected, or should the restore_command be polling for that file?

Restarting m-2 causes it to reconnect, find the .history file in the
archive and replication picks up.  But that is less than ideal.

This is Postgres 12.4, managed by Patroni.  I reported this to Patroni
first for feedback, and they pointed to it be being a Postgres issue - not
a Patroni one.

Here is a summarized timeline of events from one instance, cluster is on
Time Line 9

2020-11-04 14:51:11 m-0 is shutdown
2020-11-04 14:51:13.015 m-2 replication connection is terminated
2020-11-04 14:51:13.015 m-2 looks for 000A.history in archive, does not
find it
2020-11-04 14:51:17,137 m-2 looks for 000900B6 in archive
(this file won't ever exist)
2020-11-04 14:51:20,177 m-1 pushes 000A.history to archive

At this point, m-2 is stuck until reboot

There is a detailed bug report on the Patroni list with all the logs if
relevant:
https://github.com/zalando/patroni/issues/1747#issue-736244675

Christopher Browne

[Steve Singer]

It is with much sadness that I am letting the community know that Chris 
Browne passed away recently.


Chris had been a long time community member and was active on various 
Postgresql mailing lists.  He was a member of the funds committee, the PgCon 
program committee and served on the board of the PostgreSQL Community 
Association of Canada. Chris was a maintainer of the Slony 
replication system and worked on various PG related tools.


I worked with Chris for over 10 years and anytime someone had a problem he 
would jump at the chance to help and propose solutions. He 
always had time to listen to your problem and offer ideas or explain how 
something worked.


I will miss Chris

Steve

Re: Christopher Browne

[Bikram Majumdar]

So sorry to hear this news.

On Wed, Nov 4, 2020, 17:29 Steve Singer  wrote:

>
> It is with much sadness that I am letting the community know that Chris
> Browne passed away recently.
>
> Chris had been a long time community member and was active on various
> Postgresql mailing lists.  He was a member of the funds committee, the
> PgCon
> program committee and served on the board of the PostgreSQL Community
> Association of Canada. Chris was a maintainer of the Slony
> replication system and worked on various PG related tools.
>
> I worked with Chris for over 10 years and anytime someone had a problem he
> would jump at the chance to help and propose solutions. He
> always had time to listen to your problem and offer ideas or explain how
> something worked.
>
> I will miss Chris
>
> Steve
>
>
>
>

Re: Christopher Browne

[Christophe Pettus]

> On Nov 4, 2020, at 15:29, Steve Singer  wrote:
> 
> It is with much sadness that I am letting the community know that Chris 
> Browne passed away recently.

I'm so sorry to hear this.  I did not know him well, but he was always a kind 
and friendly face anytime I saw him at events.

--
-- Christophe Pettus
   x...@thebuild.com

Re: Christopher Browne

[Jan Wieck]

Christopher Browne was my colleague and friend at Afilias and past for more
than a decade. I have countless memories that connect us. He will be missed.


Jan

On Wed, Nov 4, 2020, 18:29 Steve Singer  wrote:

>
> It is with much sadness that I am letting the community know that Chris
> Browne passed away recently.
>
> Chris had been a long time community member and was active on various
> Postgresql mailing lists.  He was a member of the funds committee, the
> PgCon
> program committee and served on the board of the PostgreSQL Community
> Association of Canada. Chris was a maintainer of the Slony
> replication system and worked on various PG related tools.
>
> I worked with Chris for over 10 years and anytime someone had a problem he
> would jump at the chance to help and propose solutions. He
> always had time to listen to your problem and offer ideas or explain how
> something worked.
>
> I will miss Chris
>
> Steve
>
>
>
>

Re: Christopher Browne

[Pavan Pusuluri]

So sorry to hear the news. May his soul rest in peace.


Regards
Pavan

On Wed, Nov 4, 2020, 6:46 PM Jan Wieck  wrote:

> Christopher Browne was my colleague and friend at Afilias and past for
> more than a decade. I have countless memories that connect us. He will be
> missed.
>
>
> Jan
>
> On Wed, Nov 4, 2020, 18:29 Steve Singer  wrote:
>
>>
>> It is with much sadness that I am letting the community know that Chris
>> Browne passed away recently.
>>
>> Chris had been a long time community member and was active on various
>> Postgresql mailing lists.  He was a member of the funds committee, the
>> PgCon
>> program committee and served on the board of the PostgreSQL Community
>> Association of Canada. Chris was a maintainer of the Slony
>> replication system and worked on various PG related tools.
>>
>> I worked with Chris for over 10 years and anytime someone had a problem
>> he
>> would jump at the chance to help and propose solutions. He
>> always had time to listen to your problem and offer ideas or explain how
>> something worked.
>>
>> I will miss Chris
>>
>> Steve
>>
>>
>>
>>

Re: Christopher Browne

[Joe Conway]

On 11/4/20 7:45 PM, Jan Wieck wrote:
> Christopher Browne was my colleague and friend at Afilias and past for more 
> than
> a decade. I have countless memories that connect us. He will be missed.
> 
> 
> Jan


He will definitely be missed. Rest in Peace Chris.

Joe


> On Wed, Nov 4, 2020, 18:29 Steve Singer  > wrote:
> 
> 
> It is with much sadness that I am letting the community know that Chris
> Browne passed away recently.
> 
> Chris had been a long time community member and was active on various
> Postgresql mailing lists.  He was a member of the funds committee, the 
> PgCon
> program committee and served on the board of the PostgreSQL Community
> Association of Canada. Chris was a maintainer of the Slony
> replication system and worked on various PG related tools.
> 
> I worked with Chris for over 10 years and anytime someone had a problem he
> would jump at the chance to help and propose solutions. He
> always had time to listen to your problem and offer ideas or explain how
> something worked.
> 
> I will miss Chris
> 
> Steve
> 
> 
> 


-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development



signature.asc
Description: OpenPGP digital signature

Re: Christopher Browne

[raf]

On Wed, Nov 04, 2020 at 06:29:18PM -0500, Steve Singer  
wrote:

> 
> It is with much sadness that I am letting the community know that Chris
> Browne passed away recently.
> 
> Chris had been a long time community member and was active on various
> Postgresql mailing lists.  He was a member of the funds committee, the PgCon
> program committee and served on the board of the PostgreSQL Community
> Association of Canada. Chris was a maintainer of the Slony replication
> system and worked on various PG related tools.
> 
> I worked with Chris for over 10 years and anytime someone had a problem he
> would jump at the chance to help and propose solutions. He always had time
> to listen to your problem and offer ideas or explain how something worked.
> 
> I will miss Chris
> 
> Steve

He sounds exactly like the kind of person you want to have around.

Re: Christopher Browne

[Dave Cramer]

On Wed., Nov. 4, 2020, 8:10 p.m. raf,  wrote:

> On Wed, Nov 04, 2020 at 06:29:18PM -0500, Steve Singer 
> wrote:
>
> >
> > It is with much sadness that I am letting the community know that Chris
> > Browne passed away recently.
> >
> > Chris had been a long time community member and was active on various
> > Postgresql mailing lists.  He was a member of the funds committee, the
> PgCon
> > program committee and served on the board of the PostgreSQL Community
> > Association of Canada. Chris was a maintainer of the Slony replication
> > system and worked on various PG related tools.
> >
> > I worked with Chris for over 10 years and anytime someone had a problem
> he
> > would jump at the chance to help and propose solutions. He always had
> time
> > to listen to your problem and offer ideas or explain how something
> worked.
> >
> > I will miss Chris
> >
> > Steve
>
> He sounds exactly like the kind of person you want to have around.
>
>
Very sorry to hear about Chris. I considered him a friend. He will be
missed.

Dave


>

Re: Christopher Browne

[Nikolay Samokhvalov]

What a sad news.

Rest In Peace, Christopher.

Condolences to friends and family.

On Wed, Nov 4, 2020 at 15:29 Steve Singer  wrote:

>
> It is with much sadness that I am letting the community know that Chris
> Browne passed away recently.
>
> Chris had been a long time community member and was active on various
> Postgresql mailing lists.  He was a member of the funds committee, the
> PgCon
> program committee and served on the board of the PostgreSQL Community
> Association of Canada. Chris was a maintainer of the Slony
> replication system and worked on various PG related tools.
>
> I worked with Chris for over 10 years and anytime someone had a problem he
> would jump at the chance to help and propose solutions. He
> always had time to listen to your problem and offer ideas or explain how
> something worked.
>
> I will miss Chris
>
> Steve
>
>
>
>

Re: PANIC: could not write to log file {} at offset {}, length {}: Invalid argument

[Michael Paquier]

On Wed, Nov 04, 2020 at 01:24:46PM +0100, Andreas Kretschmer wrote:
>> Any ideas about what is the problem? or anything else I need to check?
> 
> wild guess: Antivirus Software?

Perhaps not.  To bring more context in here, PostgreSQL opens any
files on WIN32 with shared writes and reads allowed to have an
equivalent of what we do on all *nix platforms.  Note here that the
problem comes from a WAL segment write, which is done after the file
handle is opened in shared mode.  As long as the fd is correctly
opened, any attempt for an antivirus software to open a file with an 
exclusive write would be blocked, no?
--
Michael


signature.asc
Description: PGP signature

Re: PANIC: could not write to log file {} at offset {}, length {}: Invalid argument

[Tom Lane]

Michael Paquier  writes:
> On Wed, Nov 04, 2020 at 01:24:46PM +0100, Andreas Kretschmer wrote:
>> wild guess: Antivirus Software?

> Perhaps not.  To bring more context in here, PostgreSQL opens any
> files on WIN32 with shared writes and reads allowed to have an
> equivalent of what we do on all *nix platforms.  Note here that the
> problem comes from a WAL segment write, which is done after the file
> handle is opened in shared mode.  As long as the fd is correctly
> opened, any attempt for an antivirus software to open a file with an 
> exclusive write would be blocked, no?

The only hard data point we've got here is the "Invalid argument"
string, which should mean EINVAL, although I'm not entirely sure
where that string is determined in a Windows build.  So it seems
like there are two possibilities:

* The actual underlying Windows error code is one of the ones
that win32error.c maps to EINVAL:

ERROR_INVALID_FUNCTION, EINVAL
ERROR_INVALID_ACCESS, EINVAL
ERROR_INVALID_DATA, EINVAL
ERROR_INVALID_PARAMETER, EINVAL
ERROR_INVALID_HANDLE, EINVAL
ERROR_NEGATIVE_SEEK, EINVAL

* The actual underlying Windows error code is something that
win32error.c doesn't know, which would cause _dosmaperr() to
return EINVAL.

The latter case would result in a LOG message "unrecognized win32 error
code", so it would be good to know if any of those are showing up in
the postmaster log.

Seems like maybe it wasn't a great idea for _dosmaperr's fallback
errno to be something that is also a real error code.

regards, tom lane

Re: Christopher Browne

[Vincenzo Campanella]

RIP Chris

Il 05.11.2020 02:09, raf ha scritto:

On Wed, Nov 04, 2020 at 06:29:18PM -0500, Steve Singer  
wrote:


It is with much sadness that I am letting the community know that Chris
Browne passed away recently.

Chris had been a long time community member and was active on various
Postgresql mailing lists.  He was a member of the funds committee, the PgCon
program committee and served on the board of the PostgreSQL Community
Association of Canada. Chris was a maintainer of the Slony replication
system and worked on various PG related tools.

I worked with Chris for over 10 years and anytime someone had a problem he
would jump at the chance to help and propose solutions. He always had time
to listen to your problem and offer ideas or explain how something worked.

I will miss Chris

Steve

He sounds exactly like the kind of person you want to have around.