Fix typo in Comments of HandleProcSignalBarrierInterrupt

[Hou, Zhijie]

Hi all

In master branch, I found a typo in Comments of function 
HandleProcSignalBarrierInterrupt.
See the attachment for the patch.

Best regards.





fix-typo.patch
Description: fix-typo.patch

Re: SSL between Primary and Seconday PostgreSQL DBs

[Susan Joseph]

OK, I understand I was just hoping someone could confirm that my settings are 
correct.  
I didn't come across an error so everything seems to be working I just can't 
verify that SSL is working.  
Are there any commands you can run to verify that SSL is up and operational? 
 Testing from a client to the database doesn't prove that database to database 
is working.   
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Peter Eisentraut 
To: Susan Joseph ; pgsql-gene...@postgresql.org 

Sent: Thu, Sep 3, 2020 1:01 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

On 2020-08-27 12:57, Susan Joseph wrote:
> So has no one done this before?

I'm sure people have done this.  But I suggest that if you ask a 
question on this mailing list, you ask something more concrete, like, I 
tried to do this, and got stuck here, and tried this and got this error. 
  People can help with that sort of thing.  What we have here is a 
complex security setup and you are asking people to do an open-ended 
review.  No one wants to do that.

> -Original Message-
> From: Susan Joseph 
> To: pgsql-gene...@postgresql.org 
> Sent: Mon, Aug 24, 2020 10:10 am
> Subject: SSL between Primary and Seconday PostgreSQL DBs
> 
> I have setup a Primary and Secondary PostgreSQL DBs.  They were setup up 
> with basic replication then I went back and modified them to use SSL.  I 
> am just not sure if I did it correctly.  Everything is working but I 
> want to make sure I have the settings correctly.  I am using PostgreSQL 
> 11.2.
> 
>  * I have a PKI that I stood up so I issued 2 server certificates one
>    for each database from my CA.
>  * Primary server certificate - Primary Database
>      o The FQDN and IP address are set in the SAN field.
>      o FQDN is also the CN in the DN
>      o Key Usage is set to Digital Signature and Key encipherment
>      o EKU is set to Server Authentication and Client Authentication
>  * Rep_user certificate - Secondary Database
>      o CN is set to the rep_user account name
>      o Key Usage is set to digital signature and key encipherment
>      o EKU is set to client authentication
>  * Each certificate file contains the certificate and the subCA
>    certificate who issued the certificate and put in a file called
>    server.crt for the Primary and client.crt for the secondary.
>  * The key for each certificate is stored in a separate file
>    unencrypted (I have questions about this later on) in a file called
>    server.key and client.key
>  * The server.crt, server.key, and root.crt are put onto the primary
>    database server in the /data/pgsql/data location, the owner and
>    group of these files is set to postgres
>  * The client.crt, client.key, and root.crt are put onto the primary
>    database server in the /data/pgsql/data location, the owner and
>    group of these files is set to postgres
>  * On the Primary in postgresql.conf I set:
>      o ssl=on
>      o ssl_ca_file='root.crt'
>      o ssl_cert_file='server.crt'
>      o ssl_key_file='server.key'
>      o ssl_ciphers='HIGH:MEDIUM:+3DES:!aNULL'
>  * On the Primary in pg_hba.conf I add a replication line:
>      o hostssl    replication        
>        rep_user  cert
>  * On the Secondary I set the following information in the
>    postgresql.conf to:  (DO I NEED TO DO THIS??)
>      o ssl=on
>      o ssl_ca_file='root.crt'
>      o ssl_cert_file='client.crt'
>      o ssl_cert_fkey='client.key'
>      o ssl_ciphers='HIGH:MEDIUM:+3DES:!aNULL'
>  * On the Secondary I edit the recovery.conf file to the following:
>      o primary_conninfo = 'user=rep_user passfile=''/data/.pgpass''
>        host= port=5432 sslmode=verify-ca
>        sslcert=client.crt sslkey=client.key sslcompression=0
>        target_session_attrs=any'
>  * On the Secondary I edit the pg_hba.conf file and change the rep_user
>    line to:
>      o hostssl  replication rep_user           IP>/32  cert clientcert=1
>  * On the Secondary I move the root.crt to /data/pgsql/data/.postgresql
>  * Then I restart the databases
> 
> 
> My questions are:
> 
>  * Do I need to set the information in the Secondary postgresql.conf? 
>    Originally I did not set this and everything worked but I saw errors
>    in my log files that said to do SSL these needed to be set so I went
>    back and set them.  Are there pgsql commands I can run to test that
>    my SSL is working in both directions?
>  * Are my pg_hba.conf files set correctly?  Is that how you get SSL
>    "turned on" for communications between the primary and the rep_user
>    account?
>  * If I leave my key file encrypted then every time my databases have
>    to be started have to enter the password.  So you can either leave
>    the passwords unencrypted and set the permissions on the file to
>    0600 accessible only by postgres or you can enter the key password
>    each time the database is started up.  As some

pg_restore causing deadlocks on partitioned tables

[Domagoj Smoljanovic]

Hi all.

I tried searching for the response to this but couldn’t find any.

I have pg_restore running in parallel (3 or more) and processing large amount 
of data that is in partitioned tables. However it seems that sometime deadlock 
appears when one process is trying to process primary key on parent table while 
data still hasn’t been loaded into partitions. And acquires Exclusive Lock on 
the whole table. Then another process comes and tries to load one of the 
partitions with SharedLock but it fails.

This of course doesn’t happen always; depending on the course of actions of the 
pg_restore. But often enough to cause frustration.

Process 15858 waits for AccessShareLock on relation 233358134 of database 
233346697; blocked by process 15861.
Process 15861 waits for AccessExclusiveLock on relation 233374757 of 
database 233346697; blocked by process 15858.
Process 15858: TRUNCATE TABLE ONLY myschema."myTable:2020-09-01";
Process 15861: ALTER TABLE ONLY myschema."myTable" ADD CONSTRAINT 
"pk_myTable" PRIMARY KEY ("ID", date);

Should this be treated as a bug or am I doing something wrong?

Disclamer:  --load-via-partition-root  was NOT used. Meaning that warning from 
the pg_dump documentation should not be applicable 😊

Thanx,
Domagoj

Re: SSL between Primary and Seconday PostgreSQL DBs

[o1bigtenor]

On Thu, Sep 3, 2020 at 5:54 AM Susan Joseph  wrote:
>
> OK, I understand I was just hoping someone could confirm that my settings are 
> correct.
> I didn't come across an error so everything seems to be working I just can't 
> verify that SSL is working.
> Are there any commands you can run to verify that SSL is up and operational?
> Testing from a client to the database doesn't prove that database to database 
> is working.
>
>

I'm most definitely NOT any kind of either Postgresql nor computer
security expert.

Did a search using 'linus testing SSL' and found some possible options that
you may wish to consider. These did not seem to be specific to database access
testing but more to website testing. I believe that, website testing,
would give you the information you are looking for.

Regards

Re: SSL between Primary and Seconday PostgreSQL DBs

[Wim Bertels]

First thougt: A general solution that you could use is wireshark.

For example : Search for wireshark ssl traffic


Susan Joseph  schreef op September 3, 2020 10:54:36 AM 
UTC:
>OK, I understand I was just hoping someone could confirm that my
>settings are correct.  
>I didn't come across an error so everything seems to be working I just
>can't verify that SSL is working.  
>Are there any commands you can run to verify that SSL is up and
>operational? 
>Testing from a client to the database doesn't prove that database to
>database is working.   
> 
>Susan Joseph
>sandajos...@verizon.net
> 
> 
>-Original Message-
>From: Peter Eisentraut 
>To: Susan Joseph ;
>pgsql-gene...@postgresql.org 
>Sent: Thu, Sep 3, 2020 1:01 am
>Subject: Re: SSL between Primary and Seconday PostgreSQL DBs
>
>On 2020-08-27 12:57, Susan Joseph wrote:
>> So has no one done this before?
>
>I'm sure people have done this.  But I suggest that if you ask a 
>question on this mailing list, you ask something more concrete, like, I
>
>tried to do this, and got stuck here, and tried this and got this
>error. 
>  People can help with that sort of thing.  What we have here is a 
>complex security setup and you are asking people to do an open-ended 
>review.  No one wants to do that.
>
>> -Original Message-
>> From: Susan Joseph 
>> To: pgsql-gene...@postgresql.org 
>> Sent: Mon, Aug 24, 2020 10:10 am
>> Subject: SSL between Primary and Seconday PostgreSQL DBs
>> 
>> I have setup a Primary and Secondary PostgreSQL DBs.  They were setup
>up 
>> with basic replication then I went back and modified them to use
>SSL.  I 
>> am just not sure if I did it correctly.  Everything is working but I 
>> want to make sure I have the settings correctly.  I am using
>PostgreSQL 
>> 11.2.
>> 
>>  * I have a PKI that I stood up so I issued 2 server certificates one
>>    for each database from my CA.
>>  * Primary server certificate - Primary Database
>>      o The FQDN and IP address are set in the SAN field.
>>      o FQDN is also the CN in the DN
>>      o Key Usage is set to Digital Signature and Key encipherment
>>      o EKU is set to Server Authentication and Client Authentication
>>  * Rep_user certificate - Secondary Database
>>      o CN is set to the rep_user account name
>>      o Key Usage is set to digital signature and key encipherment
>>      o EKU is set to client authentication
>>  * Each certificate file contains the certificate and the subCA
>>    certificate who issued the certificate and put in a file called
>>    server.crt for the Primary and client.crt for the secondary.
>>  * The key for each certificate is stored in a separate file
>>    unencrypted (I have questions about this later on) in a file
>called
>>    server.key and client.key
>>  * The server.crt, server.key, and root.crt are put onto the primary
>>    database server in the /data/pgsql/data location, the owner and
>>    group of these files is set to postgres
>>  * The client.crt, client.key, and root.crt are put onto the primary
>>    database server in the /data/pgsql/data location, the owner and
>>    group of these files is set to postgres
>>  * On the Primary in postgresql.conf I set:
>>      o ssl=on
>>      o ssl_ca_file='root.crt'
>>      o ssl_cert_file='server.crt'
>>      o ssl_key_file='server.key'
>>      o ssl_ciphers='HIGH:MEDIUM:+3DES:!aNULL'
>>  * On the Primary in pg_hba.conf I add a replication line:
>>      o hostssl    replication        
>>        rep_user  cert
>>  * On the Secondary I set the following information in the
>>    postgresql.conf to:  (DO I NEED TO DO THIS??)
>>      o ssl=on
>>      o ssl_ca_file='root.crt'
>>      o ssl_cert_file='client.crt'
>>      o ssl_cert_fkey='client.key'
>>      o ssl_ciphers='HIGH:MEDIUM:+3DES:!aNULL'
>>  * On the Secondary I edit the recovery.conf file to the following:
>>      o primary_conninfo = 'user=rep_user passfile=''/data/.pgpass''
>>        host= port=5432 sslmode=verify-ca
>>        sslcert=client.crt sslkey=client.key sslcompression=0
>>        target_session_attrs=any'
>>  * On the Secondary I edit the pg_hba.conf file and change the
>rep_user
>>    line to:
>>      o hostssl  replication rep_user  
>>        IP>/32  cert clientcert=1
>>  * On the Secondary I move the root.crt to
>/data/pgsql/data/.postgresql
>>  * Then I restart the databases
>> 
>> 
>> My questions are:
>> 
>>  * Do I need to set the information in the Secondary postgresql.conf?
>
>>    Originally I did not set this and everything worked but I saw
>errors
>>    in my log files that said to do SSL these needed to be set so I
>went
>>    back and set them.  Are there pgsql commands I can run to test
>that
>>    my SSL is working in both directions?
>>  * Are my pg_hba.conf files set correctly?  Is that how you get SSL
>>    "turned on" for communications between the primary and the
>rep_user
>>    account?
>>  * If I leave my key file encrypted then eve

Re: SSL between Primary and Seconday PostgreSQL DBs

[Susan Joseph]

Unfortunately I am not allowed to use wireshark in my environment.  Good idea 
though
 
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Wim Bertels 
To: Susan Joseph ; peter.eisentr...@2ndquadrant.com 
; pgsql-gene...@postgresql.org 

Sent: Thu, Sep 3, 2020 7:44 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

First thougt: A general solution that you could use is wireshark.

For example : Search for wireshark ssl traffic


Susan Joseph  schreef op September 3, 2020 10:54:36 AM 
UTC:
OK, I understand I was just hoping someone could confirm that my settings are 
correct.  
I didn't come across an error so everything seems to be working I just can't 
verify that SSL is working.  
Are there any commands you can run to verify that SSL is up and operational? 
 Testing from a client to the database doesn't prove that database to database 
is working.   
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Peter Eisentraut 
To: Susan Joseph ; pgsql-gene...@postgresql.org 

Sent: Thu, Sep 3, 2020 1:01 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

On 2020-08-27 12:57, Susan Joseph wrote:
> So has no one done this before?

I'm sure people have done this.  But I suggest that if you ask a 
question on this mailing list, you ask something more concrete, like, I 
tried to do this, and got stuck here, and tried this and got this error. 
  People can help with that sort of thing.  What we have here is a 
complex security setup and you are asking people to do an open-ended 
review.  No one wants to do that.

> -Original Message-
> From: Susan Joseph 
> To: pgsql-gene...@postgresql.org 
> Sent: Mon, Aug 24, 2020 10:10 am
> Subject: SSL between Primary and Seconday PostgreSQL DBs
> 
> I have setup a Primary and Secondary PostgreSQL DBs.  They were setup up 
> with basic replication then I went back and modified them to use SSL.  I 
> am just not sure if I did it correctly.  Everything is working but I 
> want to make sure I have the settings correctly.  I am using PostgreSQL 
> 11.2.
> 
>  * I have a PKI that I stood up so I issued 2 server certificates one
>    for each database from my CA.
>  * Primary server certificate - Primary Database
>      o The FQDN and IP address are set in the SAN field.
>      o FQDN is also the CN in the DN
>      o Key Usage is set to Digital Signature and Key encipherment
>      o EKU is set to Server Authentication and Client Authentication
>  * Rep_user certificate - Secondary Database
>      o CN is set to the rep_user account name
>      o Key Usage is set to digital signature and key encipherment
>      o EKU is set to client authentication
>  * Each certificate file contains the certificate and the subCA
>    certificate who issued the certificate and put in a file called
>    server.crt for the Primary and client.crt for the secondary.
>  * The key for each certificate is stored in a separate file
>    unencrypted (I have questions about this later on) in a file called
>    server.key and client.key
>  * The server.crt, server.key, and root.crt are put onto the primary
>    database server in the /data/pgsql/data location, the owner and
>    group of these files is set to postgres
>  * The client.crt, client.key, and root.crt are put onto the primary
>    database server in the /data/pgsql/data location, the owner and
>    group of these files is set to postgres
>  * On the Primary in postgresql.conf I set:
>      o ssl=on
>      o ssl_ca_file='root.crt'
>      o ssl_cert_file='server.crt'
>      o ssl_key_file='server.key'
>      o ssl_ciphers='HIGH:MEDIUM:+3DES:!aNULL'
>  * On the Primary in pg_hba.conf I add a replication line:
>      o hostssl    replication        
>        rep_user  cert
>  * On the Secondary I set the following information in the
>    postgresql.conf to:  (DO I NEED TO DO THIS??)
>      o ssl=on
>      o ssl_ca_file='root.crt'
>      o ssl_cert_file='client.crt'
>      o ssl_cert_fkey='client.key'
>      o ssl_ciphers='HIGH:MEDIUM:+3DES:!aNULL'
>  * On the Secondary I edit the recovery.conf file to the following:
>      o primary_conninfo = 'user=rep_user passfile=''/data/.pgpass''
>        host= port=5432 sslmode=verify-ca
>        sslcert=client.crt sslkey=client.key sslcompression=0
>        target_session_attrs=any'
>  * On the Secondary I edit the pg_hba.conf file and change the rep_user
>    line to:
>      o hostssl  replication rep_user           IP>/32  cert clientcert=1
>  * On the Secondary I move the root.crt to /data/pgsql/data/.postgresql
>  * Then I restart the databases
> 
> 
> My questions are:
> 
>  * Do I need to set the information in the Secondary postgresql.conf? 
>    Originally I did not set this and everything worked but I saw errors
>    in my log files that said to do SSL these needed to be set so I went
>    back and set them.  Are there pgsql commands I can run to test that
>   

Re: SSL between Primary and Seconday PostgreSQL DBs

[Stephen Frost]

Greetings,

* Susan Joseph (sandajos...@verizon.net) wrote:
>- primary_conninfo = 'user=rep_user passfile=''/data/.pgpass'' 
> host= port=5432 sslmode=verify-ca sslcert=client.crt 
> sslkey=client.key sslcompression=0 target_session_attrs=any'

You really should be using sslmode=verify-full, otherwise any
certificate signed by a trusted CA on the server side is accepted.

Also, you shouldn't really need to pass in 'passfile' above...

>- On the Secondary I edit the pg_hba.conf file and change the rep_user 
> line to:
>
>- hostssl  replication rep_user   /32  
>     cert clientcert=1

Saying clientcert=1 when using cert auth really shouldn't be needed.

> My questions are:   
>- Do I need to set the information in the Secondary postgresql.conf?  
> Originally I did not set this and everything worked but I saw errors in my 
> log files that said to do SSL these needed to be set so I went back and set 
> them.  Are there pgsql commands I can run to test that my SSL is working in 
> both directions?   

The only connection you're talking about here is from the secondary to
the primary and for that you just need the primary_conninfo settings for
the secondary set up correctly and the SSL settings on the primary.
However, if you want people to be able to make SSL connections to the
secondary, then you need to configure SSL on the secondary.

As for testing the connection, see the pg_stat_ssl view.

>- Are my pg_hba.conf files set correctly?  Is that how you get SSL "turned 
> on" for communications between the primary and the rep_user account?

'hostssl' will only match an incoming connection if it's being made over
SSL.  As long as you don't have anything else in your pg_hba.conf, then
only SSL connections will be allowed.  However, first entry in the
pg_hba.conf matches, so if you have earlier entries, those might be
getting used instead.

>- If I leave my key file encrypted then every time my databases have to be 
> started have to enter the password.  So you can either leave the passwords 
> unencrypted and set the permissions on the file to 0600 accessible only by 
> postgres or you can enter the key password each time the database is started 
> up.  As someone in the security field I have a tough time leaving the key 
> unencrypted but as some setting up a production system that is located on a 
> network that you can't get to without directly accessing the server I feel 
> that is enough security that I can leave them unencrypted.  Thoughts?

You could use a vaulting system to pull the key and make it available at
startup and then remove it after, perhaps, but I would suggest that it's
pretty common to have SSL keys unencrypted on systems which are doing
SSL and otherwise secured.

>- Am I missing anything?  There are no videos out there that show how to 
> stand up a 2 way SSL communication channel between the primary and secondary, 
> or does anyone have one that they can share?

Unfortunately, there's definitely areas here where we could, and really
should, improve when it comes to logging exactly what validation has
been done on incoming connections, to provide the kind of reassurance
you're looking for.  Things like the difference between verify-ca and
verify-full aren't really very well explained, particularly since it's
very uncommon, in my experience, for people who have used SSL/TLS in
other places to have any concept of "verify-ca" since it's basically
"don't actually verify that the other side is who they claim to be"..

Thanks,

Stephen


signature.asc
Description: PGP signature

Re: SSL between Primary and Seconday PostgreSQL DBs

[Susan Joseph]

This is great, thanks.  I have not heard of verify-full but I will try that and 
let you know if I have issues.
So I will remove the passfile info for recovery.conf.  It was there because I 
first set it up using passwords to make sure it was working and then moved to 
SSL so if not needed I can remove it.
So if I don't put clientcert=1 do I just leave it at cert or should it be set 
to something else?
I will try and drop the other connection lines in the pg_hba.conf but I thought 
the last time I did that it through an error.  Let me try it again and see if 
it works.
So are you saying that "sslmode=verify-ca" doesn't actually verify that the 
certificate is from a CA it trusts?  

So my big question is:  am I really adding more security by turning on SSL or 
is it just more work for the DBA?  


 
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Stephen Frost 
To: Susan Joseph 
Cc: pgsql-gene...@postgresql.org 
Sent: Thu, Sep 3, 2020 8:28 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

Greetings,

* Susan Joseph (sandajos...@verizon.net) wrote:
>    - primary_conninfo = 'user=rep_user passfile=''/data/.pgpass'' 
>host= port=5432 sslmode=verify-ca sslcert=client.crt 
>sslkey=client.key sslcompression=0 target_session_attrs=any'

You really should be using sslmode=verify-full, otherwise any
certificate signed by a trusted CA on the server side is accepted.

Also, you shouldn't really need to pass in 'passfile' above...

>    - On the Secondary I edit the pg_hba.conf file and change the rep_user 
>line to:
>    
>    - hostssl  replication rep_user   /32  
>    cert clientcert=1

Saying clientcert=1 when using cert auth really shouldn't be needed.

> My questions are:  
>    - Do I need to set the information in the Secondary postgresql.conf?  
>Originally I did not set this and everything worked but I saw errors in my log 
>files that said to do SSL these needed to be set so I went back and set them.  
>Are there pgsql commands I can run to test that my SSL is working in both 
>directions?  

The only connection you're talking about here is from the secondary to
the primary and for that you just need the primary_conninfo settings for
the secondary set up correctly and the SSL settings on the primary.
However, if you want people to be able to make SSL connections to the
secondary, then you need to configure SSL on the secondary.

As for testing the connection, see the pg_stat_ssl view.

>    - Are my pg_hba.conf files set correctly?  Is that how you get SSL "turned 
>on" for communications between the primary and the rep_user account?

'hostssl' will only match an incoming connection if it's being made over
SSL.  As long as you don't have anything else in your pg_hba.conf, then
only SSL connections will be allowed.  However, first entry in the
pg_hba.conf matches, so if you have earlier entries, those might be
getting used instead.

>    - If I leave my key file encrypted then every time my databases have to be 
>started have to enter the password.  So you can either leave the passwords 
>unencrypted and set the permissions on the file to 0600 accessible only by 
>postgres or you can enter the key password each time the database is started 
>up.  As someone in the security field I have a tough time leaving the key 
>unencrypted but as some setting up a production system that is located on a 
>network that you can't get to without directly accessing the server I feel 
>that is enough security that I can leave them unencrypted.  Thoughts?

You could use a vaulting system to pull the key and make it available at
startup and then remove it after, perhaps, but I would suggest that it's
pretty common to have SSL keys unencrypted on systems which are doing
SSL and otherwise secured.

>    - Am I missing anything?  There are no videos out there that show how to 
>stand up a 2 way SSL communication channel between the primary and secondary, 
>or does anyone have one that they can share?

Unfortunately, there's definitely areas here where we could, and really
should, improve when it comes to logging exactly what validation has
been done on incoming connections, to provide the kind of reassurance
you're looking for.  Things like the difference between verify-ca and
verify-full aren't really very well explained, particularly since it's
very uncommon, in my experience, for people who have used SSL/TLS in
other places to have any concept of "verify-ca" since it's basically
"don't actually verify that the other side is who they claim to be"..

Thanks,

Stephen

Re: SSL between Primary and Seconday PostgreSQL DBs

[Stephen Frost]

Greetings,

We really prefer it if you don't top-post on these lists but instead
reply in-line, as I did.

* Susan Joseph (sandajos...@verizon.net) wrote:
> This is great, thanks.  I have not heard of verify-full but I will try that 
> and let you know if I have issues.
> So I will remove the passfile info for recovery.conf.  It was there because I 
> first set it up using passwords to make sure it was working and then moved to 
> SSL so if not needed I can remove it.
> So if I don't put clientcert=1 do I just leave it at cert or should it be set 
> to something else?

Leaving it as 'cert' should be sufficient.  The clientcert option is
really intended to be used with the *other* auth methods, to provide a
"check the password *and* the client certificate" option.

> I will try and drop the other connection lines in the pg_hba.conf but I 
> thought the last time I did that it through an error.  Let me try it again 
> and see if it works.

If that results in an error then it's possible you have an error in your
configuration and you aren't actually using SSL for this connection.

> So are you saying that "sslmode=verify-ca" doesn't actually verify that the 
> certificate is from a CA it trusts?  

verify-ca checks that the certificate is from a trusted CA, but it does
*not* make sure that the common name in that certificate matches the
name of the system that is being connected to.

In other words, if you configure your secondary to connect to
'server-a.mydomain.com', but the certificate installed on the server has
a common name (CN) of 'server-x.mydomain.com', the connection will still
be allowed, as long as the certificate is signed by a trusted CA.
That's not good.

> So my big question is:  am I really adding more security by turning on SSL or 
> is it just more work for the DBA?  

I'm not sure if this is a serious question or not.  If you want to
mitigate attack vectors which arise from data-in-transit then you
absolutely need to be using strong authentication and encryption when
connecting.  If your profile is such that you're not concerned about
network-based attack vectors, then perhaps it's not needed.

Thanks,

Stephen


signature.asc
Description: PGP signature

Re: SSL between Primary and Seconday PostgreSQL DBs

[Susan Joseph]

So I made the changes on the secondary to change the sslmode to verify-fullI 
removed the clientcert=1 in pg_hba.conf and removed any connections other than 
sslI removed the passfile info from recovery.confand now I am getting this 
error:
2020-09-03 13:01:49.990 UTC [7963] FATAL:  could not connect to the primary 
server: server certificate for "lc-subca-pg.theforest.sap" does not match host 
name "192.168.1.142"

My certificate for my primary is:
Version: 3
Serial: 0x1B
Issuer:  CN=LifeCycle SubCA, OU=CA, OU=DoDSAP, OU=DoD, O=U.S. Government, C=US
Subject: CN=lc-subca-pg.theforest.sap, OU=NPE, OU=DoDSAP, OU=DoD, O=U.S. 
Government, C=US
NotBefore: 20 AUG 2020 16:46:48 GMT
NotAfter: 05 AUG 2028 18:51:19 GMT
KeyType: RSA2048
Signature Algorithm: sha384RSA
Extensions: 
Enhanced Key Usage
    Server Authentication
    
    Client Authentication
Key Usage
(Critical)
    Digital signature
    Key encipherment
    (A0)
Subject Alternative Name
    DNS name:    lc-subca-pg.theforest.sap
    IP Address:  192.168.1.142
Subject Key Identifier
    FF4C0DCD62B17F99935DB3977D49711892958E20
Authority Key Identifier
    Key ID: 6495303FEB8925B9C83B7A63FF7F026C974E308E
SHA-1 Fingerprint: 51B1A879EB5ADAFA23042A8E84165CF89438F29B
 
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Susan Joseph 
To: sfr...@snowman.net 
Cc: pgsql-gene...@postgresql.org 
Sent: Thu, Sep 3, 2020 8:55 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

This is great, thanks.  I have not heard of verify-full but I will try that and 
let you know if I have issues.
So I will remove the passfile info for recovery.conf.  It was there because I 
first set it up using passwords to make sure it was working and then moved to 
SSL so if not needed I can remove it.
So if I don't put clientcert=1 do I just leave it at cert or should it be set 
to something else?
I will try and drop the other connection lines in the pg_hba.conf but I thought 
the last time I did that it through an error.  Let me try it again and see if 
it works.
So are you saying that "sslmode=verify-ca" doesn't actually verify that the 
certificate is from a CA it trusts?  

So my big question is:  am I really adding more security by turning on SSL or 
is it just more work for the DBA?  


 
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Stephen Frost 
To: Susan Joseph 
Cc: pgsql-gene...@postgresql.org 
Sent: Thu, Sep 3, 2020 8:28 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

Greetings,

* Susan Joseph (sandajos...@verizon.net) wrote:
>    - primary_conninfo = 'user=rep_user passfile=''/data/.pgpass'' 
>host= port=5432 sslmode=verify-ca sslcert=client.crt 
>sslkey=client.key sslcompression=0 target_session_attrs=any'

You really should be using sslmode=verify-full, otherwise any
certificate signed by a trusted CA on the server side is accepted.

Also, you shouldn't really need to pass in 'passfile' above...

>    - On the Secondary I edit the pg_hba.conf file and change the rep_user 
>line to:
>    
>    - hostssl  replication rep_user   /32  
>    cert clientcert=1

Saying clientcert=1 when using cert auth really shouldn't be needed.

> My questions are:  
>    - Do I need to set the information in the Secondary postgresql.conf?  
>Originally I did not set this and everything worked but I saw errors in my log 
>files that said to do SSL these needed to be set so I went back and set them.  
>Are there pgsql commands I can run to test that my SSL is working in both 
>directions?  

The only connection you're talking about here is from the secondary to
the primary and for that you just need the primary_conninfo settings for
the secondary set up correctly and the SSL settings on the primary.
However, if you want people to be able to make SSL connections to the
secondary, then you need to configure SSL on the secondary.

As for testing the connection, see the pg_stat_ssl view.

>    - Are my pg_hba.conf files set correctly?  Is that how you get SSL "turned 
>on" for communications between the primary and the rep_user account?

'hostssl' will only match an incoming connection if it's being made over
SSL.  As long as you don't have anything else in your pg_hba.conf, then
only SSL connections will be allowed.  However, first entry in the
pg_hba.conf matches, so if you have earlier entries, those might be
getting used instead.

>    - If I leave my key file encrypted then every time my databases have to be 
>started have to enter the password.  So you can either leave the passwords 
>unencrypted and set the permissions on the file to 0600 accessible only by 
>postgres or you can enter the key password each time the database is started 
>up.  As someone in the security field I have a tough time leaving the key 
>unencrypted but as some setting up a production system that is located on a 
>network that you can't get to without directly accessing the server I feel 
>that is enough security 

Re: SSL between Primary and Seconday PostgreSQL DBs

[Stephen Frost]

Greetings,

* Susan Joseph (sandajos...@verizon.net) wrote:
> So I made the changes on the secondary to change the sslmode to verify-fullI 
> removed the clientcert=1 in pg_hba.conf and removed any connections other 
> than sslI removed the passfile info from recovery.confand now I am getting 
> this error:
> 2020-09-03 13:01:49.990 UTC [7963] FATAL:  could not connect to the primary 
> server: server certificate for "lc-subca-pg.theforest.sap" does not match 
> host name "192.168.1.142"

Yes, as I explained, because of exactly the issue that the host you've
told your secondary to connect to (looks like 192.168.1.142) doesn't
match the certificate presented by the primary (which looks to be
"lc-subca-pg.theforest.sap").

The answer is to make those two match.

Thanks,

Stephen


signature.asc
Description: PGP signature

How to enumerate partitions from a window function?

[Alban Hertroys]

We are trying to add some information to a query over data from a
continuous process. Most of what we want can be done quite nicely using
window functions, but I got stuck on this particular problem:

The data has records with a timestamp and a few properties that make
subsequent rows belong to the same group. Say we have:
create table process_data (
timestamp timestamp not null,
property_A text not null,
property_B text not null,
value numeric(12, 3)
);

And a query like:
select datetime, property_A, property_B
, first_value(datetime)::time over run as swap_time
--, here I want to enumerate the runs themselves
, value
from process_data
window run as (partition by property_A, property_B order by datetime)
;

As stated above, I want to enumerate the runs, starting at 1 and
incrementing by 1 every time a partition from the 'run' window closes,
which would result in something like this:

datetime | property_A | property_B | swap_time | run_nr | value

2020-09-03 15:06 | tea | earl grey | 15:06 | 1 | 0.23
2020-09-03 15:07 | tea | earl grey | 15:06 | 1 | 0.22
2020-09-03 15:08 | tea | ceylon | 15:08 | 2 | 0.34
2020-09-03 15:09 | coffee | cappucino | 15:09 | 3 | 0.45
2020-09-03 15:10 | coffee | cappucino | 15:09 | 3 | 0.43
etc.

Is there a way to achieve this through window functions, or do we need to
wrap the thing in a subquery to achieve this?

Regards,
Alban Hertroys
-- 
If you can't see the forest for the trees,
Cut the trees and you'll see there is no forest.

Re: ODBC Driver dsplay name

[Siva postgres]

Hello,
Switching from SQL Server to PostgreSQL.
Installed Postgresql 11.8 successfully. 
Need to install ODBC to connect from front end.   When I install ODBC using 
stack builder , I see the ODBC name < in ODBC data source 
administrator>
PostgresSQL ODBC Driver(ANSI)PostgresSQL ODBC Driver(UNICODE)
When I download psqlodbc from 
https://www.postgresql.org/ftp/odbc/versions/msi/and install ODBC I get the 
following names
PostgreSQL ANSIPostgreSQL Unicode
1.  Is there any difference between two ?2.  Since I dynamically construct DSN 
from front end, I need to get the same name in all machines.   Is there any way 
to get the same name, < PostgreSQL ODBC Driver(ANSI) or PostgreSQL ODBC 
Driver(UNICODE), while installing from ODBC Driver downloaded from the above 
mentioned site ?
Happiness AlwaysBKR Sivaprakash

Default server PORT not updating in Pgadmin

[Rodrigo Martins]

Hello,



I’m creating one big installer for several software applications, including 
PostgreSQL and I’m facing the following issue:

If the default port of PostgreSQL is changed during installation the port of 
pgadmin’s default server is not updated (remains 5432).



[cid:c02b4b49-66ec-450a-b179-bd16d0a41e8a]

Even though it is easy to change the port using pgadmin, I would like to do it 
during installation (via scripting)

Is there a way to achieve this?



Btw, I can connect via psql command to the new port (in postgresql.conf the 
port is correct)



Thank you in advance!

Best regards

Re: SSL between Primary and Seconday PostgreSQL DBs

[Susan Joseph]

 
 
Susan Joseph
sandajos...@verizon.net
 
 
-Original Message-
From: Stephen Frost 
To: Susan Joseph 
Cc: pgsql-gene...@postgresql.org 
Sent: Thu, Sep 3, 2020 9:12 am
Subject: Re: SSL between Primary and Seconday PostgreSQL DBs

Greetings,

* Susan Joseph (sandajos...@verizon.net) wrote:
> So I made the changes on the secondary to change the sslmode to verify-fullI 
> removed the clientcert=1 in pg_hba.conf and removed any connections other 
> than sslI removed the passfile info from recovery.confand now I am getting 
> this error:
> 2020-09-03 13:01:49.990 UTC [7963] FATAL:  could not connect to the primary 
> server: server certificate for "lc-subca-pg.theforest.sap" does not match 
> host name "192.168.1.142"

>>Yes, as I explained, because of exactly the issue that the host you've
>>told your secondary to connect to (looks like 192.168.1.142) doesn't
>>match the certificate presented by the primary (which looks to be
>>"lc-subca-pg.theforest.sap").
OK so I fixed that in my recovery.conf file so it is not set to the IP but to 
the FQDN and it is no longer throwing this error.

>>The answer is to make those two match.

Thanks,

Stephen

Re: SSL between Primary and Seconday PostgreSQL DBs

[Peter Eisentraut]

On 2020-09-03 14:01, Susan Joseph wrote:
Unfortunately I am not allowed to use wireshark in my environment.  Good 
idea though


The system view pg_stat_ssl, in combination with pg_stat_activity, will 
show you whether a connection is using SSL.


--
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: How to enumerate partitions from a window function?

[Tom Lane]

Alban Hertroys  writes:
> As stated above, I want to enumerate the runs, starting at 1 and
> incrementing by 1 every time a partition from the 'run' window closes,
> Is there a way to achieve this through window functions, or do we need to
> wrap the thing in a subquery to achieve this?

I think this'll work:

select datetime, property_A, property_B
, first_value(datetime)::time over run as swap_time
, dense_rank() over (order by property_A, property_B)
, value
from process_data
window run as (partition by property_A, property_B order by datetime)
;

You can't do it with a window function over the "run" window because
no window function ever looks outside the current partition.  But
that's easy to fix by using a different window definition.  The
planner is smart enough to see that these windows are compatible
and only need one sort to be performed.

regards, tom lane

Re: Tuchanka

[Jehan-Guillaume de Rorthais]

On Wed, 2 Sep 2020 20:33:09 +0300
Олег Самойлов  wrote:

> The software is rather outdated. It works with PostgreSQL 11 and CentOS 7.
> The next step will be upgrading to CentOS 8 and PostgreSQL 12. Please tell
> me, is it useful and worth to continue? Where is better announce it? May be
> somewhere exists special mailing list for such things.

I answered on Pacemaker mailing list with a couple of links and questions
to discuss before answering if it useful to continue.

Thanks!

Re: How to enumerate partitions from a window function?

[Alban Hertroys]

On Thu, 3 Sep 2020 at 16:01, Tom Lane  wrote:

> Alban Hertroys  writes:
> > As stated above, I want to enumerate the runs, starting at 1 and
> > incrementing by 1 every time a partition from the 'run' window closes,
> > Is there a way to achieve this through window functions, or do we need to
> > wrap the thing in a subquery to achieve this?
>
> I think this'll work:
>
> select datetime, property_A, property_B
> , first_value(datetime)::time over run as swap_time
> , dense_rank() over (order by property_A, property_B)
> , value
> from process_data
> window run as (partition by property_A, property_B order by datetime)
> ;
>
> You can't do it with a window function over the "run" window because
> no window function ever looks outside the current partition.  But
> that's easy to fix by using a different window definition.  The
> planner is smart enough to see that these windows are compatible
> and only need one sort to be performed.
>
> regards, tom lane
>

Thanks Tom,

That gets us close, but it ignores the order of the runs over time. I think
it also reassigns the same number to later runs at the same 'day' that
happen to have the same values for property_A and _B. That's some crucial
information that I forgot to include.

To expand on my original example:

datetime | property_A | property_B | swap_time | run_nr | value

2020-09-03 15:06 | tea | earl grey | 15:06 | 1 | 0.23
2020-09-03 15:07 | tea | earl grey | 15:06 | 1 | 0.22
2020-09-03 15:08 | tea | ceylon | 15:08 | 2 | 0.34
2020-09-03 15:09 | coffee | cappucino | 15:09 | 3 | 0.45
2020-09-03 15:10 | coffee | cappucino | 15:09 | 3 | 0.43
2020-09-03 15:11 | tea | earl grey | 15:11 | 4 | 0.23
etc.

Where the last row has the same characteristic properties as the first 2
rows (from run 1), but is in run 4 due to it having started after run 3.

The runs normally start at 1 hour before midnight, with run 1, and continue
24h from there (it's a shifted day-schedule). The above example starting at
15:06 is unlikely to occur in reality, although possible (with long
downtime). That's mostly to clarify how the run numbers should function, it
would require to at least partition run_nr by a date shifted 1 hour back,
as long as they number their runs correctly along the time axis.

Regards,
Alban.
-- 
If you can't see the forest for the trees,
Cut the trees and you'll see there is no forest.

Dependency problem using community repo on Redhat 7

[Lawrence Layhee]

Hi,
We are having a dependency problem when using the community repo on redhat 7
When we install postgresql12-devel-12.4-1PGDG.rhel7.x86_64 we get the issue 
below.
Redhat doesn't support the dependencies. Any ideas? We are trying to get the 
ruby gem installed. It works with the Redhat dist of PostgreSQL but we prefer 
community.
Any help would be appreciated

Finished Dependency Resolution
Error: Package: postgresql12-devel-12.4-1PGDG.rhel7.x86_64 
(/postgresql12-devel-12.4-1PGDG.rhel7.x86_64)
   Requires: llvm-toolset-7-clang >= 4.0.1
Error: Package: llvm5.0-devel-5.0.1-7.el7.x86_64 (epel)
   Requires: libedit-devel
Error: Package: postgresql12-devel-12.4-1PGDG.rhel7.x86_64 
(/postgresql12-devel-12.4-1PGDG.rhel7.x86_64)
   Requires: libicu-devel



Thank you, Larry


Lawrence Layhee
Database Administration
Midrange Computing Division
Los Angeles County Internal Services Department
*  (562) 658-1701 - Office
Cell (562)-302-3080
* llay...@isd.lacounty.gov
P Please consider the environment before printing this e-mail and/or attachments

Re: How to enumerate partitions from a window function?

[Michael Lewis]

It seems like you are maybe wanting this- If the previous row is the same,
then get the previous row's run_nr. If it is different, then increment.

case when lag( property_A ) over() = property_A and lag( property_B )
over() = property_B then coalesce( lag( run_nr ) over(), 1 ) else lag(
run_nr ) over() + 1 end

Perhaps there is a much simpler implementation though.