Re: Waiting connections postgres 10

2018-07-23 Thread Michael Paquier
On Mon, Jul 23, 2018 at 04:41:59PM +, Nicola Contu wrote: > we used to monitor waiting connections with postgres 9.6.6 via this > query: > > select count (*) from pg_stat_activity where wait_event IS NOT NULL Please note that a wait event becomes NULL once it is reset, so what this query doe

Waiting connections postgres 10

2018-07-23 Thread Nicola Contu
Hello, we used to monitor waiting connections with postgres 9.6.6 via this query : select count (*) from pg_stat_activity where wait_event IS NOT NULL Now with postgres 10 it seems a bit harder to understand when a query from the application is waiting. I built this query but not sure I'm

Re: logical replication snapshots

2018-07-23 Thread Andres Freund
Hi, On 2018-07-23 15:23:04 -0500, Dimitri Maziuk wrote: > I'm playing with logical replication in postgres-10 and I got it to hang > on startup with > > > 2018-07-23 13:15:06.577 CDT [18624] FATAL: the database system is starting > > up > > 2018-07-23 13:15:06.577 CDT [18624] DEBUG: shmem_exit

RE: Improving pg_dump performance

2018-07-23 Thread Kevin Brannen
-Original Message- From: Adrian Klaver [mailto:adrian.kla...@aklaver.com] Sent: Monday, July 23, 2018 8:56 AM To: Ron Cc: pgsql-general@lists.postgresql.org Subject: Re: Improving pg_dump performance On 07/23/2018 06:47 AM, Ron wrote: > On 07/23/2018 08:46 AM, Stephen Frost wrote: >> Gree

logical replication snapshots

2018-07-23 Thread Dimitri Maziuk
Hi everyone, I'm playing with logical replication in postgres-10 and I got it to hang on startup with > 2018-07-23 13:15:06.577 CDT [18624] FATAL: the database system is starting up > 2018-07-23 13:15:06.577 CDT [18624] DEBUG: shmem_exit(1): 0 > before_shmem_exit callbacks to make > 2018-07-23

Re: Connections on cluster not being logged

2018-07-23 Thread Adrian Klaver
On 07/23/2018 12:50 PM, Sandy Becker wrote: Please reply to list also. Ccing list. Two servers set up in a hardware cluster for automatic failover.  That's all I know about it. Alright, so which server's logs are you looking at? Long term it would be a good thing to know how the cluster/failo

Re: Connections on cluster not being logged

2018-07-23 Thread Adrian Klaver
On 07/23/2018 08:14 AM, Sandy Becker wrote: I have postgresql 9.4 on a cluster, hardware based.  I need to be able to see which users are connecting to which database and when to be in compliance with our security policies. I have set the following in the postgresql.conf and did a pg_ctl reloa

Re: Improving pg_dump performance

2018-07-23 Thread Andres Freund
On 2018-07-23 09:17:41 -0500, Ron wrote: > On 07/23/2018 09:11 AM, Andres Freund wrote: > > Hi, > > > > On 2018-07-23 02:23:45 -0500, Ron wrote: > > > We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that > > > needs to be migrated to a new data center and then restored to v9.6.

Re: Connections on cluster not being logged

2018-07-23 Thread Sandy Becker
Actually, the last entry in the log file was when I changed the name so it was consistent with our other servers. I'm pretty new to postgresql, so I'm not really sure what I should be looking for. It looks like we're logging only statements where log_min_duration_statement = 1000. That's all I'm

Re: Connections on cluster not being logged

2018-07-23 Thread Andreas Kretschmer
Am 23.07.2018 um 17:25 schrieb Sandy Becker: Yes, they are in effect. strange. the logging is working? you can see other and actual entries in the logfile? Regards, Andreas -- 2ndQuadrant - The PostgreSQL Support Company. www.2ndQuadrant.com

Re: Connections on cluster not being logged

2018-07-23 Thread Sandy Becker
Yes, they are in effect. name |setting -+--- log_connections | on log_line_prefix | %t [%p]:[%u]:[%h]-[%d] [%1-1] Sandy On Mon, Jul 23, 2018 at 9:23 AM, Andreas Kretschmer wrote: > > > Am 23.07.2018 um 17:14 schrieb Sandy Be

Re: Connections on cluster not being logged

2018-07-23 Thread Andreas Kretschmer
Am 23.07.2018 um 17:14 schrieb Sandy Becker: I have postgresql 9.4 on a cluster, hardware based.  I need to be able to see which users are connecting to which database and when to be in compliance with our security policies. I have set the following in the postgresql.conf and did a pg_ctl r

Connections on cluster not being logged

2018-07-23 Thread Sandy Becker
I have postgresql 9.4 on a cluster, hardware based. I need to be able to see which users are connecting to which database and when to be in compliance with our security policies. I have set the following in the postgresql.conf and did a pg_ctl reload: log_connections = on log_line_pref

Re: Improving pg_dump performance

2018-07-23 Thread Ron
On 07/23/2018 09:11 AM, Andres Freund wrote: Hi, On 2018-07-23 02:23:45 -0500, Ron wrote: We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data center and then restored to v9.6.9. Have you considered using pg_upgrade instead? Yes, but: 1.

Re: Improving pg_dump performance

2018-07-23 Thread Ron
On 07/23/2018 08:56 AM, Adrian Klaver wrote: On 07/23/2018 06:47 AM, Ron wrote: On 07/23/2018 08:46 AM, Stephen Frost wrote: Greetings, * Ron (ronljohnso...@gmail.com) wrote: An interesting idea.  To clarify: it's possible to parallel backup a running 8.4 cluster remotely from a 9.6 syste

Replication protocol question for logical replication.

2018-07-23 Thread Chris Travers
While we are building a streaming logical backup solution for our data warehouse environment, we made a surprising discovery that select statements could be executed in a replication connection but I cannot find any clear documentation that says this is supported but I think it ought to be since th

Re: Improving pg_dump performance

2018-07-23 Thread Andres Freund
Hi, On 2018-07-23 02:23:45 -0500, Ron wrote: > We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that > needs to be migrated to a new data center and then restored to v9.6.9. Have you considered using pg_upgrade instead? Greetings, Andres Freund

Re: Improving pg_dump performance

2018-07-23 Thread Adrian Klaver
On 07/23/2018 06:47 AM, Ron wrote: On 07/23/2018 08:46 AM, Stephen Frost wrote: Greetings, * Ron (ronljohnso...@gmail.com) wrote: An interesting idea.  To clarify: it's possible to parallel backup a running 8.4 cluster remotely from a 9.6 system? Yes, you can do a parallel backup, but you wo

Re: Improving pg_dump performance

2018-07-23 Thread Ron
On 07/23/2018 08:46 AM, Stephen Frost wrote: Greetings, * Ron (ronljohnso...@gmail.com) wrote: An interesting idea.  To clarify: it's possible to parallel backup a running 8.4 cluster remotely from a 9.6 system? Yes, you can do a parallel backup, but you won't be able to get a consistent snaps

Re: Improving pg_dump performance

2018-07-23 Thread Stephen Frost
Greetings, * Ron (ronljohnso...@gmail.com) wrote: > An interesting idea.  To clarify: it's possible to parallel backup a running > 8.4 cluster remotely from a 9.6 system? Yes, you can do a parallel backup, but you won't be able to get a consistent snapshot. You'll need to pause all changes to th

Re: Improving pg_dump performance

2018-07-23 Thread Ron
On 07/23/2018 08:27 AM, Andreas Kretschmer wrote: Am 23.07.2018 um 15:06 schrieb Ron: On 07/23/2018 02:32 AM, Andreas Kretschmer wrote: Am 23.07.2018 um 09:23 schrieb Ron: Hi, We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data cente

Re: Improving pg_dump performance

2018-07-23 Thread Adrian Klaver
On 07/23/2018 12:23 AM, Ron wrote: Hi, We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data center and then restored to v9.6.9. The database has many large tables full of bytea columns containing pdf images, and so the dump file is going

Re: Improving pg_dump performance

2018-07-23 Thread Stephen Frost
Greetings, * Ron (ronljohnso...@gmail.com) wrote: > We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that > needs to be migrated to a new data center and then restored to v9.6.9. You should be using 9.6's pg_dump to perform the export. Might be a bit annoying to do, but you sh

Re: Improving pg_dump performance

2018-07-23 Thread Andreas Kretschmer
Am 23.07.2018 um 15:06 schrieb Ron: On 07/23/2018 02:32 AM, Andreas Kretschmer wrote: Am 23.07.2018 um 09:23 schrieb Ron: Hi, We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data center and then restored to v9.6.9. you can use th

Re: copyManager Implementation :: in postgreSQL community Driver

2018-07-23 Thread Adrian Klaver
On 07/23/2018 02:11 AM, Yogi S wrote: Hi All, there is a copyManager() implementation in community version JDBC driver. From what I understand from client end, we provide the file as a fileReader and pass it on to the copyManager() instance. I see the performance of this copyManager is not co

copyManager Implementation :: in postgreSQL community Driver

2018-07-23 Thread Yogi S
Hi All, there is a copyManager() implementation in community version JDBC driver. >From what I understand from client end, we provide the file as a fileReader and pass it on to the copyManager() instance. I see the performance of this copyManager is not comparable to PSQL command line copy. Hence

Re: Improving pg_dump performance

2018-07-23 Thread Ron
On 07/23/2018 02:32 AM, Andreas Kretschmer wrote: Am 23.07.2018 um 09:23 schrieb Ron: Hi, We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data center and then restored to v9.6.9. you can use the pg_dump from the newer version (9.6) to

Re: Improving pg_dump performance

2018-07-23 Thread Andreas Kretschmer
Am 23.07.2018 um 09:23 schrieb Ron: Hi, We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data center and then restored to v9.6.9. you can use the pg_dump from the newer version (9.6) to dump the old database, over the net. In this wa

Improving pg_dump performance

2018-07-23 Thread Ron
Hi, We've got an old (v8.4.17, thus no parallel backups) 2.9TB database that needs to be migrated to a new data center and then restored to v9.6.9. The database has many large tables full of bytea columns containing pdf images, and so the dump file is going to be more than 2x larger than the