Re: How is PG replication typically used to create a High Availability (HA) config ?
Hi David, please don't top-post. > On 11. Aug, 2020, at 22:57, David Gauthier wrote: > > Thanks for the response Paul :-) > > Our code is actually perl which uses DBI which has functions to ping a DB on > a specific server and connect to it. > But my question was more along the lines of whether or not the onus to do > this sort of thing typically lies with the app or something outside which is > orchestrating the HA cfg. it should be handled outside the app, im my opinion. But then, many installations don't use pg-bouncer, HA-proxy, virtual IP addresses or something like that. That's why I suggested using libpq. libpq can handle it. I'm not sure if and how it can in done in Perl, though. I played around a little with perl-DBI, perl-DBI-Pg, perl-URI and perl-URI-db and, though I managed to get connected, I did not manage to specifically select a connect to the primary or replica database cluster. Also, your initial steps should be done differently: 1. select count(*) from pg_stat_replication; => p 2. select count(*) from pg_stat_wal_receiver; => r if: p = 0 & r = 0 => single database cluster, no replication p > 0 & r = 0 => primary database cluster p = 0 & r > 0 => replica database cluster p > 0 & r > 0 => primary and replica database cluster The last case can for example happen, if you have database cluster A replicate to B, and B replicate to C, and then connect to B. Also, the test that many people do to select pg_is_in_recovery(); is not a good idea because B and C of the above example are probably in recovery mode, so you still don't know which end you're on. Also, pg_is_in_recovery() will probably not work with logical but only streaming replication (both async and sync) because I expect B and C to not be in recovery mode when using logical replication. I didn't try logical replication, so someone please correct me if I'm wrong here. If you just want to know, whether your connection is read-write or read-only, you can simply: show transaction_read_only; Cheers, Paul
Re: pglogical 2.3.2 on 9.4 --> 12.3 CONFLICT: remote DELETE (tuple not found). Resolution: skip.
replication identity correctly displayed on detail after resolution message: LOG: CONFLICT: remote DELETE on relation public.a_ replica identity index a_XXX_pk (tuple not found). Resolution: skip. DETAIL: remote tuple {created_by[int8]:(null) created_date[timestamp]:(null) updated_by[int8]:(null) updated_date[timestamp]:(null) pk_col_1[varchar]:ZZZ18 pk_col_2[bpchar]:12 pk_col_3[bpchar]:G << this is oldval status_data[varchar]:(null)} in xact origin=2,timestamp=2020-08-12 08:06:33.956683+08,commit_lsn=0/CA6499D0 I am curious about bpchar in log DETAILS. Is it the correct error message for the data type char? On Wed, Aug 12, 2020 at 10:50 AM milist ujang wrote: > Configured pglogical source is 9.4 target is 12.3 > > a table has PK from multiple column (varchar(50),char(2),char(1)). > > initial syncronization is OK, but when at source there is delete , but got > error at target... > > LOG: CONFLICT: remote DELETE on relation public.a_ replica identity > index a_XXX_pk (tuple not found). Resolution: skip. > > replica identity was set to default for the table so yes there is old > image on those 3 columns: > > but why delete was error at target? > collation exactly same on all GUC. > > any direction to print debug verbosity on why delete on target didnt find > row and skipped? > > > > -- > regards > > ujang jaenudin | DBA Consultant (Freelancer) > http://ora62.wordpress.com > http://id.linkedin.com/pub/ujang-jaenudin/12/64/bab > -- regards ujang jaenudin | DBA Consultant (Freelancer) http://ora62.wordpress.com http://id.linkedin.com/pub/ujang-jaenudin/12/64/bab
Postgres automatic minor version upgrade
Hello, We are using Postgres 11.6 through AWS relational database service. As part of its RDS service AWS offers automatic minor version upgrade. If we turn this setting on the minor versions will get upgraded without us even knowing about it. We are in a security sensitive vertical so we would like to ensure we don't miss out on critical security patches and bug fixes. We also use AWS Multi Availability Zones which I believe is built upon Postgres hot standby HA feature. We are not very clear on what are the possible risks we may be opening ourselves up to if we did automatic minor version. I would appreciate any suggestions on how to go about understanding the potential risks with using automatic minor version upgrade. Thanks, Shantanu
insert on conflict postgres returning distinction
is there any way to distinct between updated and inserted rows in RETURNING clause when ON CONFLICT UPDATE was used? -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Postgres automatic minor version upgrade
On 8/12/20 5:46 AM, Shantanu Shekhar wrote: Hello, We are using Postgres 11.6 through AWS relational database service. As part of its RDS service AWS offers automatic minor version upgrade. If we turn this setting on the minor versions will get upgraded without us even knowing about it. We are in a security sensitive vertical so we would like to ensure we don't miss out on critical security patches and bug fixes. We also use AWS Multi Availability Zones which I believe is built upon Postgres hot standby HA feature. We are not very clear on what are the possible risks we may be opening ourselves up to if we did automatic minor version. I would appreciate any suggestions on how to go about understanding the potential risks with using automatic minor version upgrade. https://aws.amazon.com/premiumsupport/ Thanks, Shantanu -- Adrian Klaver adrian.kla...@aklaver.com
Re: insert on conflict postgres returning distinction
On 8/12/20 7:23 AM, pinker wrote: is there any way to distinct between updated and inserted rows in RETURNING clause when ON CONFLICT UPDATE was used? Do you want to use that information immediately in the query or store it somewhere? If the first case I don't think that is possible. For the second then, I haven't tried it but, maybe a trigger using transition tables per example here: https://www.postgresql.org/docs/12/plpgsql-trigger.html Example 42.7. Auditing with Transition Tables -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html -- Adrian Klaver adrian.kla...@aklaver.com
Re: insert on conflict postgres returning distinction
thank you Adrian, the background of it is that I have already written the python script that translates Oracle MERGE clause to Postgres INSERT ... ON CONFLICT, but in order to be able to add DELETE part from MERGE i need to distinct those operations. thank you for the idea with trigger, i haven't thought about it at the beginning, but it does complicate the code a lot :/ not saying about performance... -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Implement a new data type
> On Aug 11, 2020, at 8:01 PM, raf wrote: > > On Tue, Aug 11, 2020 at 06:38:39AM -0700, Miles Elam > wrote: > >> Also of note: PostgreSQL already has a money type ( >> https://www.postgresql.org/docs/current/datatype-money.html) >> But you shouldn't use it ( >> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money). >> >> I only bring it up so that you can know to make your money type a slightly >> different name to avoid a conflict. Money is deceptively hard to implement >> correctly. I'd recommend reading the second link if you have not already to >> avoid previously known issues. > > I use decimal(10,2) for whole cents, and decimal(12,6) > for sub-cents. Single currency only. I didn't know > there was a money type originally, but it wouldn't be > usable for me anyway without the ability to specify the > scale and precision. > > I recommend considering passing values to the database > as "decimal '1.23'" rather than bare numeric literals, > just so there's no chance of the value being > interpreted as a float at any stage by postgres. Maybe > that's being too paranoid but that's a good idea when > it comes to money. :-) Yes, I agree, this is also important (and easy to overlook) if you’re accessing the database via a non-SQL language. We use Python which, like most (all?) languages that rely on the underlying C library for floating point support, is vulnerable to floating point noise. Python has a fixed precision type, and like Postgres it also accepts character and float input. The float input can give surprising results. >>> decimal.Decimal('1.79') # This is OK Decimal('1.79') >>> decimal.Decimal(1.79) # This will not end well! Decimal('1.79003552713678800500929355621337890625') >>> In the case of a Postgres column like numeric(10,2), input like 1.79003552713678800500929355621337890625 will get rounded to 1.79 anyway and no harm will be done. But like you said, raf, it’s a good idea to be too paranoid. :-) Cheers Philip
Re: Sizing PostgreSQL VM server sizing
On Tue, Aug 11, 2020 at 10:19 AM Samarendra Sahoo < sahoo.samaren...@gmail.com> wrote: > Dear all, > We are getting ready to install in production and would like to know what > are key considerations and how do we use them to provision VMs for the same? > It is going to be highly dependent on how the database is used. If you have a terabyte of data, but most of it is (almost) never accessed and you have just one process inserting data, vs wanting to support thousands of concurrent users with a mixture of small transaction queries and also huge reporting queries off the same database... You might need 4 core and 16GB of ram, or you might need 48 cores and 512GB of ram with very fast storage. There are not hard and fast rules, or even clear guidelines that I am aware of. If you expect to deal with many users, you will likely want a connection pooler, whether bundled with your software that accesses the database, or something like pgpool or pg bouncer.
Re: insert on conflict postgres returning distinction
how about this solution? Does it have any caveats? WITH upsert AS (INSERT INTO GUCIO (ID, NAZWA) SELECT A.ID, A.NAZWA FROM ALA A ON CONFLICT (ID) DO UPDATE SET nazwa = excluded.nazwa RETURNING xmax,xmin, *) select xmax as xmax_value into txmaxu from upsert; delete from gucio where xmax = (select xmax_value from txmaxu where xmax_value <> 0 limit 1); -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: insert on conflict postgres returning distinction
Od course inside transaction block -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Server goes to Recovery Mode when run a SQL
Hi Michael, so sorry for the delay. I did never had that problem again, so tougth it was solved, but seems not. We have two different SQL which pushs server to recovery mode. This SQL is the hardest one, because you´ll have to wait some time to get the error. Just runs this SQL, opens your HTOP and wait, you´ll see your memory growing and growing until ... PostgreSQL 11.5 (Ubuntu 11.5-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit But same problem occurs on version 10. with feriados as ( SELECT dia, repete FROM (VALUES ('2014-10-11'::DATE, FALSE), ('2014-10-13', FALSE),('2014-10-14', FALSE), ('2014-10-15', FALSE), ('2014-10-16', FALSE), ('2014-10-17', FALSE), ('2014-12-19', FALSE), ('2014-12-20',FALSE), ('2014-12-22', FALSE), ('2014-12-23', FALSE), ('2014-12-24', FALSE), ('2014-12-26', FALSE), ('2014-12-27',FALSE), ('2014-12-29', FALSE), ('2014-12-30', FALSE), ('2014-12-31', FALSE), ('2015-01-02', FALSE), ('2015-01-03',FALSE), ('2015-01-04', FALSE), ('2015-02-16', FALSE), ('2015-02-17', FALSE), ('2015-04-03', FALSE), ('2015-04-04',FALSE), ('2015-06-04', FALSE), ('2015-12-18', FALSE), ('2015-12-19', FALSE), ('2015-12-21', FALSE), ('2015-12-22',FALSE), ('2015-12-23', FALSE), ('2015-12-24', FALSE), ('2015-12-26', FALSE), ('2015-12-28', FALSE), ('2015-12-29',FALSE), ('2015-12-30', FALSE), ('2015-12-31', FALSE), ('2016-01-02', FALSE), ('2016-01-04', FALSE), ('2016-01-05',FALSE), ('2016-01-06', FALSE), ('2016-01-07', FALSE), ('2016-01-08', FALSE), ('2016-01-09', FALSE), ('2016-02-08',FALSE), ('2016-03-09', FALSE), ('2016-03-25', FALSE), ('2016-03-26', FALSE), ('2016-05-26', FALSE), ('2016-12-24',FALSE), ('2016-12-28', FALSE), ('2016-12-29', FALSE), ('2016-12-30', FALSE), ('2016-12-31', FALSE), ('2017-01-02',FALSE), ('2017-01-03', FALSE), ('2017-01-04', FALSE), ('2017-01-05', FALSE), ('2017-01-06', FALSE), ('2017-01-07',FALSE), ('2017-02-25', FALSE), ('2017-02-27', FALSE), ('2017-03-09', FALSE), ('2017-04-14', FALSE), ('2017-04-15',FALSE), ('2017-06-15', FALSE), ('2017-09-30', FALSE), ('2017-12-18', FALSE), ('2017-12-19', FALSE), ('2017-12-20',FALSE), ('2017-12-21', FALSE), ('2017-12-22', FALSE), ('2017-12-23', FALSE), ('2017-12-26', FALSE), ('2017-12-27',FALSE), ('2017-12-28', FALSE), ('2017-12-29', FALSE), ('2017-12-30', FALSE), ('2018-01-02', FALSE), ('2018-01-03',FALSE), ('2018-01-04', FALSE), ('2018-01-05', FALSE), ('2018-01-06', FALSE), ('2018-01-07', FALSE), ('2018-02-12',FALSE), ('2018-02-13', FALSE), ('2018-03-30', FALSE), ('2018-03-31', FALSE), ('2018-04-30', FALSE), ('2018-05-31',FALSE), ('2018-10-15', FALSE), ('2018-12-18', FALSE), ('2018-12-19', FALSE), ('2018-12-20', FALSE), ('2018-12-21',FALSE), ('2018-12-22', FALSE), ('2018-12-24', FALSE), ('2018-12-26', FALSE), ('2018-12-27', FALSE), ('2018-12-28',FALSE), ('2018-12-29', FALSE), ('2018-12-31', FALSE), ('2019-01-01', TRUE), ('2019-01-02', FALSE), ('2019-01-03',FALSE), ('2019-01-04', FALSE), ('2019-01-05', FALSE), ('2019-01-07', FALSE), ('2019-02-09', TRUE), ('2019-03-09',TRUE), ('2019-04-21', TRUE), ('2019-05-01', TRUE), ('2019-09-07', TRUE), ('2019-10-12', TRUE), ('2019-11-02', TRUE),('2019-11-15', TRUE), ('2019-12-19', TRUE), ('2019-12-20', TRUE), ('2019-12-21', TRUE), ('2019-12-22', TRUE),('2019-12-23', TRUE), ('2019-12-25', TRUE), ('2019-12-26', TRUE), ('2019-12-27', TRUE)) x (dia, repete) ), materias as ( SELECT * from (VALUES (593, 11091, 'AAC - Ética e Segurança Digital', 9, 120, '2019/01/30'::DATE, '2019/01/30'::DATE, 60, 120, 0), (593, 11085, 'AAC - Computação nas Nuvens', 12, 60, NULL, NULL, 60, 120, 60) ) x (turma_id, materia_id, materia, sequencia, previsto, dataini, datafim, tempoatividade, minutosaula, minutosrestantes) ), col_diasaula(turma_id, dia, tempoaula) as(select * from (values(593, 3, time '02:00')) as x(turma_id, dia, tempoaula) ), aulasporsemana as ( select * from (values (593,1)) x (turma_id, quantidade) ), assistidas (turma_id, sequencia, conteudo_id, conteudo, prevista, aularealizada, tempoatividade, aulasNoDia, dia, minutoaula, minutosassistidos, cargaconteudo, cargarestante) as ( SELECT materias.turma_id, materias.sequencia, materias.materia_id, materias.materia, coalesce(realizada.prevista, 1), realizada.aularealizada, materias.tempoatividade, (realizada.minutosassistidos / materias.tempoatividade), realizada.dia, materias.minutosaula, realizada.minutosassistidos, materias.previsto, coalesce(materias.previsto - (row_number() OVER AulaDaMateria * realizada.minutosassistidos), materias.previsto) FROM materias LEFT JOIN LATERAL ( SELECT TRUE, tsrange(col_aula.data, (col_aula.data + ( col_aula.tempo ||' minute')::INTERVAL)) dia, 0 prevista, (extract(EPOCH FROM col_aula.tempo) / 60) minutosassistidos FROM (VALUES (593, 11091, TIMESTAMP '2019-01-30 19:00:00', TIME '02:00'), (593, 11091, '2019-02
psql and jdbc default sslmode
Hi , We are making connections to the postgres server through jdbc and psql (libpq) . I have set the ssl as on the postgres server . It can take ssl as well as non ssl connections. I made a connection through a psql client to postgres server and could confirm that the default sslmode (when no sslmode parameter is supplied while making connection) is "prefer". Please note i have not supplied the sslmode parameter in the connection string from psql. Still connection is secured psql "postgresql://$POSTGRES_HOST:$PG_PORT/postgres" -U postgres psql (9.6.18, server 9.6.12) SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type "help" for help. I have read in the AWS documentation for jdbc connections to server the default mode is "verify-full". I created a jdbc connection to the postgres server by supplying no sslmode to the connection string . The connection was successful . I just want to confirm what is the default sslmode for jdbc connections to the postgres server when ssl is turned on the server. Regards, Shankar
Re: Server goes to Recovery Mode when run a SQL
PegoraroF10 writes: > Hi Michael, so sorry for the delay. I did never had that problem again, so > tougth it was solved, but seems not. We have two different SQL which pushs > server to recovery mode. This SQL is the hardest one, because you´ll have to > wait some time to get the error. Just runs this SQL, opens your HTOP and > wait, you´ll see your memory growing and growing until ... Are you sure this query ever terminates? It looks to me like it just keeps producing tuples indefinitely until you kill it or something runs out of memory. If the response to "out of memory" is a server crash rather than something nicer, you probably need to corral your OOM killer. See https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT regards, tom lane
Re: Implement a new data type
On Wed, Aug 12, 2020 at 12:44:21PM -0400, Philip Semanchuk wrote: > > On Aug 11, 2020, at 8:01 PM, raf wrote: > > > > On Tue, Aug 11, 2020 at 06:38:39AM -0700, Miles Elam > > wrote: > > > >> Also of note: PostgreSQL already has a money type ( > >> https://www.postgresql.org/docs/current/datatype-money.html) > >> But you shouldn't use it ( > >> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money). > >> > >> I only bring it up so that you can know to make your money type a slightly > >> different name to avoid a conflict. Money is deceptively hard to implement > >> correctly. I'd recommend reading the second link if you have not already to > >> avoid previously known issues. > > > > I use decimal(10,2) for whole cents, and decimal(12,6) > > for sub-cents. Single currency only. I didn't know > > there was a money type originally, but it wouldn't be > > usable for me anyway without the ability to specify the > > scale and precision. > > > > I recommend considering passing values to the database > > as "decimal '1.23'" rather than bare numeric literals, > > just so there's no chance of the value being > > interpreted as a float at any stage by postgres. Maybe > > that's being too paranoid but that's a good idea when > > it comes to money. :-) > > Yes, I agree, this is also important (and easy to overlook) if > you’re accessing the database via a non-SQL language. We use Python > which, like most (all?) languages that rely on the underlying C > library for floating point support, is vulnerable to floating point > noise. Python has a fixed precision type, and like Postgres it also > accepts character and float input. The float input can give surprising > results. > > >>> decimal.Decimal('1.79') # This is OK > Decimal('1.79') > >>> decimal.Decimal(1.79) # This will not end well! > Decimal('1.79003552713678800500929355621337890625') > >>> > > In the case of a Postgres column like numeric(10,2), input like > 1.79003552713678800500929355621337890625 will get rounded > to 1.79 anyway and no harm will be done. But like you said, raf, it’s > a good idea to be too paranoid. :-) > > Cheers > Philip Chris Travers pointed out to me that Postgres itself parses floating point literals as the numeric type (i.e. "select pg_typeof(1.23);" returns numeric) so Postgres has made the right choice for its parser, unlike most(?) languages (except raku). But yes, in Python, it's decimal.Decimal with integer/string input all the way (with dec=decimal.Decimal if you have a lot of them). cheers, raf
Re: How is PG replication typically used to create a High Availability (HA) config ?
Thanks again Paul and Rob. I'm going to need more specifics from my IT department regarding exactly what they did... what tool they used to create what they are calling this "High Availability" DB (pg-bouncer, etc...). If I can determine that, then maybe there are already some hooks in place that I can leverage. But for this to be seamless, I suspect I'll also have to do something on the app end beyond making a hard connection from my perl script using DBI. I did find something about an enhanced version of the DBI connect method which might redirect connections in the event of a primary/backup swap. -dave On Wed, Aug 12, 2020 at 4:10 AM Paul Förster wrote: > Hi David, > > please don't top-post. > > > On 11. Aug, 2020, at 22:57, David Gauthier > wrote: > > > > Thanks for the response Paul :-) > > > > Our code is actually perl which uses DBI which has functions to ping a > DB on a specific server and connect to it. > > But my question was more along the lines of whether or not the onus to > do this sort of thing typically lies with the app or something outside > which is orchestrating the HA cfg. > > it should be handled outside the app, im my opinion. But then, many > installations don't use pg-bouncer, HA-proxy, virtual IP addresses or > something like that. That's why I suggested using libpq. libpq can handle > it. I'm not sure if and how it can in done in Perl, though. > > I played around a little with perl-DBI, perl-DBI-Pg, perl-URI and > perl-URI-db and, though I managed to get connected, I did not manage to > specifically select a connect to the primary or replica database cluster. > > Also, your initial steps should be done differently: > > 1. select count(*) from pg_stat_replication; => p > 2. select count(*) from pg_stat_wal_receiver; => r > > if: > > p = 0 & r = 0 => single database cluster, no replication > p > 0 & r = 0 => primary database cluster > p = 0 & r > 0 => replica database cluster > p > 0 & r > 0 => primary and replica database cluster > > The last case can for example happen, if you have database cluster A > replicate to B, and B replicate to C, and then connect to B. > > Also, the test that many people do to select pg_is_in_recovery(); is not a > good idea because B and C of the above example are probably in recovery > mode, so you still don't know which end you're on. > > Also, pg_is_in_recovery() will probably not work with logical but only > streaming replication (both async and sync) because I expect B and C to not > be in recovery mode when using logical replication. I didn't try logical > replication, so someone please correct me if I'm wrong here. > > If you just want to know, whether your connection is read-write or > read-only, you can simply: > > show transaction_read_only; > > Cheers, > Paul
hash joins are causing no space left error
This is PostgreSQL 11 on AWS, there is a mview query in this OLAP database, the tables involved are huge - 50-100m records on average records hundreds of columns in most cases. The query runs for a while and then errors out saying "No space left on device". I could see it generating around 500gb of temp file data. At times it goes thru and at times it fails - probably due to other queries running at the same time and causing failure. The tables are partitioned and indexed on the PKs and FKs. Using parallelism (4) with increased work_mem (4gb). The joins are happening on around 10 tables and all are joining on the PK and FK columns. I see partition pruning happening but the hash joins are killing the query. Is there any way to avoid hash joins? If we create hash indexes on the joining columns, would PostgreSQL avoid hashing operation and instead use hash indexes on the tables and join them. That way I feel resource intensive hashing would be avoided and there wont be any need of temp files. I tried but does not seem to work, when I query the table with specific values then it uses the hash index but when I am joining the tables it seems to do its own hash join. My question is how to optimize massive table joins in PostgreSQL to resolve - avoid space failures and make it run fast - takes a couple of hours to complete now. Any best practices or suggestions.
Re: hash joins are causing no space left error
Ayub M writes: > This is PostgreSQL 11 on AWS, there is a mview query in this OLAP database, > the tables involved are huge - 50-100m records on average records hundreds > of columns in most cases. The query runs for a while and then errors out > saying "No space left on device". I could see it generating around 500gb of > temp file data. At times it goes thru and at times it fails - probably due > to other queries running at the same time and causing failure. Are you sure that these queries are actually producing the answers you want? It sounds suspiciously like you are computing underconstrained joins. > The joins are happening on around 10 tables and all are joining on the PK > and FK columns. I see partition pruning happening but the hash joins are > killing the query. > Is there any way to avoid hash joins? TBH, you are asking the wrong question. A merge join would take about as much temporary space, and a nestloop join over so much data would probably not finish in an amount of time you're willing to wait. Indexes are NOT a magic solution here. What you need to be thinking about is how to not need to process so much data. If you really need to have this proven to you, you can try "set enable_hashjoin = off", but I don't think you'll find that better. regards, tom lane