Re: How is PG replication typically used to create a High Availability (HA) config ?

2020-08-12 Thread Paul Förster
Hi David,

please don't top-post.

> On 11. Aug, 2020, at 22:57, David Gauthier  wrote:
> 
> Thanks for the response Paul :-)
> 
> Our code is actually perl which uses DBI which has functions to ping a DB on 
> a specific server and connect to it.
> But my question was more along the lines  of whether or not the onus to do 
> this sort of thing typically lies with the app or something outside which is 
> orchestrating the HA cfg.

it should be handled outside the app, im my opinion. But then, many 
installations don't use pg-bouncer, HA-proxy, virtual IP addresses or something 
like that. That's why I suggested using libpq. libpq can handle it. I'm not 
sure if and how it can in done in Perl, though.

I played around a little with perl-DBI, perl-DBI-Pg, perl-URI and perl-URI-db 
and, though I managed to get connected, I did not manage to specifically select 
a connect to the primary or replica database cluster.

Also, your initial steps should be done differently:

1. select count(*) from pg_stat_replication; => p
2. select count(*) from pg_stat_wal_receiver; => r

if:

p = 0 & r = 0 => single database cluster, no replication
p > 0 & r = 0 => primary database cluster
p = 0 & r > 0 => replica database cluster
p > 0 & r > 0 => primary and replica database cluster

The last case can for example happen, if you have database cluster A replicate 
to B, and B replicate to C, and then connect to B.

Also, the test that many people do to select pg_is_in_recovery(); is not a good 
idea because B and C of the above example are probably in recovery mode, so you 
still don't know which end you're on.

Also, pg_is_in_recovery() will probably not work with logical but only 
streaming replication (both async and sync) because I expect B and C to not be 
in recovery mode when using logical replication. I didn't try logical 
replication, so someone please correct me if I'm wrong here.

If you just want to know, whether your connection is read-write or read-only, 
you can simply:

show transaction_read_only;

Cheers,
Paul



Re: pglogical 2.3.2 on 9.4 --> 12.3 CONFLICT: remote DELETE (tuple not found). Resolution: skip.

2020-08-12 Thread milist ujang
replication identity correctly displayed on detail after resolution message:

  LOG:  CONFLICT: remote DELETE on relation public.a_ replica identity
index a_XXX_pk (tuple not found). Resolution: skip.

DETAIL:  remote tuple {created_by[int8]:(null)
created_date[timestamp]:(null) updated_by[int8]:(null)
updated_date[timestamp]:(null)
pk_col_1[varchar]:ZZZ18 pk_col_2[bpchar]:12 pk_col_3[bpchar]:G   <<
this is oldval
status_data[varchar]:(null)} in xact origin=2,timestamp=2020-08-12
08:06:33.956683+08,commit_lsn=0/CA6499D0

I am curious about  bpchar in log DETAILS.
Is it the correct error message for the data type char?



On Wed, Aug 12, 2020 at 10:50 AM milist ujang 
wrote:

> Configured pglogical source is 9.4 target is 12.3
>
> a table has PK from multiple column (varchar(50),char(2),char(1)).
>
> initial syncronization is OK, but when at source there is delete , but got
> error at target...
>
> LOG:  CONFLICT: remote DELETE on relation public.a_ replica identity
> index a_XXX_pk (tuple not found). Resolution: skip.
>
> replica identity was set to default for the table so yes there is old
> image on those 3 columns:
>
> but why delete was error at target?
> collation exactly same on all GUC.
>
> any direction to print debug verbosity on why delete on target didnt find
> row and skipped?
>
>
>
> --
> regards
>
> ujang jaenudin | DBA Consultant (Freelancer)
> http://ora62.wordpress.com
> http://id.linkedin.com/pub/ujang-jaenudin/12/64/bab
>


-- 
regards

ujang jaenudin | DBA Consultant (Freelancer)
http://ora62.wordpress.com
http://id.linkedin.com/pub/ujang-jaenudin/12/64/bab


Postgres automatic minor version upgrade

2020-08-12 Thread Shantanu Shekhar
Hello,
We are using Postgres 11.6 through AWS relational database service. As part of 
its RDS service AWS offers automatic minor version upgrade. If we turn this 
setting on the minor versions will get upgraded without us even knowing about 
it. We are in a security sensitive vertical so we would like to ensure we don't 
miss out on critical security patches and bug fixes. We also use AWS Multi 
Availability Zones which I believe is built upon Postgres hot standby HA 
feature.
We are not very clear on what are the possible risks we may be opening 
ourselves up to if we did automatic minor version. I would appreciate any 
suggestions on how to go about  understanding the potential risks with using 
automatic minor version upgrade.
Thanks,
Shantanu

insert on conflict postgres returning distinction

2020-08-12 Thread pinker
is there any way to distinct between updated and inserted rows in RETURNING
clause when ON CONFLICT UPDATE was used?



--
Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html




Re: Postgres automatic minor version upgrade

2020-08-12 Thread Adrian Klaver

On 8/12/20 5:46 AM, Shantanu Shekhar wrote:

Hello,

We are using Postgres 11.6 through AWS relational database service. As 
part of its RDS service AWS offers automatic minor version upgrade. If 
we turn this setting on the minor versions will get upgraded without us 
even knowing about it. We are in a security sensitive vertical so we 
would like to ensure we don't miss out on critical security patches and 
bug fixes. We also use AWS Multi Availability Zones which I believe is 
built upon Postgres hot standby HA feature.


We are not very clear on what are the possible risks we may be opening 
ourselves up to if we did automatic minor version. I would appreciate 
any suggestions on how to go about  understanding the potential risks 
with using automatic minor version upgrade.


https://aws.amazon.com/premiumsupport/


Thanks,

Shantanu



--
Adrian Klaver
adrian.kla...@aklaver.com




Re: insert on conflict postgres returning distinction

2020-08-12 Thread Adrian Klaver

On 8/12/20 7:23 AM, pinker wrote:

is there any way to distinct between updated and inserted rows in RETURNING
clause when ON CONFLICT UPDATE was used?


Do you want to use that information immediately in the query or store it 
somewhere? If the first case I don't think that is possible. For the 
second then, I haven't tried it but,  maybe a trigger using transition 
tables per example here:


https://www.postgresql.org/docs/12/plpgsql-trigger.html

Example 42.7. Auditing with Transition Tables





--
Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html





--
Adrian Klaver
adrian.kla...@aklaver.com




Re: insert on conflict postgres returning distinction

2020-08-12 Thread pinker
thank you Adrian,
the background of it is that I have already written the python script that
translates Oracle MERGE clause to Postgres INSERT ... ON CONFLICT, but in
order to be able to add DELETE part from MERGE i need to distinct those
operations. 

thank you for the idea with trigger, i haven't thought about it at the
beginning, but it does complicate the code a lot :/ not saying about
performance... 



--
Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html




Re: Implement a new data type

2020-08-12 Thread Philip Semanchuk



> On Aug 11, 2020, at 8:01 PM, raf  wrote:
> 
> On Tue, Aug 11, 2020 at 06:38:39AM -0700, Miles Elam 
>  wrote:
> 
>> Also of note: PostgreSQL already has a money type (
>> https://www.postgresql.org/docs/current/datatype-money.html)
>> But you shouldn't use it (
>> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money).
>> 
>> I only bring it up so that you can know to make your money type a slightly
>> different name to avoid a conflict. Money is deceptively hard to implement
>> correctly. I'd recommend reading the second link if you have not already to
>> avoid previously known issues.
> 
> I use decimal(10,2) for whole cents, and decimal(12,6)
> for sub-cents. Single currency only. I didn't know
> there was a money type originally, but it wouldn't be
> usable for me anyway without the ability to specify the
> scale and precision.
> 
> I recommend considering passing values to the database
> as "decimal '1.23'" rather than bare numeric literals,
> just so there's no chance of the value being
> interpreted as a float at any stage by postgres. Maybe
> that's being too paranoid but that's a good idea when
> it comes to money. :-)

Yes, I agree, this is also important (and easy to overlook) if you’re accessing 
the database via a non-SQL language. We use Python which, like most (all?) 
languages that rely on the underlying C library for floating point support, is 
vulnerable to floating point noise. Python has a fixed precision type, and like 
Postgres it also accepts character and float input. The float input can give 
surprising results.

>>> decimal.Decimal('1.79')   # This is OK
Decimal('1.79')
>>> decimal.Decimal(1.79) # This will not end well!
Decimal('1.79003552713678800500929355621337890625')
>>>

In the case of a Postgres column like numeric(10,2), input like 
1.79003552713678800500929355621337890625 will get rounded to 1.79 
anyway and no harm will be done. But like you said, raf, it’s a good idea to be 
too paranoid. :-)

Cheers
Philip



Re: Sizing PostgreSQL VM server sizing

2020-08-12 Thread Michael Lewis
On Tue, Aug 11, 2020 at 10:19 AM Samarendra Sahoo <
sahoo.samaren...@gmail.com> wrote:

> Dear all,
> We are getting ready to install in production and would like to know what
> are key considerations and how do we use them to provision VMs for the same?
>

It is going to be highly dependent on how the database is used. If you have
a terabyte of data, but most of it is (almost) never accessed and you have
just one process inserting data, vs wanting to support thousands of
concurrent users with a mixture of small transaction queries and also huge
reporting queries off the same database... You might need 4 core and 16GB
of ram, or you might need 48 cores and 512GB of ram with very fast storage.
There are not hard and fast rules, or even clear guidelines that I am aware
of. If you expect to deal with many users, you will likely want a
connection pooler, whether bundled with your software that accesses the
database, or something like pgpool or pg bouncer.


Re: insert on conflict postgres returning distinction

2020-08-12 Thread pinker
how about this solution?
Does it have any caveats?

WITH upsert AS (INSERT INTO GUCIO (ID, NAZWA)
SELECT A.ID, A.NAZWA
FROM ALA A
ON CONFLICT (ID) DO UPDATE SET
nazwa = excluded.nazwa
RETURNING xmax,xmin, *)
select xmax as xmax_value
into txmaxu
from upsert;

delete
from gucio
where xmax = (select xmax_value from txmaxu where xmax_value <> 0 limit 1);



--
Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html




Re: insert on conflict postgres returning distinction

2020-08-12 Thread pinker
Od course inside transaction block



--
Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html




Re: Server goes to Recovery Mode when run a SQL

2020-08-12 Thread PegoraroF10
Hi Michael, so sorry for the delay. I did never had that problem again, so
tougth it was solved, but seems not. We have two different SQL which pushs
server to recovery mode. This SQL is the hardest one, because you´ll have to
wait some time to get the error. Just runs this SQL, opens your HTOP and
wait, you´ll see your memory growing and growing until ... 

PostgreSQL 11.5 (Ubuntu 11.5-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled
by gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0, 64-bit

But same problem occurs on version 10.

with feriados as (
  SELECT dia, repete
  FROM (VALUES ('2014-10-11'::DATE, FALSE), ('2014-10-13',
FALSE),('2014-10-14', FALSE), ('2014-10-15', FALSE), ('2014-10-16', FALSE),
   ('2014-10-17', FALSE), ('2014-12-19', FALSE), ('2014-12-20',FALSE),
('2014-12-22', FALSE), ('2014-12-23', FALSE),
   ('2014-12-24', FALSE), ('2014-12-26', FALSE), ('2014-12-27',FALSE),
('2014-12-29', FALSE), ('2014-12-30', FALSE),
   ('2014-12-31', FALSE), ('2015-01-02', FALSE), ('2015-01-03',FALSE),
('2015-01-04', FALSE), ('2015-02-16', FALSE),
   ('2015-02-17', FALSE), ('2015-04-03', FALSE), ('2015-04-04',FALSE),
('2015-06-04', FALSE), ('2015-12-18', FALSE),
   ('2015-12-19', FALSE), ('2015-12-21', FALSE), ('2015-12-22',FALSE),
('2015-12-23', FALSE), ('2015-12-24', FALSE),
   ('2015-12-26', FALSE), ('2015-12-28', FALSE), ('2015-12-29',FALSE),
('2015-12-30', FALSE), ('2015-12-31', FALSE),
   ('2016-01-02', FALSE), ('2016-01-04', FALSE), ('2016-01-05',FALSE),
('2016-01-06', FALSE), ('2016-01-07', FALSE),
   ('2016-01-08', FALSE), ('2016-01-09', FALSE), ('2016-02-08',FALSE),
('2016-03-09', FALSE), ('2016-03-25', FALSE),
   ('2016-03-26', FALSE), ('2016-05-26', FALSE), ('2016-12-24',FALSE),
('2016-12-28', FALSE), ('2016-12-29', FALSE),
   ('2016-12-30', FALSE), ('2016-12-31', FALSE), ('2017-01-02',FALSE),
('2017-01-03', FALSE), ('2017-01-04', FALSE),
   ('2017-01-05', FALSE), ('2017-01-06', FALSE), ('2017-01-07',FALSE),
('2017-02-25', FALSE), ('2017-02-27', FALSE),
   ('2017-03-09', FALSE), ('2017-04-14', FALSE), ('2017-04-15',FALSE),
('2017-06-15', FALSE), ('2017-09-30', FALSE),
   ('2017-12-18', FALSE), ('2017-12-19', FALSE), ('2017-12-20',FALSE),
('2017-12-21', FALSE), ('2017-12-22', FALSE),
   ('2017-12-23', FALSE), ('2017-12-26', FALSE), ('2017-12-27',FALSE),
('2017-12-28', FALSE), ('2017-12-29', FALSE),
   ('2017-12-30', FALSE), ('2018-01-02', FALSE), ('2018-01-03',FALSE),
('2018-01-04', FALSE), ('2018-01-05', FALSE),
   ('2018-01-06', FALSE), ('2018-01-07', FALSE), ('2018-02-12',FALSE),
('2018-02-13', FALSE), ('2018-03-30', FALSE),
   ('2018-03-31', FALSE), ('2018-04-30', FALSE), ('2018-05-31',FALSE),
('2018-10-15', FALSE), ('2018-12-18', FALSE),
   ('2018-12-19', FALSE), ('2018-12-20', FALSE), ('2018-12-21',FALSE),
('2018-12-22', FALSE), ('2018-12-24', FALSE),
   ('2018-12-26', FALSE), ('2018-12-27', FALSE), ('2018-12-28',FALSE),
('2018-12-29', FALSE), ('2018-12-31', FALSE),
   ('2019-01-01', TRUE), ('2019-01-02', FALSE), ('2019-01-03',FALSE),
('2019-01-04', FALSE), ('2019-01-05', FALSE),
   ('2019-01-07', FALSE), ('2019-02-09', TRUE), ('2019-03-09',TRUE),
('2019-04-21', TRUE), ('2019-05-01', TRUE),
   ('2019-09-07', TRUE), ('2019-10-12', TRUE), ('2019-11-02',
TRUE),('2019-11-15', TRUE), ('2019-12-19', TRUE),
   ('2019-12-20', TRUE), ('2019-12-21', TRUE), ('2019-12-22',
TRUE),('2019-12-23', TRUE), ('2019-12-25', TRUE),
   ('2019-12-26', TRUE), ('2019-12-27', TRUE)) x (dia, repete)
), materias as (
  SELECT * from (VALUES
(593, 11091, 'AAC - Ética e Segurança Digital', 9, 120,
'2019/01/30'::DATE, '2019/01/30'::DATE, 60, 120, 0),
(593, 11085, 'AAC - Computação nas Nuvens', 12, 60, NULL, NULL, 60, 120,
60)
  ) x (turma_id, materia_id, materia, sequencia, previsto, dataini, datafim,
tempoatividade, minutosaula, minutosrestantes)
), col_diasaula(turma_id, dia, tempoaula) as(select * from (values(593, 3,
time '02:00')) as x(turma_id, dia, tempoaula)
), aulasporsemana as (
   select * from (values (593,1)) x (turma_id, quantidade)
), assistidas (turma_id, sequencia, conteudo_id, conteudo, prevista,
aularealizada, tempoatividade, aulasNoDia, dia, minutoaula,
minutosassistidos, cargaconteudo, cargarestante) as (
  SELECT
materias.turma_id,
materias.sequencia,
materias.materia_id,
materias.materia,
coalesce(realizada.prevista, 1),
realizada.aularealizada,
materias.tempoatividade,
(realizada.minutosassistidos / materias.tempoatividade),
realizada.dia,
materias.minutosaula,
realizada.minutosassistidos,
materias.previsto,
coalesce(materias.previsto - (row_number() OVER AulaDaMateria *
realizada.minutosassistidos), materias.previsto)
  FROM materias
  LEFT JOIN LATERAL (
SELECT TRUE, tsrange(col_aula.data, (col_aula.data + ( col_aula.tempo
||' minute')::INTERVAL)) dia, 0 prevista, (extract(EPOCH FROM
col_aula.tempo) / 60) minutosassistidos
FROM (VALUES
 (593, 11091, TIMESTAMP '2019-01-30 19:00:00', TIME '02:00'),
 (593, 11091, '2019-02

psql and jdbc default sslmode

2020-08-12 Thread Shankar Bhaskaran
Hi ,

We are making connections to the postgres server through jdbc and psql
(libpq) . I have set the ssl as on the postgres server . It can take ssl as
well as non ssl connections.
I made a connection through a psql client to postgres server and could
confirm that the default sslmode (when no sslmode parameter is supplied
while making connection) is "prefer". Please note i have not supplied the
sslmode parameter in the connection string from psql. Still connection is
secured

 psql "postgresql://$POSTGRES_HOST:$PG_PORT/postgres" -U postgres
psql (9.6.18, server 9.6.12)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384,
bits: 256, compression: off)
Type "help" for help.

I have read in the AWS documentation for jdbc connections to server  the
default mode is "verify-full". I created a jdbc connection to the postgres
server by supplying no sslmode to the connection string . The connection
was successful . I just want to confirm what is the default sslmode for
jdbc connections to the postgres server when ssl is turned on the server.

Regards,
Shankar


Re: Server goes to Recovery Mode when run a SQL

2020-08-12 Thread Tom Lane
PegoraroF10  writes:
> Hi Michael, so sorry for the delay. I did never had that problem again, so
> tougth it was solved, but seems not. We have two different SQL which pushs
> server to recovery mode. This SQL is the hardest one, because you´ll have to
> wait some time to get the error. Just runs this SQL, opens your HTOP and
> wait, you´ll see your memory growing and growing until ... 

Are you sure this query ever terminates?  It looks to me like it just
keeps producing tuples indefinitely until you kill it or something runs
out of memory.

If the response to "out of memory" is a server crash rather than something
nicer, you probably need to corral your OOM killer.  See

https://www.postgresql.org/docs/current/kernel-resources.html#LINUX-MEMORY-OVERCOMMIT

regards, tom lane




Re: Implement a new data type

2020-08-12 Thread raf
On Wed, Aug 12, 2020 at 12:44:21PM -0400, Philip Semanchuk 
 wrote:

> > On Aug 11, 2020, at 8:01 PM, raf  wrote:
> > 
> > On Tue, Aug 11, 2020 at 06:38:39AM -0700, Miles Elam 
> >  wrote:
> > 
> >> Also of note: PostgreSQL already has a money type (
> >> https://www.postgresql.org/docs/current/datatype-money.html)
> >> But you shouldn't use it (
> >> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money).
> >> 
> >> I only bring it up so that you can know to make your money type a slightly
> >> different name to avoid a conflict. Money is deceptively hard to implement
> >> correctly. I'd recommend reading the second link if you have not already to
> >> avoid previously known issues.
> > 
> > I use decimal(10,2) for whole cents, and decimal(12,6)
> > for sub-cents. Single currency only. I didn't know
> > there was a money type originally, but it wouldn't be
> > usable for me anyway without the ability to specify the
> > scale and precision.
> > 
> > I recommend considering passing values to the database
> > as "decimal '1.23'" rather than bare numeric literals,
> > just so there's no chance of the value being
> > interpreted as a float at any stage by postgres. Maybe
> > that's being too paranoid but that's a good idea when
> > it comes to money. :-)
> 
> Yes, I agree, this is also important (and easy to overlook) if
> you’re accessing the database via a non-SQL language. We use Python
> which, like most (all?) languages that rely on the underlying C
> library for floating point support, is vulnerable to floating point
> noise. Python has a fixed precision type, and like Postgres it also
> accepts character and float input. The float input can give surprising
> results.
> 
> >>> decimal.Decimal('1.79')   # This is OK
> Decimal('1.79')
> >>> decimal.Decimal(1.79) # This will not end well!
> Decimal('1.79003552713678800500929355621337890625')
> >>>
> 
> In the case of a Postgres column like numeric(10,2), input like
> 1.79003552713678800500929355621337890625 will get rounded
> to 1.79 anyway and no harm will be done. But like you said, raf, it’s
> a good idea to be too paranoid. :-)
> 
> Cheers
> Philip

Chris Travers pointed out to me that Postgres itself
parses floating point literals as the numeric type
(i.e. "select pg_typeof(1.23);" returns numeric) so
Postgres has made the right choice for its parser,
unlike most(?) languages (except raku). But yes, in
Python, it's decimal.Decimal with integer/string input
all the way (with dec=decimal.Decimal if you have a lot
of them).

cheers,
raf





Re: How is PG replication typically used to create a High Availability (HA) config ?

2020-08-12 Thread David Gauthier
Thanks again Paul and Rob.

I'm going to need more specifics from my IT department regarding exactly
what they did... what tool they used to create what they are calling this
"High Availability" DB (pg-bouncer, etc...). If I can determine that, then
maybe there are already some hooks in place that I can leverage.  But for
this to be seamless, I suspect I'll also have to do something on the app
end beyond making a hard connection from my perl script using DBI.  I did
find something about an enhanced version of the DBI connect method which
might redirect connections in the event of a primary/backup swap.

-dave

On Wed, Aug 12, 2020 at 4:10 AM Paul Förster 
wrote:

> Hi David,
>
> please don't top-post.
>
> > On 11. Aug, 2020, at 22:57, David Gauthier 
> wrote:
> >
> > Thanks for the response Paul :-)
> >
> > Our code is actually perl which uses DBI which has functions to ping a
> DB on a specific server and connect to it.
> > But my question was more along the lines  of whether or not the onus to
> do this sort of thing typically lies with the app or something outside
> which is orchestrating the HA cfg.
>
> it should be handled outside the app, im my opinion. But then, many
> installations don't use pg-bouncer, HA-proxy, virtual IP addresses or
> something like that. That's why I suggested using libpq. libpq can handle
> it. I'm not sure if and how it can in done in Perl, though.
>
> I played around a little with perl-DBI, perl-DBI-Pg, perl-URI and
> perl-URI-db and, though I managed to get connected, I did not manage to
> specifically select a connect to the primary or replica database cluster.
>
> Also, your initial steps should be done differently:
>
> 1. select count(*) from pg_stat_replication; => p
> 2. select count(*) from pg_stat_wal_receiver; => r
>
> if:
>
> p = 0 & r = 0 => single database cluster, no replication
> p > 0 & r = 0 => primary database cluster
> p = 0 & r > 0 => replica database cluster
> p > 0 & r > 0 => primary and replica database cluster
>
> The last case can for example happen, if you have database cluster A
> replicate to B, and B replicate to C, and then connect to B.
>
> Also, the test that many people do to select pg_is_in_recovery(); is not a
> good idea because B and C of the above example are probably in recovery
> mode, so you still don't know which end you're on.
>
> Also, pg_is_in_recovery() will probably not work with logical but only
> streaming replication (both async and sync) because I expect B and C to not
> be in recovery mode when using logical replication. I didn't try logical
> replication, so someone please correct me if I'm wrong here.
>
> If you just want to know, whether your connection is read-write or
> read-only, you can simply:
>
> show transaction_read_only;
>
> Cheers,
> Paul


hash joins are causing no space left error

2020-08-12 Thread Ayub M
This is PostgreSQL 11 on AWS, there is a mview query in this OLAP database,
the tables involved are huge - 50-100m records on average records hundreds
of columns in most cases. The query runs for a while and then errors out
saying "No space left on device". I could see it generating around 500gb of
temp file data. At times it goes thru and at times it fails - probably due
to other queries running at the same time and causing failure.

The tables are partitioned and indexed on the PKs and FKs. Using
parallelism (4) with increased work_mem (4gb).

The joins are happening on around 10 tables and all are joining on the PK
and FK columns. I see partition pruning happening but the hash joins are
killing the query.

Is there any way to avoid hash joins? If we create hash indexes on the
joining columns, would PostgreSQL avoid hashing operation and instead use
hash indexes on the tables and join them. That way I feel resource
intensive hashing would be avoided and there wont be any need of temp
files. I tried but does not seem to work, when I query the table with
specific values then it uses the hash index but when I am joining the
tables it seems to do its own hash join.

My question is how to optimize massive table joins in PostgreSQL to resolve
- avoid space failures and make it run fast - takes a couple of hours to
complete now. Any best practices or suggestions.


Re: hash joins are causing no space left error

2020-08-12 Thread Tom Lane
Ayub M  writes:
> This is PostgreSQL 11 on AWS, there is a mview query in this OLAP database,
> the tables involved are huge - 50-100m records on average records hundreds
> of columns in most cases. The query runs for a while and then errors out
> saying "No space left on device". I could see it generating around 500gb of
> temp file data. At times it goes thru and at times it fails - probably due
> to other queries running at the same time and causing failure.

Are you sure that these queries are actually producing the answers you
want?  It sounds suspiciously like you are computing underconstrained
joins.

> The joins are happening on around 10 tables and all are joining on the PK
> and FK columns. I see partition pruning happening but the hash joins are
> killing the query.
> Is there any way to avoid hash joins?

TBH, you are asking the wrong question.  A merge join would take about as
much temporary space, and a nestloop join over so much data would probably
not finish in an amount of time you're willing to wait.  Indexes are NOT
a magic solution here.  What you need to be thinking about is how to not
need to process so much data.

If you really need to have this proven to you, you can try "set
enable_hashjoin = off", but I don't think you'll find that better.

regards, tom lane