Re: speed up full table scan using psql

2023-05-31 Thread Lian Jiang
The whole command is: psql %(pg_uri)s -c %(sql)s | %(sed)s | %(pv)s | %(split)s) 2>&1 | %(tr)s where: sql is "copy (select row_to_json(x_tmp_uniq) from public.mytable x_tmp_uniq) to stdout" sed, pv, split, tr together format and split the stdout into jsonl files. Hope this helps. On Wed, May 3

Re: speed up full table scan using psql

2023-05-31 Thread Adrian Klaver
On 5/31/23 13:57, Lian Jiang wrote: The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq) from public.mytable x_tmp_uniq) to stdout" What is taking the stdout and what it is it doing? postgres version:  14.7 Does this mean COPY and java CopyManager may not help since my psql

Re: speed up full table scan using psql

2023-05-31 Thread Thorsten Glaser
On Wed, 31 May 2023, Adrian Klaver wrote: > Given that I would just COPY the data out as CSV. I recently did something similar. I found the JSON functions not quite satisfying and the extra spaces redundant, but it turns out that, for a numerical table, exporting as CSV, loading that via AJAX the

Re: speed up full table scan using psql

2023-05-31 Thread Adrian Klaver
On 5/31/23 13:57, Lian Jiang wrote: The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq) from public.mytable x_tmp_uniq) to stdout" postgres version:  14.7 Does this mean COPY and java CopyManager may not help since my psql command already uses copy? Regarding pg_dump, it doe

Re: speed up full table scan using psql

2023-05-31 Thread Adrian Klaver
On 5/31/23 13:57, Lian Jiang wrote: The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq) from public.mytable x_tmp_uniq) to stdout" postgres version:  14.7 Does this mean COPY and java CopyManager may not help since my psql command already uses copy? I don't think the issue i

Re: speed up full table scan using psql

2023-05-31 Thread Lian Jiang
The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq) from public.mytable x_tmp_uniq) to stdout" postgres version: 14.7 Does this mean COPY and java CopyManager may not help since my psql command already uses copy? Regarding pg_dump, it does not support json format which means ext

Re: EXTERNAL: Re: Question - Does PostgreSQL have an Evaluation Assurance Level?

2023-05-31 Thread Laurenz Albe
On Wed, 2023-05-31 at 19:51 +, Mayer, Nicholas J wrote: > We found out that while PostgreSQL does not have EAL, the 'Crunchy Data' does > have EAL of 2. I see. I guess you are aware that a closed source fork of PostgreSQL is probably no more secure than the original. But this is more about

Re: Question - Does PostgreSQL have an Evaluation Assurance Level?

2023-05-31 Thread Laurenz Albe
On Tue, 2023-05-30 at 13:48 +, Mayer, Nicholas J wrote: > My name is Nick Mayer, and I had a question concerning PostgreSQL’s EAL. Has > PostgreSQL > been put through any audit/security testing, and does it have an EAL? If so, > would I be > able to get this information? I would appreciate an

Re: Hash Index on Partitioned Table

2023-05-31 Thread peter.boris...@kartographia.com
Hi Tom, Thanks so much for your quick response. As luck would have it, the index FINALLY finished about an hour ago. For a size comparison: BTree: 6,433 GB Hash: 8,647 GB Although I don't have a proper benchmark to compare performance, I can say the hash is working as good as if not fast

Re: Help needed to understand query planner regression with incremental sort

2023-05-31 Thread Tom Lane
"Henrik Peinar (nodeSWAT.com)" writes: > *Quick description: *After upgrading our Aurora PostgreSQL engine from v11 > to v15, one of our often run SQL's started taking tens of seconds instead > of running sub 100ms. Explain analyze showed that the query planner had > switched to using incremental

Re: speed up full table scan using psql

2023-05-31 Thread Adrian Klaver
On 5/30/23 21:25, Lian Jiang wrote: hi, I am using psql to periodically dump the postgres tables into json files which are imported into snowflake. For large tables (e.g. 70M rows), it The command you are using is? Postgres version? takes hours for psql to complete. Using spark to read the

Re: Is there a bug in psql? (SELECT ''';)

2023-05-31 Thread Adrian Klaver
On 5/31/23 00:17, Wen Yi wrote: Hi team, when I learn the postgres, I try to store the ' into the database, but something unexpected happend. postgres=# CREATE TABLE test (str varchar); CREATE TABLE postgres=# INSERT INTO test values ('''); postgres'# SELECT * FROM test; postgres'# exit Use cont

Help needed to understand query planner regression with incremental sort

2023-05-31 Thread Henrik Peinar (nodeSWAT.com)
*Quick description: *After upgrading our Aurora PostgreSQL engine from v11 to v15, one of our often run SQL's started taking tens of seconds instead of running sub 100ms. Explain analyze showed that the query planner had switched to using incremental sort. Running ANALYZE on the table fixed the iss

Re: Hash Index on Partitioned Table

2023-05-31 Thread Tom Lane
"peter.boris...@kartographia.com" writes: > I have a rather large database with ~250 billion records in a partitioned > table. The database has been running and ingesting data continuously for > about 3 years. > I had a "regular" BTree index on one of the fields (a unique bigint column) > but

Re: Pg 16: will pg_dump & pg_restore be faster?

2023-05-31 Thread Andres Freund
Hi, On 2023-05-30 21:13:08 -0400, Bruce Momjian wrote: > On Wed, May 31, 2023 at 09:14:20AM +1200, David Rowley wrote: > > On Wed, 31 May 2023 at 08:54, Ron wrote: > > > https://www.postgresql.org/about/news/postgresql-16-beta-1-released-2643/ > > > says "PostgreSQL 16 can also improve the perfor

Hash Index on Partitioned Table

2023-05-31 Thread peter.boris...@kartographia.com
Dear PostgreSQL Community, I have a rather large database with ~250 billion records in a partitioned table. The database has been running and ingesting data continuously for about 3 years. I had a "regular" BTree index on one of the fields (a unique bigint column) but it was getting too big

Re: Is there a bug in psql? (SELECT ''';)

2023-05-31 Thread hubert depesz lubaczewski
On Wed, May 31, 2023 at 03:17:14PM +0800, Wen Yi wrote: > Hi team, > when I learn the postgres, I try to store the ' into the database, > > but something unexpected happend. > > > postgres=# CREATE TABLE test (str varchar); > CREATE TABLE > postgres=# INSERT INTO test values ('''); > postgres'#

Re: Pg 16: will pg_dump & pg_restore be faster?

2023-05-31 Thread Bruce Momjian
On Tue, May 30, 2023 at 10:28:58PM -0400, Bruce Momjian wrote: > On Wed, May 31, 2023 at 02:18:25PM +1200, David Rowley wrote: > > On Wed, 31 May 2023 at 14:11, Bruce Momjian wrote: > > > > > > On Wed, May 31, 2023 at 02:05:10PM +1200, David Rowley wrote: > > > > "Relation extensions have been imp

AW: Is there a bug in psql? (SELECT ''';)

2023-05-31 Thread Marco Lechner
You have to mask the single quote: https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-CONSTANTS Von: Wen Yi <896634...@qq.com> Gesendet: Mittwoch, 31. Mai 2023 09:17 An: pgsql-general Betreff: Is there a bug in psql? (SELECT ''';) Hi team, when I learn the postgres, I try

Is there a bug in psql? (SELECT ''';)

2023-05-31 Thread Wen Yi
Hi team, when I learn the postgres, I try to store the ' into the database, but something unexpected happend. postgres=# CREATE TABLE test (str varchar); CREATE TABLE postgres=# INSERT INTO test values ('''); postgres'# SELECT * FROM test; postgres'# exit Use control-D to quit. postgres'# \q Us