Re: How to run in parallel in Postgres

2019-12-07 Thread Lars Aksel Opsahl 
 >From: Ondrej Ivanič 

 >Sent: Saturday, December 7, 2019 2:23 AM

 >Cc: [email protected] 
 >

 >Subject: Re: How to run in parallel in Postgres

 >

 >Hi Lars,

 >

 >I have two suggestions:

 >

 >- `xargs` almost always present and it can run in parallel (-P) but script 
 >needs to be changed:

 >for((i=1;i<=28;i++)); do echo "SELECT 
 >find_overlap_gap_single_cell('test_data.overlap_gap_input_t1','geom',4258,'test_data.overlap_gap_input_t1_res',${I},28);";
 > done | xargs -n1 -P 10 psql ...

 >

 >- `UNION ALL` might trigger parallel execution (you need to mess with the 
 >cost of the function and perhaps other settings):

 >SELECT

 > find_overlap_gap_single_cell('test_data.overlap_gap_input_t1','geom',4258,'test_data.overlap_gap_input_t1_res',1,28)

 >UNION ALL

 >SELECT

 > find_overlap_gap_single_cell('test_data.overlap_gap_input_t1','geom',4258,'test_data.overlap_gap_input_t1_res',2,28)

 >...

 >

 >

 >Cheers,

 >

Hi Ondrej


  *   Yes using xargs seems be an alternative to GNU parallel and I will have 
that in mind.
  *   I did a test using UNION ALL in the branch 
https://github.com/larsop/find-overlap-and-gap/tree/union_all_parallel but I 
was not able to trigger Postgres parallel .  That may be related to what Justin 
say about create tables.


Thanks.

Lars

Re: How to run in parallel in Postgres

2019-12-07 Thread Lars Aksel Opsahl 
> From: Justin Pryzby 

> Sent: Saturday, December 7, 2019 2:25 AM

> To: Lars Aksel Opsahl 

> Cc: [email protected] 
> 

> Subject: Re: How to run in parallel in Postgres

>

> On Thu, Dec 05, 2019 at 12:10:42PM +, Lars Aksel Opsahl wrote:

> > I have a function that prepares data, so the big job can be run it in 
> > parallel.

> >

> > Since Postgres now supports parallel I was wondering if it's easy to 
> > trigger parallel dynamically created SQL calls.

> >

> > If you look at 
> > https://github.com/larsop/find-overlap-and-gap/blob/master/src/test/sql/regress/find_overlap_and_gap.sql
> >   you see that

> >

> > find_overlap_gap_make_run_cmd generates as set of 28 sql calls.

> >

> > So is it in a simple way possible to use Postgres parallel functionality to 
> > call this 28 functions i parallel so I don't have dependent on externally 
> > install programs  ?

>

> SELECT 
> find_overlap_gap_single_cell('test_data.overlap_gap_input_t1','geom',4258,'test_data.overlap_gap_input_t1_res',1,28);

> SELECT 
> find_overlap_gap_single_cell('test_data.overlap_gap_input_t1','geom',4258,'test_data.overlap_gap_input_t1_res',2,28);

> SELECT 
> find_overlap_gap_single_cell('test_data.overlap_gap_input_t1','geom',4258,'test_data.overlap_gap_input_t1_res',3,28);

> ...

>

> I see that find_overlap_gap_single_cell creates tables, so cannot be run in 
> parallel.

> Maybe you could consider rewriting it to return data to its caller instead.

> You'd also need to mark it as PARALLEL SAFE, of course.

> Your other functions involved should be PARALLEL SAFE too.

>

> Justin

Hi Justin

The reason why I don't return the results Is that on very bug tables I usually  
get memory problems if I return all the results to the master function. So I 
usually break thing up into small unlogged tables. Then I work on each table 
separately or in groups. When all steps are done i merge all the small tables 
together. I this case we only single step, but usually I work many more steps.

But I will keep mind that it may work i parallel if I don't create any child 
tables but returns the result.

Thanks.

Lars

Re: Legal disclaimers on emails to this group

2019-12-07 Thread Jean-David Beyer 
On 12/6/19 1:42 PM, Craig James wrote:
> (I've changed the original subject, "autovacuum locking question", of
> the sender's email so as not to hijack that thread.)
> 
> On Thu, Dec 5, 2019 at 2:26 PM Mike Schanne  > wrote:
> 
> Hi,
> 
> I am investigating a performance problem...
> 
> ... This email is non-binding, is subject to contract, and neither
> Kulicke and Soffa Industries, Inc. nor its subsidiaries (each and
> collectively “K&S”) shall have any obligation to you to consummate
> the transactions herein or to enter into any agreement, other than
> in accordance with the terms and conditions of a definitive
> agreement if and when negotiated, finalized and executed between the
> parties. This email and all its contents are protected by
> International and United States copyright laws. Any reproduction or
> use of all or any part of this email without the express written
> consent of K&S is prohibited.
> 
> 
> Sorry to be off topic, but this bugs me. Language is important. This
> isn't directed at you specifically, but I see these disclaimers all the
> time. How can you post to a public newsgroup that automatically
> reproduces your email to thousands of subscribers, and additionally
> publishes it on publicly accessible archives, in direct conflict with
> your company's policy appended to your email? And why on Earth do your
> company's lawyers think this sort of disclaimer is helpful and even
> legally useful? Not to mention, do they realize it's vaguely offensive
> to every customer and colleague who receives it?
> 
> Craig

People should probably not post anything on newsgroups from computers
owned by their employers. They are probably violating the terms of their
employment.

It would be perfectly acceptable to me if all news servers automatically
deleted all such would-be posts instead of actually posting them.



-- 
  .~.  Jean-David Beyer
  /V\  PGP-Key:166D840A 0C610C8B
 /( )\ Shrewsbury, New Jersey
 ^^-^^ 07:05:02 up 23 days, 7:00, 2 users, load average: 5.31, 4.71, 4.38