Re: very slow largeobject transfers through JDBC

2018-09-06 Thread Mate Varga
msec query + 1.3 sec transfer). Regards, Mate On Thu, Sep 6, 2018 at 10:56 AM Dave Cramer wrote: > Hi > > Can you be more explicit how you fixed the problem ? > > Thanks > Dave Cramer > > da...@postgresintl.com > www.postgresintl.com > > > On Thu, 6

Re: very slow largeobject transfers through JDBC

2018-09-06 Thread Mate Varga
After inlining the data, performance issues have been solved. Thanks for the help. On Mon, Sep 3, 2018 at 9:57 PM Mate Varga wrote: > Thanks, > 1) we'll try to move stuff out from LOBs > 2) we might raise a PR for the JDBC driver > > Mate > > On Mon, 3 Sep 2018,

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
Thanks, 1) we'll try to move stuff out from LOBs 2) we might raise a PR for the JDBC driver Mate On Mon, 3 Sep 2018, 19:35 Dave Cramer, wrote: > > > On Mon, 3 Sep 2018 at 13:00, Mate Varga wrote: > >> More precisely: when fetching 10k rows, JDBC driver just does a l

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
More precisely: when fetching 10k rows, JDBC driver just does a large bunch of socket reads. With lobs, it's ping-pong: one read, one write per lob... On Mon, Sep 3, 2018 at 6:54 PM Mate Varga wrote: > So I have detailed profiling results now. Basically it takes very long > that fo

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
Cramer wrote: > > On Mon, 3 Sep 2018 at 10:48, Mate Varga wrote: > >> That's 1690 msec (1.69 seconds, and that is how long it takes to fetch >> 20k (small-ish) rows without LOBs (LOBs are a few lines below on the >> screenshot) >> > > that sound high as well

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
...@postgresintl.com > www.postgresintl.com > > > On Mon, 3 Sep 2018 at 10:38, Mate Varga wrote: > >> Which frame do you refer to? >> >> On Mon, Sep 3, 2018 at 3:57 PM Dave Cramer wrote: >> >>> Not sure why reading from a socket is taking 1ms ? >>

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
Which frame do you refer to? On Mon, Sep 3, 2018 at 3:57 PM Dave Cramer wrote: > Not sure why reading from a socket is taking 1ms ? > > Dave Cramer > > da...@postgresintl.com > www.postgresintl.com > > > On Mon, 3 Sep 2018 at 09:39, Mate Varga wrote: > >> H

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
sands rows + few thousand largeobjects, each largeobject is around 1 kByte. (Yes I know this is not a proper use of LOBs -- it's a legacy db structure that's hard to change.) Thanks. Mate On Mon, Sep 3, 2018 at 11:52 AM Mate Varga wrote: > Hey, > > we'll try to test th

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
Hey, we'll try to test this with pure JDBC versus hibernate. Thanks! On Mon, Sep 3, 2018 at 11:48 AM Dave Cramer wrote: > > > On Mon, 3 Sep 2018 at 03:55, Mate Varga wrote: > >> Basically there's a class with a byte[] field, the class is mapped to >> table

Re: very slow largeobject transfers through JDBC

2018-09-03 Thread Mate Varga
that turning of SSL between the app and PSQL speeds up things by at least 50%. Ah, one addition -- the binary objects are encrypted, so their entropy is very high. Mate On Sun, Sep 2, 2018 at 12:55 AM Dave Cramer wrote: > > > > On Fri, 31 Aug 2018 at 10:15, Mate Varga wrote: >

Re: very slow largeobject transfers through JDBC

2018-08-31 Thread Mate Varga
I see -- we could try that, though we're mostly using an ORM (Hibernate) to do this. Thanks! On Fri, Aug 31, 2018 at 3:57 PM Dmitry Igrishin wrote: > пт, 31 авг. 2018 г. в 16:35, Mate Varga : > > > > Hi, > > > > we're fetching binary data from pg_largeobjec

very slow largeobject transfers through JDBC

2018-08-31 Thread Mate Varga
Hi, we're fetching binary data from pg_largeobject table. The data is not very large, but we ended up storing it there. If I'm copying the data to a file from the psql console, then it takes X time (e.g. a second), fetching it through the JDBC driver takes at least 10x more. We don't see this diff

Re: pg_upgrade with large pg_largeobject table

2018-08-14 Thread Mate Varga
hat? Thanks again. Mate On Tue, Aug 14, 2018 at 7:58 PM Tom Lane wrote: > Mate Varga writes: > > We have a PSQL 9.5 DB with 16G physical RAM and ~ 1 TB data mostly stored > > in the pg_largeobject system table. This table has 250M rows at the > moment. > > You mean

pg_upgrade with large pg_largeobject table

2018-08-14 Thread Mate Varga
Hi. hanks in advance for any advice. We have a PSQL 9.5 DB with 16G physical RAM and ~ 1 TB data mostly stored in the pg_largeobject system table. This table has 250M rows at the moment. We're trying to upgrade this to 10.x with an in-place upgrade. The command I'm using is:sudo -u postgres /usr/