Re: pgbouncer best practices

2023-07-08 Thread Ben Chobot
Rita wrote on 7/8/23 4:27 AM: I am not sure if it allows transaction pooling. Well, take the time to figure it out. With transaction pooling enabled, you can get away with a much lower number of server connections. For example, our application regularly has thousands of clients connected to

Re: pgbouncer best practices

2023-07-07 Thread Ben Chobot
Rita wrote on 7/7/23 9:23 AM: I have an  application that does many db calls  from a server farm. I've increased my max connections on postgresql to 1000 and tuned the server accordingly. However, I still get can't connect to postgresql some times. I installed pgbouncer on few servers in the fa

Re: pgbouncer

2023-06-28 Thread Ben Chobot
Laurenz Albe wrote on 6/28/23 5:27 AM: On Wed, 2023-06-28 at 07:19 -0400, Rita wrote: seems like I may need to deploy pgbouncer for my webapp. should i deploy it on the db server or on the webserver? On the database server. You don't want network latency between pgbouncer and PostgreSQL, so

Re: typical active table count?

2023-06-28 Thread Ben Chobot
Jeremy Schneider wrote on 6/27/23 11:47 AM: Thank Ben, not a concern but I'm trying to better understand how common this might be. And I think sharing general statistics about how people use PostgreSQL is a great help to the developers who build and maintain it. One really nice thing about Postg

Re: typical active table count?

2023-06-27 Thread Ben Chobot
We certainly have databases where far more than 100 tables are updated within a 10 second period. Is there a specific concern you have? Jeremy Schneider wrote on 6/27/23 9:01 AM: Question for other PostgreSQL users On your moderately busy DB, how many different tables might receive at least o

Re: Active Active PostgreSQL Solution

2023-06-09 Thread Ben Chobot
Mohsin Kazmi wrote on 6/9/23 3:38 AM: Now in order to deploy PostgreSQL in our production servers, I need to configure it in Active Active mode. Can anyone help me to do so? We don't need multi-master postgres very often, but when we do, we find the open source bucardo project works fairly wel

Re: PostgreSQL vs MariaDB

2023-03-24 Thread Ben Chobot
Inzamam Shafiq wrote on 3/24/23 4:07 AM: Hi Team, Hope you are doing well. Can someone please list pros and cons of MariaDB vs PostgreSQL that actually needs serious consideration while choosing the right database for large OLTP DBs (Terabytes)? That's a very broad question, which will take

Re: Reducing bandwidth usage of database replication

2022-11-03 Thread Ben Chobot
Sascha Zenglein wrote on 11/2/22 7:56 AM: Hi all, I want to use the postgres-native logical replication to have multiple clients receive and send data to a central database. Real-time is far less important than network usage, and with my current test setup it appears both instances communicate

Re: could not open relation with OID

2022-01-27 Thread Ben Chobot
Michael Paquier wrote on 1/26/22 9:14 PM: On Wed, Jan 26, 2022 at 05:30:01PM -0800, Ben Chobot wrote: Other things we've considered:     - we run pg_repack, which certainly seems like it could make an error like this, but we see this error in places and times that pg_repack isn't

could not open relation with OID

2022-01-26 Thread Ben Chobot
We do a lot of queries per day, over a lot of hosts, all of which are on 12.9. We've recently started doing a better job at analyzing our db logs and have found that, a few times a day, every day, we see some of our queries fail with errors like: could not open relation with OID 201940279 In

NIST 800-53v4 scanning?

2022-01-25 Thread Ben Chobot
Hello pgsql-general, I've been tasked with scanning our Ubuntu-hosted databases for NIST 800-53v4 compliance. I'm finding a paucity of tools out there that will do this. I found a few that might work, when pointed at Postgres 9 on RHEL, but not much else. Is this a problem anybody else has t

Re: Database Scalability

2021-12-01 Thread Ben Chobot
Mladen Gogala wrote on 11/30/21 7:52 PM: To my knowledge PostgreSQL doesn't support sharding, which is well and good because sharding is mostly useless, at least in my opinion. OK I'll bite. Not only does PostgreSQL natively support table partitioning (which is absolutely a form of shardin

Re: Database Scalability

2021-12-01 Thread Ben Chobot
Saurav Sarkar wrote on 11/30/21 7:08 PM: So are all the schemas on one DB or are distributed/sharded across multiple DBs ? In our use case, every db entirely homes one or more schemas. Some dbs host many schemas for small customers, some dbs host a handful of schemas for medium customers, and s

Re: Database Scalability

2021-11-30 Thread Ben Chobot
Saurav Sarkar wrote on 11/29/21 10:13 PM: Hi All, We have some multi-tenant solutions which are separating the tenant data in Postgresql mainly in the following manner. 1. Using different schemas 2. Using different tables for different tenants. Without more details it's impossible to give y

Re: Pause streaming replication

2021-11-10 Thread Ben Chobot
Rita wrote on 11/10/21 5:36 PM: Yes, I have read the manual and seen this. It pauses the replication (select pg_is_wal_replay_paused()). But on the primary, when I look at pg_stat_replication, it still says 'streaming' in the state column. My question was how do I get it from 'streaming'  to an

Re: Pause streaming replication

2021-11-10 Thread Ben Chobot
Rita wrote on 11/10/21 1:25 PM: Hello. I am testing alerting on my primary and standby setup. I have async replication working but I would like to temporarily pause it so the value of 'state' isn't streaming. (select * from pg_stat_replication). How can I do that? By reading the fine manua

Re: To JIT (or not!) in Ubuntu packages

2021-09-14 Thread Ben Chobot
Thomas Munro wrote on 9/14/21 5:50 PM: On Tue, Sep 14, 2021 at 10:11 AM Ben Chobot wrote: We've noticed that the Ubuntu postgresql-12 package has --with-llvm enabled on x86_64, but not on aarch64. Does anybody know if this was intentional, or just an oversight? For what it's worth

To JIT (or not!) in Ubuntu packages

2021-09-13 Thread Ben Chobot
We've noticed that the Ubuntu postgresql-12 package has --with-llvm enabled on x86_64, but not on aarch64. Does anybody know if this was intentional, or just an oversight? For what it's worth, it seems the beta postgresql-14 package for Ubuntu still doesn't have --with-llvm. I'm not sure if

Re: Licensing

2021-08-18 Thread Ben Chobot
Bryan Boone wrote on 8/18/21 11:39 AM: Can someone tell me if I am able to use PostgreSQL for the small company I work for? Yes, you are.

Re: looping over multirange segments?

2021-07-15 Thread Ben Chobot
hubert depesz lubaczewski wrote on 7/15/21 5:15 AM: select * from regexp_matches(_YOUR_MULTIRANGE_::text, '[\[(][^\])]+[\])]', 'g'); I wrote more, including explanation, and ready-to-use function, in here: https://www.depesz.com/2021/07/15/how-to-get-list-of-elements-from-multiranges/ So ugly

looping over multirange segments?

2021-07-14 Thread Ben Chobot
I'm really, really liking the multirange types in PG14. Thank you for making them! Unfortunately I am struggling with how to loop over the segments of a multirange. There doesn't seem to be a way to convert them to arrays, and I can't just use plpgsql's FOREACH on one. Am I missing something ob

Re: Doubt on pgbouncer

2021-07-03 Thread Ben Chobot
Rama Krishnan wrote on 7/3/21 8:35 AM: Hi Team, How can I split read and write queries using pgbouncer You do it with your application. Make a pgbouncer database for read/write queries and point it at your postgresql primary, and then make a second pgbouncer database for read-only queries a

Re: 12.3 replicas falling over during WAL redo

2020-09-04 Thread Ben Chobot
Alvaro Herrera wrote on 8/3/20 4:54 PM: On 2020-Aug-03, Ben Chobot wrote: Alvaro Herrera wrote on 8/3/20 2:34 PM: On 2020-Aug-03, Ben Chobot wrote: dd if=16605/16613/60529051 bs=8192 count=1 seek=6501 of=/tmp/page.6501 If I use skip instead of seek Argh, yes, I did correct that in my

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Alvaro Herrera wrote on 8/3/20 2:34 PM: On 2020-Aug-03, Ben Chobot wrote: Alvaro Herrera wrote on 8/3/20 12:34 PM: On 2020-Aug-03, Ben Chobot wrote: Yep. Looking at the ones in block 6501, rmgr: Btree   len (rec/tot): 72/    72, tx:   76393394, lsn: A0A/AB2C43D0, prev A0A/AB2C4378

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Peter Geoghegan wrote on 8/3/20 3:04 PM: On Mon, Aug 3, 2020 at 2:35 PM Alvaro Herrera wrote: You can use pageinspect's page_header() function to obtain the page's LSN. You can use dd to obtain the page from the file, dd if=16605/16613/60529051 bs=8192 count=1 seek=6501 of=/tmp/page.6501 Ben

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Alvaro Herrera wrote on 8/1/20 9:35 AM: On 2020-Aug-01, Ben Chobot wrote: Can you find out what the index is being modified by those LSNs -- is it always the same index?  Can you have a look at nearby WAL records that touch the same page of the same index in each case? They turn out to be

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Peter Geoghegan wrote on 8/3/20 11:25 AM: On Sun, Aug 2, 2020 at 9:39 PM Kyotaro Horiguchi wrote: All of the cited log lines seem suggesting relation with deleted btree page items. As a possibility I can guess, that can happen if the pages were flushed out during a vacuum after the last checkpo

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Alvaro Herrera wrote on 8/3/20 12:34 PM: On 2020-Aug-03, Ben Chobot wrote: Yep. Looking at the ones in block 6501, rmgr: Btree   len (rec/tot): 72/    72, tx:   76393394, lsn: A0A/AB2C43D0, prev A0A/AB2C4378, desc: INSERT_LEAF off 41, blkref #0: rel 16605/16613/60529051 blk 6501 rmgr

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Ben Chobot wrote on 8/1/20 9:58 AM: Alvaro Herrera wrote on 8/1/20 9:35 AM: On 2020-Aug-01, Ben Chobot wrote: Can you find out what the index is being modified by those LSNs -- is it always the same index?  Can you have a look at nearby WAL records that touch the same page of the same index in

Re: 12.3 replicas falling over during WAL redo

2020-08-03 Thread Ben Chobot
Kyotaro Horiguchi wrote on 8/2/20 9:39 PM: At Sat, 1 Aug 2020 09:58:05 -0700, Ben Chobot wrote in All of the cited log lines seem suggesting relation with deleted btree page items. As a possibility I can guess, that can happen if the pages were flushed out during a vacuum after the last

Re: 12.3 replicas falling over during WAL redo

2020-08-01 Thread Ben Chobot
Alvaro Herrera wrote on 8/1/20 9:35 AM: On 2020-Aug-01, Ben Chobot wrote: We have a few hundred postgres servers in AWS EC2, all of which do streaming replication to at least two replicas. As we've transitioned our fleet to from 9.5 to 12.3, we've noticed an alarming incre

12.3 replicas falling over during WAL redo

2020-08-01 Thread Ben Chobot
We have a few hundred postgres servers in AWS EC2, all of which do streaming replication to at least two replicas. As we've transitioned our fleet to from 9.5 to 12.3, we've noticed an alarming increase in the frequency of a streaming replica dying during replay. Postgres will log something lik

Re: Postgresql 9.6 -> AWS RDS Postgresql 12.2 with pg_logical

2020-05-31 Thread Ben Chobot
Igor Polishchuk wrote on 5/30/20 9:33 PM: Hello, I need to replicate Postgresql 9.6 to  AWS RDS Postgresql 12.2 with pg_logical. AWS RDS Pg 12.2 (target) only supports pg_logical 2.3.0. Can I use v2.3.1 on the source and v2.3.0 on the target? We had lots of issues with pglogical when we accid

streaming slaves can't keep up?

2020-03-31 Thread Ben Chobot
We have a few busy 9.5 dbs, both streaming to a few slaves each. The master and slaves are identical hardware and are getting no small amount of load - about 45k transactions/s on the master and ~36k transactions/s on the slave actively serving clients. During these busy times, queries are all

Re: Redis 16 times faster than Postgres?

2019-09-29 Thread Ben Chobot
On Sep 29, 2019, at 8:44 PM, Ron wrote: > > On 9/29/19 8:09 PM, Steve Litt wrote: >> On Mon, 30 Sep 2019 07:46:14 +1000 >> Nathan Woodrow wrote: >> >>> Redis is a in memory database so I would except it to be always much >>> faster.. >> Is there a way to have Redis periodically update an on-dis

Re: Question from someone who is not trained in computer sciences

2019-09-05 Thread Ben Chobot
On Sep 5, 2019, at 2:00 PM, Judith Lacoste wrote: > > Hi, > > I plan to install the database on a server in the office. Me and my four > colleagues will occasionally connect to this database when we are working in > other locations (usually hospitals or universities). In such remote > locatio

Re: Monitor the ddl and dml activities in logs

2019-03-06 Thread Ben Chobot
> On Mar 6, 2019, at 5:58 AM, Nanda Kumar > wrote: > > Hello Team, > > I would like to know where I can monitor the ddl and dml operations happens > in the production environment . The documentation is your friend, particularly https://www.postgresql.org/docs/9.6/runtime-config-logging.htm

Re: master-> 2 hot standbys

2019-03-04 Thread Ben Chobot
> On Mar 4, 2019, at 1:59 PM, Julie Nishimura wrote: > > Hello, > Our current master 9.2 has two active standbys. Can you please help me out > with the right sequence of events if we would like to promote one of current > standbys to master and convert master to standby? It depends on how yo

Re: automated refresh of dev from prod

2019-02-28 Thread Ben Chobot
On Feb 28, 2019, at 8:04 AM, Stephen Frost wrote: > > Greetings, > > * Julie Nishimura (juliez...@hotmail.com) wrote: >> Hello everybody, I am new to postgresql environment, but trying to get up to >> speed. >> Can you please share your experience on how you can automate refreshment of >> dev

constraint exclusion with a tsrange type

2018-09-12 Thread Ben Chobot
Hey everybody, I'm having trouble getting constraint exclusion to work on a table partitioned with a tsrange type. I've distilled it down to this: create table t ( id serial primary key, observed_window tsrange not null ); create index t_window on t(observed_window); create table p1 (like t

Re: Renice on Postgresql process

2018-05-23 Thread Ben Chobot
> On May 7, 2018, at 11:50 PM, Ayappan P2 wrote: > > We are doing "renice" on the main Postgresql process to give higher > scheduling priority because other critical operations depends on the database. > You are saying that the database processes take longer to relinquish their > resources an

Re: Renice on Postgresql process

2018-05-07 Thread Ben Chobot
On May 7, 2018, at 7:46 AM, Ayappan P2 wrote: > > Hi All, > > We are using Postgresql in AIX. Unlike some other databases, Postgresql has > lot of other process running in the background along with the main process. > > We do "renice" only on the Postgres main process. Is it sufficient to hav

Re: Using AWS ephemeral SSD storage for production database workload?

2018-01-29 Thread Ben Chobot
> On Jan 29, 2018, at 8:05 PM, Sam Gendler wrote: > > Why not use EBS storage, but don’t use provisioned iops SSDs (io1) for the > ebs volume. Just use the default storage type (gp2) and live with the 3000 > IOPS peak for 30 minutes that that allows. You’d be amazed at just how much > I/o can