Segmentation fault with core dump
elect_nocancel () from /lib64/libc.so.6
No symbol table info available.
#14 0x0046ef32 in ServerLoop () at postmaster.c:1683
timeout = {tv_sec = 59, tv_usec = 999708}
rmask = {fds_bits = {120, 0 }}
selres =
now =
readmask = {fds_bits = {120, 0 }}
last_lockfile_recheck_time = 1515438348
last_touch_time = 1515438348
__func__ = "ServerLoop"
#15 0x00675b69 in PostmasterMain (argc=argc@entry=3,
argv=argv@entry=0x27ca210)
at postmaster.c:1327
opt =
status =
userDoption =
listen_addr_saved =
i =
output_config_variable =
__func__ = "PostmasterMain"
#16 0x0047053e in main (argc=3, argv=0x27ca210) at main.c:228
Kind Regards,
Glauco Torres
Re: Segmentation fault with core dump
> > That would result in nonsensical gdb output, most likely; but Glauco's > trace is internally consistent enough that I doubt gdb is lying to us. > In any case, the crash is an observable fact :-( > > > The system is a CentOS 7, and PG was installed using PGDG's YUM repository. We are pretty sure that the same binary that crashed was using on `gdb` command. More specifically, the path used was `/usr/pgsql-9.6/bin/postmaster`, and we were running 9.6.6 (most recent 9.6 minor release today) for a few weeks, so there shouldn't have any upgrade on the binaries since the server was up, specially because we restarted the service in order to allow core dump creation, this is not the first crash (although the only one with core dump generated so far), we can send new gdb stack if it happens again. More information: $ uname -a Linux pg-iii.br 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux $ /usr/pgsql-9.6/bin/pg_config BINDIR = /usr/pgsql-9.6/bin DOCDIR = /usr/pgsql-9.6/doc HTMLDIR = /usr/pgsql-9.6/doc/html INCLUDEDIR = /usr/pgsql-9.6/include PKGINCLUDEDIR = /usr/pgsql-9.6/include INCLUDEDIR-SERVER = /usr/pgsql-9.6/include/server LIBDIR = /usr/pgsql-9.6/lib PKGLIBDIR = /usr/pgsql-9.6/lib LOCALEDIR = /usr/pgsql-9.6/share/locale MANDIR = /usr/pgsql-9.6/share/man SHAREDIR = /usr/pgsql-9.6/share SYSCONFDIR = /etc/sysconfig/pgsql PGXS = /usr/pgsql-9.6/lib/pgxs/src/makefiles/pgxs.mk CONFIGURE = '--enable-rpath' '--prefix=/usr/pgsql-9.6' '--includedir=/usr/pgsql-9.6/include' '--mandir=/usr/pgsql-9.6/share/man' '--datadir=/usr/pgsql-9.6/share' '--libdir=/usr/pgsql-9.6/lib' '--with-perl' '--with-python' '--with-tcl' '--with-tclconfig=/usr/lib64' '--with-openssl' '--with-pam' '--with-gssapi' '--with-includes=/usr/include' '--with-libraries=/usr/lib64' '--enable-nls' '--enable-dtrace' '--with-uuid=e2fs' '--with-libxml' '--with-libxslt' '--with-ldap' '--with-selinux' '--with-systemd' '--with-system-tzdata=/usr/share/zoneinfo' '--sysconfdir=/etc/sysconfig/pgsql' '--docdir=/usr/pgsql-9.6/doc' '--htmldir=/usr/pgsql-9.6/doc/html' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'LDFLAGS=-Wl,--as-needed' CC = gcc CPPFLAGS = -DFRONTEND -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic CFLAGS_SL = -fPIC LDFLAGS = -L../../src/common -Wl,--as-needed -L/usr/lib64 -Wl,--as-needed -Wl,-rpath,'/usr/pgsql-9.6/lib',--enable-new-dtags LDFLAGS_EX = LDFLAGS_SL = LIBS = -lpgcommon -lpgport -lselinux -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lrt -lcrypt -ldl -lm VERSION = PostgreSQL 9.6.6 Regards, Glauco
Re: Segmentation fault with core dump
> > Might be worth comparing sha1sum's of the postgres executable between > this server and one that's not having the problem, just to eliminate > the corrupted-binary theory. > > > The return is the same for the two servers, $ sha1sum /usr/pgsql-9.6/bin/postmaster 56bcb4d644a8b00f07e9bd42f9a3f02be7ff2523 /usr/pgsql-9.6/bin/postmaster Today I left to generate more core-dump, follow the return, (gdb) bt #0 tbm_comparator (left=left@entry=0x1d5ca08, right=right@entry=0x3acdb70) at tidbitmap.c:1031 #1 0x00801268 in med3 (a=0x1d5ca08 "\350>\337\001", b=0x3acdb70 , c=0x583ecd8 , cmp=0x603ca0 ) at qsort.c:107 #2 0x00801621 in pg_qsort (a=0x1d5ca08, n=, n@entry=10477, es=es@entry=8, cmp=cmp@entry=0x603ca0 ) at qsort.c:157 #3 0x00604a7b in tbm_begin_iterate (tbm=tbm@entry=0x1dd8a00) at tidbitmap.c:635 #4 0x005d3a89 in BitmapHeapNext (node=node@entry=0x1dc2ef0) at nodeBitmapHeapscan.c:110 #5 0x005caf1a in ExecScanFetch (recheckMtd=0x5d35b0 , accessMtd=0x5d35f0 , node=0x1dc2ef0) at execScan.c:95 #6 ExecScan (node=node@entry=0x1dc2ef0, accessMtd=accessMtd@entry=0x5d35f0 , recheckMtd=recheckMtd@entry=0x5d35b0 ) at execScan.c:180 #7 0x005d3cff in ExecBitmapHeapScan (node=node@entry=0x1dc2ef0) at nodeBitmapHeapscan.c:440 #8 0x005c3fb8 in ExecProcNode (node=node@entry=0x1dc2ef0) at execProcnode.c:437 #9 0x005de877 in ExecNestLoop (node=node@entry=0x1dc0148) at nodeNestloop.c:174 #10 0x005c3f28 in ExecProcNode (node=node@entry=0x1dc0148) at execProcnode.c:476 #11 0x005de7d7 in ExecNestLoop (node=node@entry=0x1dbfdd8) at nodeNestloop.c:123 #12 0x005c3f28 in ExecProcNode (node=node@entry=0x1dbfdd8) at execProcnode.c:476 #13 0x005d624d in MultiExecHash (node=node@entry=0x1dbf9b8) at nodeHash.c:104 #14 0x005c40c0 in MultiExecProcNode (node=node@entry=0x1dbf9b8) at execProcnode.c:577 #15 0x005d6cb9 in ExecHashJoin (node=node@entry=0x1dbe688) at nodeHashjoin.c:178 #16 0x005c3f08 in ExecProcNode (node=node@entry=0x1dbe688) at execProcnode.c:484 #17 0x005de7d7 in ExecNestLoop (node=node@entry=0x1dbc6e0) at nodeNestloop.c:123 #18 0x005c3f28 in ExecProcNode (node=node@entry=0x1dbc6e0) at execProcnode.c:476 #19 0x005de7d7 in ExecNestLoop (node=node@entry=0x1dbc520) at nodeNestloop.c:123 #20 0x005c3f28 in ExecProcNode (node=0x1dbc520) at execProcnode.c:476 #21 0x005cf619 in fetch_input_tuple (aggstate=aggstate@entry=0x1dbbc48) at nodeAgg.c:598 #22 0x005d10ff in agg_retrieve_direct (aggstate=0x1dbbc48) at nodeAgg.c:2067 #23 ExecAgg (node=node@entry=0x1dbbc48) at nodeAgg.c:1892 #24 0x005c3ec8 in ExecProcNode (node=node@entry=0x1dbbc48) at execProcnode.c:503 #25 0x005c03a7 in ExecutePlan (dest=0x1b607c8, direction=, numberTuples=0, sendTuples=1 '\001', operation=CMD_SELECT, use_parallel_mode=, planstate=0x1dbbc48, estate=0x1dbba58) at execMain.c:1566 #26 standard_ExecutorRun (queryDesc=0x1c98de0, direction=, count=0) at execMain.c:338 #27 0x7f016577e0a5 in pgss_ExecutorRun (queryDesc=0x1c98de0, direction=ForwardScanDirection, count=0) at pg_stat_statements.c:877 #28 0x006d3a97 in PortalRunSelect (portal=portal@entry=0x1ad9278, forward=forward@entry=1 '\001', count=0, count@entry=9223372036854775807, dest=dest@entry=0x1b607c8) at pquery.c:948 #29 0x006d4eab in PortalRun (portal=0x1ad9278, count=9223372036854775807, isTopLevel=, dest=0x1b607c8, altdest=0x1b607c8, completionTag=0x7ffdfe32a700 "") at pquery.c:789 #30 0x006d2371 in PostgresMain (argc=, argv=, dbname=, username=) at postgres.c:1969 #31 0x0046f8d4 in BackendRun (port=0x1add6d0) at postmaster.c:4294 #32 BackendStartup (port=0x1add6d0) at postmaster.c:3968 #33 ServerLoop () at postmaster.c:1719 #34 0x00675b69 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x1ab1210) at postmaster.c:1327 #35 0x0047053e in main (argc=3, argv=0x1ab1210) at main.c:228 Regards, Glauco
Re: pgpool Connections Distributions Among Nodes
> > > > > We have 4-node cluster (1 master and 3 hot standby). We are using pgpool > as load balancer. We have an observation where if application requests for > 3 connections, pgpool connects to all 4 servers and I see 3 connections on > each of them. I was expecting it have a total of 3 connections from either > of 4 servers but I can easily see 12 connections in all. > > > > Can somebody shed some light on it. > > > > Please let me know if you need more information. > > > Hello, Which version pgpool you are using? How did you balance the reading by pgpool? Should all readings go to the replicas? or reading goes to all nodes? Regards, Glauco Torres
