Re: pgbench: could not connect to server: Resource temporarily unavailable
OK, here's some proposed patches. 0001 adds a para about how to raise the listen queue length. 0002 isn't quite related, but while writing 0001 I noticed a nearby use of /proc/sys/... which I thought should be converted to sysctl. IMO /proc/sys pretty much sucks, at least for documentation purposes, for multiple reasons: * It's unlike the way you do things on other platforms. * "man sysctl" will lead you to useful documentation about how to use that command. There's no obvious way to find documentation about /proc/sys. * It's not at all sudo-friendly. Compare sudo sh -c 'echo 0 >/proc/sys/kernel/randomize_va_space' sudo sysctl -w kernel.randomize_va_space=0 The former is a lot longer and it's far from obvious why you have to do it that way. * You have to think in sysctl terms anyway if you want to make the setting persist across reboots, which you almost always do. * Everywhere else in runtime.sgml, we use sysctl not /proc/sys. 0003 removes PG_SOMAXCONN. While doing that I noticed that this computation hadn't been touched throughout all the various changes fooling with exactly what gets counted in MaxBackends. I think the most appropriate definition for the listen queue length is now MaxConnections * 2, not MaxBackends * 2, because the other processes counted in MaxBackends don't correspond to incoming connections. I propose 0003 for HEAD only, but the docs changes could be back-patched. regards, tom lane diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 963b18ed85..1192faa6ae 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -1298,6 +1298,22 @@ default:\ linkend="guc-max-files-per-process"/> configuration parameter to limit the consumption of open files. + + +Another kernel limit that may be of concern when supporting large +numbers of client connections is the maximum socket connection queue +length. If more than that many connection requests arrive within a +very short period, some may get rejected before the postmaster can +service the requests, with those clients receiving unhelpful +connection failure errors such as Resource temporarily +unavailable. The default queue length limit is 128 on many +platforms. To raise it, adjust the appropriate kernel parameter +via sysctl, then restart the postmaster. +The parameter is variously named net.core.somaxconn +on Linux, kern.ipc.soacceptqueue on newer FreeBSD, +and kern.ipc.somaxconn on macOS and other BSD +variants. + diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 963b18ed85..1192faa6ae 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -1258,11 +1258,12 @@ default:\ - On Linux - /proc/sys/fs/file-max determines the - maximum number of open files that the kernel will support. It can - be changed by writing a different number into the file or by - adding an assignment in /etc/sysctl.conf. + On Linux the kernel parameter + fs.file-max determines the maximum number of open + files that the kernel will support. It can be changed with + sysctl -w fs.file-max=N. + To make the setting persist across reboots, add an assignment + in /etc/sysctl.conf. The maximum limit of files per process is fixed at the time the kernel is compiled; see /usr/src/linux/Documentation/proc.txt for diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 8a038d1b2a..1664fcee2a 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -4891,7 +4891,7 @@ SubPostmasterMain(int argc, char *argv[]) * If testing EXEC_BACKEND on Linux, you should run this as root before * starting the postmaster: * - * echo 0 >/proc/sys/kernel/randomize_va_space + * sysctl -w kernel.randomize_va_space=0 * * This prevents using randomized stack and code addresses that cause the * child process's memory map to be different from the parent's, making it diff --git a/src/backend/libpq/pqcomm.c b/src/backend/libpq/pqcomm.c index 8ff3be611d..7112e9751b 100644 --- a/src/backend/libpq/pqcomm.c +++ b/src/backend/libpq/pqcomm.c @@ -537,13 +537,11 @@ StreamServerPort(int family, const char *hostName, unsigned short portNumber, } /* - * Select appropriate accept-queue length limit. PG_SOMAXCONN is only - * intended to provide a clamp on the request on platforms where an - * overly large request provokes a kernel error (are there any?). + * Select appropriate accept-queue length limit. It seems reasonable + * to use a value similar to the maximum number of child processes + * that the postmaster will permit. */ - maxconn = MaxBackends * 2; - if (maxconn > PG_SOMAXCONN) - maxconn = PG_SOMAXCONN; + maxconn = MaxConnections * 2; err = listen(fd, maxconn); if (err < 0)
Re: pgbench: could not connect to server: Resource temporarily unavailable
Thanks for your input everyone! I wanted to confirm that increasing the somaxconn also fixed the issue for me. Kevin > $ cat /proc/sys/net/core/somaxconn > 128 > > by default, which is right about where the problem starts. After > > $ sudo sh -c 'echo 1000 >/proc/sys/net/core/somaxconn' > > *and restarting the PG server*, I can do a lot more threads without > a problem. Evidently, the server's socket's listen queue length > is fixed at creation and adjusting the kernel limit won't immediately > change it. > > >
Re: pgbench: could not connect to server: Resource temporarily unavailable
On Tue, Aug 23, 2022 at 4:57 AM Tom Lane wrote: > 0001 adds a para about how to raise the listen queue length. +service the requests, with those clients receiving unhelpful +connection failure errors such as Resource temporarily +unavailable. LGTM but I guess I would add "... or Connection refused"? > 0002 isn't quite related, but while writing 0001 I noticed a nearby > use of /proc/sys/... which I thought should be converted to sysctl. > IMO /proc/sys pretty much sucks, at least for documentation purposes, > for multiple reasons: +1 > 0003 removes PG_SOMAXCONN. While doing that I noticed that this > computation hadn't been touched throughout all the various > changes fooling with exactly what gets counted in MaxBackends. > I think the most appropriate definition for the listen queue > length is now MaxConnections * 2, not MaxBackends * 2, because > the other processes counted in MaxBackends don't correspond to > incoming connections. +1 > I propose 0003 for HEAD only, but the docs changes could be > back-patched. +1
Re: pgbench: could not connect to server: Resource temporarily unavailable
Just curious, *backlog* defines the maximum pending connections, why do we need to double the MaxConnections as the queue size? It seems *listen* with larger *backlog* will tell the OS maintain a larger buffer? - maxconn = MaxBackends * 2; - if (maxconn > PG_SOMAXCONN) - maxconn = PG_SOMAXCONN; + maxconn = MaxConnections * 2; On Tue, Aug 23, 2022 at 12:57 AM Tom Lane wrote: > > OK, here's some proposed patches. > > 0001 adds a para about how to raise the listen queue length. > > 0002 isn't quite related, but while writing 0001 I noticed a nearby > use of /proc/sys/... which I thought should be converted to sysctl. > IMO /proc/sys pretty much sucks, at least for documentation purposes, > for multiple reasons: > > * It's unlike the way you do things on other platforms. > > * "man sysctl" will lead you to useful documentation about how to > use that command. There's no obvious way to find documentation > about /proc/sys. > > * It's not at all sudo-friendly. Compare > sudo sh -c 'echo 0 >/proc/sys/kernel/randomize_va_space' > sudo sysctl -w kernel.randomize_va_space=0 > The former is a lot longer and it's far from obvious why you have > to do it that way. > > * You have to think in sysctl terms anyway if you want to make the > setting persist across reboots, which you almost always do. > > * Everywhere else in runtime.sgml, we use sysctl not /proc/sys. > > 0003 removes PG_SOMAXCONN. While doing that I noticed that this > computation hadn't been touched throughout all the various > changes fooling with exactly what gets counted in MaxBackends. > I think the most appropriate definition for the listen queue > length is now MaxConnections * 2, not MaxBackends * 2, because > the other processes counted in MaxBackends don't correspond to > incoming connections. > > I propose 0003 for HEAD only, but the docs changes could be > back-patched. > > regards, tom lane > -- Regards Junwang Zhao
Re: pgbench: could not connect to server: Resource temporarily unavailable
Junwang Zhao writes: > Just curious, *backlog* defines the maximum pending connections, > why do we need to double the MaxConnections as the queue size? The postmaster allows up to twice MaxConnections child processes to exist, per the comment in canAcceptConnections: * We allow more connections here than we can have backends because some * might still be authenticating; they might fail auth, or some existing * backend might exit before the auth cycle is completed. The exact * MaxBackends limit is enforced when a new backend tries to join the * shared-inval backend array. You can argue that 2X might not be the right multiplier, and you can argue that the optimal listen queue length might be more or less than the limit on number of child processes, but that's how we've historically done it. I'm not especially interested in changing that without somebody making a well-reasoned case for some other number. regards, tom lane
Re: pgbench: could not connect to server: Resource temporarily unavailable
Ok, thanks for the clarification. On Tue, Aug 23, 2022 at 11:37 AM Tom Lane wrote: > > Junwang Zhao writes: > > Just curious, *backlog* defines the maximum pending connections, > > why do we need to double the MaxConnections as the queue size? > > The postmaster allows up to twice MaxConnections child processes > to exist, per the comment in canAcceptConnections: > > * We allow more connections here than we can have backends because some > * might still be authenticating; they might fail auth, or some existing > * backend might exit before the auth cycle is completed. The exact > * MaxBackends limit is enforced when a new backend tries to join the > * shared-inval backend array. > > You can argue that 2X might not be the right multiplier, and you > can argue that the optimal listen queue length might be more or > less than the limit on number of child processes, but that's how > we've historically done it. I'm not especially interested in > changing that without somebody making a well-reasoned case for > some other number. > > regards, tom lane -- Regards Junwang Zhao
Re: pgbench: could not connect to server: Resource temporarily unavailable
Thomas Munro writes: > On Tue, Aug 23, 2022 at 4:57 AM Tom Lane wrote: > +service the requests, with those clients receiving unhelpful > +connection failure errors such as Resource temporarily > +unavailable. > LGTM but I guess I would add "... or Connection refused"? Is that the spelling that appears on FreeBSD? Happy to add it. regards, tom lane
Re: pgbench: could not connect to server: Resource temporarily unavailable
On Tue, Aug 23, 2022 at 3:53 PM Tom Lane wrote: > Thomas Munro writes: > > On Tue, Aug 23, 2022 at 4:57 AM Tom Lane wrote: > > +service the requests, with those clients receiving unhelpful > > +connection failure errors such as Resource temporarily > > +unavailable. > > > LGTM but I guess I would add "... or Connection refused"? > > Is that the spelling that appears on FreeBSD? Happy to add it. Yep.
