PostgreSQL 10.20 crashes / Antivirus
I've been having problems with several PostgreSQL crashes over the last month (originally with 10.16, then more recently with 10.20, all on a Windows Server 2012R2 platform), always with a low level permissions error: "server process (PID 5664) was terminated by exception 0xC022" (This NT error translates as "Access Denied: A process has requested access to an object but has not been granted those access rights"). I was certainly having issues earlier with McAfee Antivirus causing some of these problems but this has been reduced after following the advice here to prevent scanning of PGDATA folders and postgres.exe (if someone can confirm that reference to "postgresql.exe" is a typo on that page I will correct it, but don't want to in case it refers to a different version of PostgreSQL): https://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows#Antivirus_software I have also got our IT to disable antivirus on our tablespace folder and on the folder running our backup PowerShell script (which calls pg_dump). The last few crashes have occurred during the backup process, with the most recent one apparently after one database pg_dump completes (successfully) but before the next database pgdump can begin - although the new pg_dump log file was created (is empty) it does not get as far as creating the .backup file and the PostgreSQL log file shows no new operations (I have logging set quite verbose including log_min_duration_statement = 0to log every statement). I have copied out the bit running up to the failure at the end of this email where you can see the final part (dumping of sde_xml_indexes) of the pg_dump on database a9sde, then it disconnecting from that database. Any suggestions for how to fix/diagnose? Some things I have tried already: * Ran "sfc" (System File Checker) to check for/fix any Windows OS corruptions (none found that were relevant) * Ran "amcheck" on several of the databases * Unfortunately checksums are not turned on with the database (not an option with the EDB PostgreSQL Windows Installer) and can't be turned on after initdb on version 10 * I hoped to run pg_catcheck but cannot find a binary version of this online and I think it would take me quite a lot of work to build it myself (I don't have Visual Studio, only VSC). Richard 2022-04-02 02:14:53 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOG: 0: duration: 0.816 ms statement: COPY sde.sde_xml_indexes (index_id, index_name, owner, index_type, description) TO stdout; 2022-04-02 02:14:53 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOCATION: exec_simple_query, postgres.c:1200 2022-04-02 02:14:54 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOG: 0: disconnection: session time: 0:36:52.430 user=scde_suad database=a9sde host=a9dualling.atkinsgeospatial.com port=59467 2022-04-02 02:14:54 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOCATION: log_disconnections, postgres.c:4577 2022-04-02 02:14:55 BST: [5016] LOG: 0: server process (PID 100) was terminated by exception 0xC022 At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you. NOTICE - This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.
RE: PostgreSQL 10.20 crashes / Antivirus
Ah... no - the AV is still allowed to run riot in the folder where the dump file is written to. Do you think that could be the issue? Richard -Original Message- From: Adrian Klaver Sent: 06 April 2022 15:42 To: Thomas, Richard ; pgsql-general@lists.postgresql.org Subject: Re: PostgreSQL 10.20 crashes / Antivirus On 4/6/22 03:59, Thomas, Richard wrote: > I’ve been having problems with several PostgreSQL crashes over the > last month (originally with 10.16, then more recently with 10.20, all > on a Windows Server 2012R2 platform), always with a low level permissions > error: > > "server process (PID 5664) was terminated by exception 0xC022" > > (This NT error translates as "Access Denied: A process has requested > access to an object but has not been granted those access rights"). > > I have also got our IT to disable antivirus on our tablespace folder > and on the folder running our backup PowerShell script (which calls pg_dump). > Is either one of those directories where the dump file is being output to? If not is the directory that file is being created in have AV checks disabled? -- Adrian Klaver adrian.kla...@aklaver.com At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you. NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.
RE: PostgreSQL 10.20 crashes / Antivirus
A request for further suggestions for how to fix/diagnose this PG crash (having just got back from holiday I discover that PG is still crashing even though I have changed the pgdump target output location to a folder excluded from the McAfee Antivirus scanning - but thanks for the suggestion Adrian). The last few crashes have occurred only during the backup process, when database pg_dump completes (successfully - including the disconnection message in the log) but before the next database pgdump can begin (that pg_dump command gets the error that it cannot connect to the database as the server is no longer responding). This will happen after dumping several databases - sometimes it manages the complete multiple database dump fine. One new thing I've noticed is that when it fails the file access 0xC022 error occurs a second time when PG is trying to restart itself after shutting down all the other processes, suggesting whatever caused the crash is still around (a later manual restart works fine): 2022-04-15 03:54:06 BST: [3720] WARNING: 57P02: terminating connection because of crash of another server process 2022-04-15 03:54:06 BST: [3720] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2022-04-15 03:54:06 BST: [3720] HINT: In a moment you should be able to reconnect to the database and repeat your command. 2022-04-15 03:54:06 BST: [3720] LOCATION: quickdie, postgres.c:2622 2022-04-15 03:54:06 BST: [7900] LOG: 0: all server processes terminated; reinitializing 2022-04-15 03:54:06 BST: [7900] LOCATION: PostmasterStateMachine, postmaster.c:3924 2022-04-15 03:54:08 BST: [7900] LOG: 0: startup process (PID 3652) was terminated by exception 0xC022 2022-04-15 03:54:08 BST: [7900] HINT: See C include file "ntstatus.h" for a description of the hexadecimal value. 2022-04-15 03:54:08 BST: [7900] LOCATION: LogChildExit, postmaster.c:3650 2022-04-15 03:54:08 BST: [7900] LOG: 0: aborting startup due to startup process failure 2022-04-15 03:54:08 BST: [7900] LOCATION: reaper, postmaster.c:2907 2022-04-15 03:54:09 BST: [7900] LOG: 0: database system is shut down 2022-04-15 03:54:09 BST: [7900] LOCATION: UnlinkLockFiles, miscinit.c:765 Some details from my original email (as it was 2 weeks ago): I’ve been having problems with several PostgreSQL crashes over the last month (originally with 10.16, then more recently with 10.20, all on a Windows Server 2012R2 platform), always with a low level permissions error: "server process (PID 5664) was terminated by exception 0xC022" (This NT error translates as "Access Denied: A process has requested access to an object but has not been granted those access rights"). I was certainly having issues earlier with McAfee Antivirus causing some of these problems but this has been reduced after following the advice here to prevent scanning of PGDATA folders and postgres.exe (if someone can confirm that reference to “postgresql.exe” is a typo on that page I will correct it, but don’t want to in case it refers to a different version of PostgreSQL): https://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows#Antivirus_software I have also got our IT to disable antivirus on our tablespace folder and on the folder running our backup PowerShell script (which calls pg_dump). Any suggestions for how to fix/diagnose? Some things I have tried already: - Ran “sfc” (System File Checker) to check for/fix any Windows OS corruptions (none found that were relevant) - Ran “amcheck” on several of the databases - Unfortunately checksums are not turned on with the database (not an option with the EDB PostgreSQL Windows Installer) and can’t be turned on after initdb on version 10 - I hoped to run pg_catcheck but cannot find a binary version of this online and I think it would take me quite a lot of work to build it myself (I don’t have Visual Studio, only VSC). Richard 2022-04-02 02:14:53 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOG: 0: duration: 0.816 ms statement: COPY sde.sde_xml_indexes (index_id, index_name, owner, index_type, description) TO stdout; 2022-04-02 02:14:53 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOCATION: exec_simple_query, postgres.c:1200 2022-04-02 02:14:54 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOG: 0: disconnection: session time: 0:36:52.430 user=scde_suad database=a9sde host=a9dualling.atkinsgeospatial.com port=59467 2022-04-02 02:14:54 BST: [7480] user=scde_suad,db=a9sde,app=pg_dump,host=a9dualling.atkinsgeospatial.com LOCATION: log_disconnections, postgres.c:4577 2022-04-02 02:14:55 BST: [5016] LOG: 0: server process (PID 100) was terminated by exception 0xC022 At Atkins - m
RE: PostgreSQL 10.20 crashes / Antivirus
Adrian Klaver wrote:
> What are the actual commands you are using to do the above?
The command used in a PowerShell script (run with Windows task scheduler) to
dump each database should evaluate to:
"C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" -b -v -F c -d $dbName -h
localhost -p 6488 -U backup_su -f $backupFile 2`>`&1 | Out-File $pgdumpLogFile
where:
- "backup_su" is a superuser role (with password stored in the user's
pgpass.conf file)
- $backupFile and $pgdumpLogFile are in folders excluded from McAfee scanning
- pg_dump.exe executable is not excluded from McAfee on-access scanning
(although as recommended postgres.exe is)
The actual script segment building the command is:
$pgdumpCmd = "C:\Program Files\PostgreSQL\10\bin\pg_dump.exe"
$pgdumpArgs = @("-b", "-v",
"-F", "c",
"-d", $dbName,
"-h", "localhost",
"-p", "6488",
"-U", " backup_su",
"-f", $backupFile)
cmd /c $pgdumpCmd $pgdumpArgs 2`>`&1 | Out-File $pgdumpLogFile
Note that the .backup file for the first failing pg_dump (after several
successful ones) is now being produced but is of length 0, with the associated
pg_dump log file simply reading:
pg_dump: [archiver (db)] connection to database "dbexample" failed: could not
connect to server: Connection refused (0x274D/10061)
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 6488?
could not connect to server: Connection refused (0x274D/10061)
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 6488?
Richard
-Original Message-
From: Adrian Klaver
Sent: 19 April 2022 18:11
To: Thomas, Richard ;
pgsql-general@lists.postgresql.org
Subject: Re: PostgreSQL 10.20 crashes / Antivirus
On 4/19/22 04:59, Thomas, Richard wrote:
> A request for further suggestions for how to fix/diagnose this PG crash
> (having just got back from holiday I discover that PG is still crashing even
> though I have changed the pgdump target output location to a folder excluded
> from the McAfee Antivirus scanning - but thanks for the suggestion Adrian).
> The last few crashes have occurred only during the backup process, when
> database pg_dump completes (successfully - including the disconnection
> message in the log) but before the next database pgdump can begin (that
> pg_dump command gets the error that it cannot connect to the database as the
> server is no longer responding). This will happen after dumping several
> databases - sometimes it manages the complete multiple database dump fine.
> One new thing I've noticed is that when it fails the file access 0xC022
> error occurs a second time when PG is trying to restart itself after shutting
> down all the other processes, suggesting whatever caused the crash is still
> around (a later manual restart works fine):
From you original post:
"... although the new pg_dump log file was created (is empty) it does not get
as far as creating the .backup file ..."
What are the actual commands you are using to do the above?
At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the
world. Although I have sent this email at a time convenient for me, I don't
expect you to respond until it works for you.
NOTICE – This email message and any attachments may contain information or
material that is confidential, privileged, and/or subject to copyright or other
rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or
other use of or reliance on this message or anything contained therein is
strictly prohibited and may be unlawful. If you believe you may have received
this message in error, kindly inform the sender by return email and delete this
message from your system. Thank you.
RE: PostgreSQL 10.20 crashes / Antivirus
Adrian Klaver wrote: > On 4/20/22 01:06, Thomas, Richard wrote: > > The command used in a PowerShell script (run with Windows task scheduler) > to dump each database should evaluate to: > > > > "C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" -b -v -F c -d $dbName > > -h localhost -p 6488 -U backup_su -f $backupFile 2`>`&1 | Out-File > > $pgdumpLogFile > > Do you have large objects(blobs) in the database? No, but do have PostGIS geometries in almost all tables. > > - pg_dump.exe executable is not excluded from McAfee on-access > > scanning (although as recommended postgres.exe is) > > Why not? > > I would think the whole C:\Program Files\PostgreSQL\10\bin\ would be > excluded. I was following the instructions here: https://wiki.postgresql.org/wiki/Running_%26_Installing_PostgreSQL_On_Native_Windows#Antivirus_software I am not particularly familiar with how our McAfee is configured or operates (all under control of our IT, including it seems access to many of the logs). With the executable postgres.exe, they have specified that in McAfee as an executable not to include in "on-access scanning" (wherever the executable lives). This differs from the file read/write scanning where the folders to exclude are specified. I have put in a request earlier today to add exclusions for pg_dump.exe; maybe I'll hear back from IT in a day or so ;-( > What does the Windows event log show? Only events: - Info: "The Windows Error Reporting Service service entered the running state" (goes into a stopped state 2 minutes later) - (Error event messages generated by my PowerShell script) Not sure if WER is actually writing data somewhere (none of the "*\AppData\Local\Microsoft\Windows\WER\ReportArchive" folders on the server have any data from the last 2 years) > Same for the A/V software log. I can't find much separate McAfee log information beyond the last hour in "Endpoint Security" app. Previously (before I got the PostgreSQL backup PowerShell script removed from scanning, McAfee used to put messages in the Windows Event Viewer reporting an error that it was a violating one of its rules, but it would allow the operation to continue (no longer get that). Richard At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you. NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.
RE: PostgreSQL 10.20 crashes / Antivirus
Adrian (all), Thanks for your suggestions. Running the equivalent pg_dump commands at the command line instead of in a script works fine and generates no errors/warnings/McAfee messages in the Windows Event Viewer or on stdout/stderr. (Most days the backup script itself completes running multiple pg_dump without problem). I too suspect McAfee of causing the problem - even if it isn't consciously taking any action, it must be processing all file I/O accesses in order to be able to filter which ones it should do virus scanning on. Unfortunately it is a very painful process for me to get our IT to modify virus scanning or report back on AV messages (even if not errors), so any diagnostics I can perform help speed the process. (And unfortunately, my hands are currently tied on having to use an existing AWS EC2 Windows Server, much as I would prefer to use a Linux system or a cloud-based PaaS.) Richard > > Adrian Klaver wrote: > >> On 4/20/22 01:06, Thomas, Richard wrote: > > >>> - pg_dump.exe executable is not excluded from McAfee on-access > >>> scanning (although as recommended postgres.exe is) > >> > >> Why not? > >> > >> I would think the whole C:\Program Files\PostgreSQL\10\bin\ would be > >> excluded. > > > > I was following the instructions here: > > > > > I am not particularly familiar with how our McAfee is configured or > > operates (all under control of our IT, including it seems access to > > many of the logs). With the executable postgres.exe, they have > > specified that in McAfee as an executable not to include in "on-access > > scanning" (wherever the executable lives). This differs from the file > > read/write scanning where the folders to exclude are specified. I have > > put in a request earlier today to add exclusions for pg_dump.exe; > > maybe I'll hear back from IT in a day or so ;-( > > That link also has: > > "Specific issues have also been reported with McAfee and Panda anti-virus > software and NetLimiter network monitoring software. While some people do > have PostgreSQL working with these software packages, there is no specific or > even recommend solutions that have not worked in some cases, so the issues > would appear to be installation specific, sometimes even requiring > uninstallation. " > > Now the last edit to the page was 2013 so not sure how relevant the above is > anymore. Also, I don't use Windows anymore(other then occasionally working > on peoples machines) so I am not the best person to comment on current > reality. Still the Postgres process seems to be killed by an external program > and > my suspicion is that the AV software is involved. Looks like this is > something the > Sys admins are going to have to look into via the tools and files they have > access to. At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you. NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.
10.22 Windows binaries download? (zip "invalid" on Enterprisedb)
I'm trying to download the Windows x64 binaries only for PostgreSQL 10.22 in order to perform a minor upgrade. However, the download (postgresql-10.22-1-windows-x64-binaries.zip) from my usual go-to place: (https://www.enterprisedb.com/download-postgresql-binaries) gives me a file that Windows zip declares invalid. Other zip files (e.g. the 32-bit version) download and unzip quite happily. I've dropped Enterprise an email, but in the meantime wonder if there are any other (reputable!) binaries download sites I could use? Richard At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you. NOTICE - This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.
RE: 10.22 Windows binaries download? (zip "invalid" on Enterprisedb)
Thanks very much - 7-zip (from https://www.7-zip.org/) was just the job to unpack the archive where Windows "Extract all" failed. Richard > -Original Message- > From: Erik Wienhold > Sent: 21 September 2022 14:44 > To: Thomas, Richard ; pgsql- > gene...@lists.postgresql.org > Subject: Re: 10.22 Windows binaries download? (zip "invalid" on Enterprisedb) > > > On 21/09/2022 15:34 CEST Thomas, Richard > wrote: > > > > I’m trying to download the Windows x64 binaries only for PostgreSQL 10.22 in > order to perform a minor upgrade. However, the download > (https://urldefense.com/v3/__http://postgresql-10.22-1-windows-x64- > binaries.zip__;!!OepYZ6Q!8OcLEO8KpfZpItfLWpOliejGRWOyHm9zuZBvTxc6ifxC > c7xA5i3TOGADp-wurt7EIz-8Z3QQj5_0fmh2l21mMw$ ) from my usual go-to > place: (https://urldefense.com/v3/__https://www.enterprisedb.com/download- > postgresql- > binaries__;!!OepYZ6Q!8OcLEO8KpfZpItfLWpOliejGRWOyHm9zuZBvTxc6ifxCc7x > A5i3TOGADp-wurt7EIz-8Z3QQj5_0fmjyao_6kg$ ) gives me a file that Windows > zip declares invalid. > > There's already a thread on that issue: > > https://urldefense.com/v3/__https://www.postgresql.org/message- > id/flat/CA*2BRCjaw*3DGH-Lyxe5*3D5-un6sQjO- > cQjhTeBDomiXoWQDrvVYwxA*40mail.gmail.com__;JSUlJQ!!OepYZ6Q!8OcLEO > 8KpfZpItfLWpOliejGRWOyHm9zuZBvTxc6ifxCc7xA5i3TOGADp-wurt7EIz- > 8Z3QQj5_0fmhPwsTSbA$ > > -- > Erik At Atkins - member of the SNC-Lavalin Group, we work flexible hours around the world. Although I have sent this email at a time convenient for me, I don't expect you to respond until it works for you. NOTICE – This email message and any attachments may contain information or material that is confidential, privileged, and/or subject to copyright or other rights. Any unauthorized viewing, disclosure, retransmission, dissemination, or other use of or reliance on this message or anything contained therein is strictly prohibited and may be unlawful. If you believe you may have received this message in error, kindly inform the sender by return email and delete this message from your system. Thank you.
