Hanging cygwin processes in Windows Server 2019
My cygwin install looks good and things seem to work perfectly for some time (hours or days), then a cygwin process started by a windows process that is usually a child process of a windows service written in perl will hang and then most all other cygwin processes will hang or take a very long time to return. The simplest example is a call to \cygwin\bin\ps from powershell. That normally returns with sub-second response time. Once the problem occurs that call to \cygwin\bin\ps will often take several minutes to return. I've found that killing all cygwin processes will temporarily resolve the problem. I've read about redirecting NUL to stdin when making the call to cygwin processes from windows and that didn't seem to help. I've tried "set-processmitigation -name \cygwin\bin\ps.exe -disable ForceRelocateImages" with no real benefit. I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup before the sshd service starts and that didn't seem to help. Any suggestions on how to prevent this problem will be greatly appreciated. Thanks!chris $ cygcheck -v /usr/bin/perl C:\Windows\Cygwin\bin\perl.exe - os=4.0 img=1.0 sys=4.0 "perl.exe" v0.0 ts=2021-02-06 11:46 C:\Windows\Cygwin\bin\cygperl5_32.dll - os=4.0 img=1.0 sys=4.0 "cygperl5_32.dll" v0.0 ts=2021-02-06 11:44 C:\Windows\Cygwin\bin\cygcrypt-2.dll - os=4.0 img=1.0 sys=4.0 "cygcrypt-2.dll" v0.0 ts=2021-05-20 09:20 C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0 "cygwin1.dll" v0.0 ts=2021-03-29 09:37 C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0 "KERNEL32.dll" v0.0 C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0 "ntdll.dll" v0.0 ts=2010-05-08 14:49 C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0 "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\ntdll.dll (already done) C:\Windows\Cygwin\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0 "cyggcc_s-1.dll" v0.0 ts=2020-09-12 12:49 C:\Windows\Cygwin\bin\cygwin1.dll (already done) C:\Windows\system32\KERNEL32.dll (already done) C:\Windows\system32\KERNEL32.dll (already done) C:\Windows\Cygwin\bin\cygwin1.dll (already done) C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done) C:\Windows\system32\KERNEL32.dll (already done) C:\Windows\Cygwin\bin\cygwin1.dll (already done) C:\Windows\system32\KERNEL32.dll (already done) $ cygcheck -v /usr/bin/ps C:\Windows\Cygwin\bin\ps.exe - os=4.0 img=1.0 sys=4.0 C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0 "cygwin1.dll" v0.0 ts=2021-03-29 09:37 C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0 "KERNEL32.dll" v0.0 C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0 "ntdll.dll" v0.0 ts=2010-05-08 14:49 C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0 "KERNELBASE.dll" v0.0 ts=1979-12-01 09:56 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\ADVAPI32.dll - os=10.0 img=10.0 sys=10.0 "ADVAPI32.dll" v0.0 C:\Windows\system32\msvcrt.dll - os=10.0 img=10.0 sys=10.0 "msvcrt.dll" v0.0 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\KERNELBASE.dll (already done) C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\KERNELBASE.dll (already done) C:\Windows\system32\SECHOST.dll - os=10.0 img=10.0 sys=10.0 "SECHOST.dll" v0.0 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\RPCRT4.dll - os=10.0 img=10.0 sys=10.0 "RPCRT4.dll" v0.0 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\SspiCli.dll - os=10.0 img=10.0 sys=10.0 "SspiCli.dll" v0.0 ts=2036-05-08 21:49 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\RPCRT4.dll (recursive) C:\Windows\system32\CRYPTBASE.dll - os=10.0 img=10.0 sys=10.0 "CRYPTBASE.dll" v0.0 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\bcryptPrimitives.dll - os=10.0 img=10.0 sys=10.0 "bcryptPrimitives.dll" v0.0 C:\Windows\system32\ntdll.dll (already done) C:\Windows\system32\KERNELBASE.dll (already done) C:\Windows\system32\KERNEL32.dll (already done) C:\Windows\system32\RPCRT4.dll (already done) C:\Windows\system32\KERNEL32.dll (already done) C:\Windows\system32\ntdll.dll (already done) $ cygcheck -v /usr/bin/ssh C:\Windows\Cygwin\bin\ssh.exe - os=4.0 img=1.0 sys=4.0 C:\Windows\Cygwin\bin\cygcrypto-1.1.dll - os=4.0 img=1.0 sys=4.0 "cygcrypto-1.1.dll" v0.0 ts=2020-04-08 19:02 C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0 "cygwin1.dll" v0.0 ts=2021-03-29 09:37 C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 s
Re: Hanging cygwin processes in Windows Server 2019
The problem seems to be that my cygwin processes that are always the child or
grandchild of a windows service use a console device that is block buffered
rather than character buffered. So using mintty to start those processes
forces character buffered which solves the problem. I'm looking for cleaner
way to force character buffered IO. Here's an example of the change:
From
chomp(@output = `/cygwin/bin/ps -W`);TO
chomp(@output = `/cygwin/bin/mintty.exe -w hide /bin/dash -c "/bin/ps -W >
/tmp/${PID}.out"; sleep 1; type /cygwin/tmp/${PID}.out; del
/cygwin/tmp/${PID}.out`);
Hopefully someone can suggest a simpler solution.
Thanks!chris
On Thursday, September 30, 2021, 01:26:37 PM EDT, chris.hardison--- via
Cygwin wrote:
My cygwin install looks good and things seem to work perfectly for some time
(hours or days), then a cygwin process started by a windows process that is
usually a child process of a windows service written in perl will hang and then
most all other cygwin processes will hang or take a very long time to return.
The simplest example is a call to \cygwin\bin\ps from powershell. That normally
returns with sub-second response time. Once the problem occurs that call to
\cygwin\bin\ps will often take several minutes to return. I've found that
killing all cygwin processes will temporarily resolve the problem.
I've read about redirecting NUL to stdin when making the call to cygwin
processes from windows and that didn't seem to help.
I've tried "set-processmitigation -name \cygwin\bin\ps.exe -disable
ForceRelocateImages" with no real benefit.
I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup before the sshd
service starts and that didn't seem to help.
Any suggestions on how to prevent this problem will be greatly appreciated.
Thanks!chris
$ cygcheck -v /usr/bin/perl
C:\Windows\Cygwin\bin\perl.exe - os=4.0 img=1.0 sys=4.0
"perl.exe" v0.0 ts=2021-02-06 11:46
C:\Windows\Cygwin\bin\cygperl5_32.dll - os=4.0 img=1.0 sys=4.0
"cygperl5_32.dll" v0.0 ts=2021-02-06 11:44
C:\Windows\Cygwin\bin\cygcrypt-2.dll - os=4.0 img=1.0 sys=4.0
"cygcrypt-2.dll" v0.0 ts=2021-05-20 09:20
C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
"cygwin1.dll" v0.0 ts=2021-03-29 09:37
C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
"KERNEL32.dll" v0.0
C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
"ntdll.dll" v0.0 ts=2010-05-08 14:49
C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
"KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\Cygwin\bin\cyggcc_s-1.dll - os=4.0 img=1.0 sys=4.0
"cyggcc_s-1.dll" v0.0 ts=2020-09-12 12:49
C:\Windows\Cygwin\bin\cygwin1.dll (already done)
C:\Windows\system32\KERNEL32.dll (already done)
C:\Windows\system32\KERNEL32.dll (already done)
C:\Windows\Cygwin\bin\cygwin1.dll (already done)
C:\Windows\Cygwin\bin\cyggcc_s-1.dll (already done)
C:\Windows\system32\KERNEL32.dll (already done)
C:\Windows\Cygwin\bin\cygwin1.dll (already done)
C:\Windows\system32\KERNEL32.dll (already done)
$ cygcheck -v /usr/bin/ps
C:\Windows\Cygwin\bin\ps.exe - os=4.0 img=1.0 sys=4.0
C:\Windows\Cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
"cygwin1.dll" v0.0 ts=2021-03-29 09:37
C:\Windows\system32\KERNEL32.dll - os=10.0 img=10.0 sys=10.0
"KERNEL32.dll" v0.0
C:\Windows\system32\ntdll.dll - os=10.0 img=10.0 sys=10.0
"ntdll.dll" v0.0 ts=2010-05-08 14:49
C:\Windows\system32\KERNELBASE.dll - os=10.0 img=10.0 sys=10.0
"KERNELBASE.dll" v0.0 ts=1979-12-01 09:56
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\ADVAPI32.dll - os=10.0 img=10.0 sys=10.0
"ADVAPI32.dll" v0.0
C:\Windows\system32\msvcrt.dll - os=10.0 img=10.0 sys=10.0
"msvcrt.dll" v0.0
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\KERNELBASE.dll (already done)
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\KERNELBASE.dll (already done)
C:\Windows\system32\SECHOST.dll - os=10.0 img=10.0 sys=10.0
"SECHOST.dll" v0.0
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\RPCRT4.dll - os=10.0 img=10.0 sys=10.0
"RPCRT4.dll" v0.0
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\system32\SspiCli.dll - os=10.0 img=10.0 sys=10.0
"SspiCli.dll" v0.0 ts=2036-05-08 21:49
C:\Windows\system32\ntdll.dll (already done)
C:\Windows\sys
Re: Hanging cygwin processes in Windows Server 2019
Brian,
Thanks for the feedback. I came across stdbuf earlier today and thought for
sure that was the answer but when I showed it to my co-worker he said he had
already tied it with no luck, both with -i0 -e0 -o0 and with -oL.
Actually, what is working is:`\\cygwin\\bin\\mintty.exe -w hide /bin/dash -c
"/bin/ps -W > /tmp/${PID}.out"`;
chomp(@output = `type \\cygwin\\tmp\\${PID}.out && del
\\cygwin\\tmp\\${PID}.out`);
I'm still hoping for a better solution.
Thanks!
chris
On Tuesday, October 19, 2021, 12:11:42 PM EDT, Brian Inglis
wrote:
On 2021-10-18 06:01, chris.hardison wrote:
> On Thursday, September 30, 2021, 01:26:37 PM EDT, chris.hardison wrote:
>> My cygwin install looks good and things seem to work perfectly for
some time (hours or days), then a cygwin process started by a windows
process that is usually a child process of a windows service written in
perl will hang and then most all other cygwin processes will hang or
take a very long time to return. The simplest example is a call to
\cygwin\bin\ps from powershell. That normally returns with sub-second
response time. Once the problem occurs that call to \cygwin\bin\ps will
often take several minutes to return. I've found that killing all
cygwin processes will temporarily resolve the problem.
>> I've read about redirecting NUL to stdin when making the call to
cygwin processes from windows and that didn't seem to help.
>> I've tried "set-processmitigation -name \cygwin\bin\ps.exe
>> -disable ForceRelocateImages" with no real benefit.
>> I've also tried "\cygwin\bin\dash -c /bin/rebaseall" on startup
before the sshd service starts and that didn't seem to help.
>> Any suggestions on how to prevent this problem will be greatly
appreciated.
> The problem seems to be that my cygwin processes that are always the
child or grandchild of a windows service use a console device that is
block buffered rather than character buffered. So using mintty to start
those processes forces character buffered which solves the problem. I'm
looking for cleaner way to force character buffered IO. Here's an
example of the change:
> From
> chomp(@output = `/cygwin/bin/ps -W`);TO
> chomp(@output = `/cygwin/bin/mintty.exe -w hide /bin/dash -c "/bin/ps -W >
> /tmp/${PID}.out"; sleep 1; type /cygwin/tmp/${PID}.out; del
> /cygwin/tmp/${PID}.out`);
> Hopefully someone can suggest a simpler solution.
Most Cygwin processes don't care about buffering, as they may be line or
character buffered from the terminal but block buffered when used with
pipe or file I/O.
Try using stdbuf(1) instead of mintty to change buffering.
You use it as a command prefix like nohup or time.
Perl may also have ways of changing I/O buffering internally.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation: https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
