Re: Tee and file redirections are very slow to write anything.

[Hamish McIntyre-Bhatty via Cygwin]

On 25/02/2021 07:50, Brian Inglis wrote:

On 2021-02-24 16:50, Brian Inglis wrote:

On 2021-02-24 15:41, Duncan Roe wrote:

On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote:

On 2/24/2021 3:48 PM, ASSI wrote:

Hamish McIntyre-Bhatty via Cygwin writes:

I found recently when trying to save output from a script for later
inspection that "tee" and file redirections seem to have massive
delays when run in Cygwin - usually nothing is written to file or
stdout until after the command has finished - not very helpful.

You will want to switch from fully buffered to line-buffered or even
unbuffered output.

And this does not have to do with Cygwin.  The same happens on Linux.
The default is that terminal I/O is unbuffered while other stream are

changes to get around this, but most programs won't override the
default behavior on their own ...

buffered.  Pipes come under "other streams".  One can make programmatic

The (Linux) default is that terminal I/O is *line* buffered


grep does.

The man page for tee doesn't show an option to change buffering, while that for

     $ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq

I believe the default for both Cygwin and Linux is 64KB pipe buffer, so if you want to see smaller chunks as they are generated, you need to add some utility that may allow you to change that e.g. but read the disclaimers on the stdbuf and grep man pages, which is why it is not done more, especially under Cygwin where Windows adds its own performance penalties. Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they can and don't care about text or lines, for more efficient access to disk files, rather than buffered stream I/O functions.

From what I have been able to find, Cygwin <stdio.h> BUFSIZ is only 1K, compared to Linux 8K, and Cygwin internal 64K, and that is used in many places in coreutils like tee, which will slow everything down by a factor of at least 8 plus increased overhead. Suggest <stdio.h> BUFSIZ be bumped to at least Linux value of 8K, if not 64K.

Perhaps:

#1: It depends on the program being run and how it sets its buffers up?

#2: stdbuf -o L enables line buffering and should fix my problem?



Hamish

So from the discussion above, I've not sure I fully understand why the behaviour is different on Cygwin to Linux for me, especially if Linux's buffer size is sometimes bigger. I'm now thinking it may have been a Python script that was behaving this way - I might go and double check in case this isn't what I think it is.

OpenPGP_0x18F1759B3457223F.asc
Description: application/pgp-keys

OpenPGP_signature
Description: OpenPGP digital signature

--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple