Dear Thomas, thanks for your detailed description. I could reproduce the behaviour on a 48 core machine. Even when running a dummy program (attached) that forks a lot and lets each fork simply output a line to stderr, I can reproduce the loss of output lines. For analysis I let the child processes also output the offset of stdout before and after the actual write() call to stderr; thus, calling the program via
( ./dummy 1000 100 > /tmp/list ) 2>&1 | cat > /tmp/offets
shows that the lost lines get overwritten by other forks that have
inherited the stdout file descriptor with the _same offset_.
When the dummy program (or multiple parallel running md5sum instances)
is outputting to a pipe, there is no file position available (and small
write() calls are guaranteed to be atomic).
Thus, the described behaviour is actually not a problem caused by
parallel; but though I think I will have to add a note about this
problem to the manpage and I probably will add an option that lets
parallel create an output pipe for stdout and stderr internally in
order to simply get rid of this problem.
Is this explanation acceptable for you?
Kind regards,
Nicolas
signature.asc
Description: Digital signature
