On 4/9/19 7:48 PM, L A Walsh wrote:
> ----
> I am aware of that, however, if a pipe implementation
> *stops* on reaching a full condition from some 'tmp-storage-space'
> and awaits for space to become available, a similar dynamic would
> apply. That's all.
Is there a pipe implementation out there that uses temporary files? And
if there is, should I worry about that less-than-the-lowest-common-
denominator case?
>
> Example: Suppose output from a program
> was buffered to disk files 64k in size. The reader
> process would get input from those buffers on disk and
> free the files as they are read. If the writer ran out of
> space, then sleeping and retrying the operation would make
> since, as it would be expected that the reader would be
> freeing blocks on disk as it read them. It's not always
> a safe assumption, but what else can it do?
This doesn't have anything to do with the issue being discussed.
>
>> | Using a file doesn't sequence -- the writer can still continue
>> | execution pass the point of bash possibly flagging an internal
>> | error for a non-existent tmp file (writable media) and the
>> | reader won't get that the "pipe" (file) had no successful writer,
>> | but instead get an EOF indication and continue, not knowing that
>> | a fatal error had just occurred.
>>
>> I doubt that is what happens.
>>
> ----
> That is what appeared to happen in the post mentioned by Chet.
> The boot process got a /dev/df/99 not found and continued on
> seemingly as though though there had been no input.
I'm not sure what this means. Failing to open a file isn't a fatal script
error.
>> | However, that would
>> | be code in the pipe implementation or an IO library on top
>> | of some StdIO implementation using such.
>>
>> Pipes are implemented in the kernel - userland does nothing different
>> at all (except the way they are created.)
>>
> ----
> They usually are. That doesn't prevent a stdlib implementation
> putting a wrapper around some "non-compliant" kernel call
> to implement a different 'view' to the users of that lib.
If we're going to go off into hypotheticals and speculation, it would be
nice if memfd_create were available universally.
>
>> | W/pipes, there is the race condition of the reader not being able
>> | to read in the condition where the writer has already gone away.
>>
>> Huh? That's nonsense. It is perfectly normal for a reader
>> to read long after the writer has finished and exited. Try this
>>
>> printf %s\\n hello | { sleep 5; cat; }
>>
> ===
> It may be normal in some cases, but:
>
> https://superuser.com/questions/554855/how-can-i-fix-a-broken-pipe-error
That case is exactly the opposite of the one being discussed here: it's a
write on a pipe when the reader has exited.
>> You are still missing Chet's point. There is no "< <()" operator.
>> That is two bash syntax elements being combined. "<" (redirect stdin)
>> and "<()" (create a name to refer to the output from the command).
>>
> ----
> I've never seen <() used without '<', so I thought it was
> part of the syntax '< <()'.
You can't claim ignorance here. You've been told many times that these
are two separate elements.
> We are talking tradeoffs of using pipes to communicate
> heredocs vs. using a temporary file (presumably in/on /tmp), no?
> The statement reflected my thinking about how, currently,
> the entire contents of the pipe is being "spilled to disk"
> (spilled in the sense of their being insufficient room in
> memory -- or, in this case of there being no 'in-memory'
> implementation at all).
This makes no sense the way you wrote it. If you mean the current
implementation of using temporary files for here documents, you're
correct.
>> | If bash uses /tmp, it can have a pipe of size 4.7G. If
>> | it uses memory, it would have pipe of 79G.
>>
>> That's gibberish.
>>
> Oh please, its not that obtuse. If bash currently writes the
> entire contents of "whatever" it is (the here doc), to a temporary
> file, then it is limited by the space on the temporary file system.
If you want to make that point, use `file' instead of `pipe'. What you
wrote doesn't make any sense otherwise. The word `pipe' has a specific
meaning in this context.
>
> But the implementation of process substitution in bash
> isn't implemented that way in the currently released version. It
> uses a tmp file on a disk of fixed size to store *all* of the output
> of the 'writer' before the reader is called.
This is absolutely not true. If you believe this, it might be a reason
you have made incorrect assumptions about other things. Process
substitution uses pipes: anonymous pipes exposed through /dev/fd or FIFOs.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/
