Ubuntu [Bug 791263] !$ works with `` but not $()
Hi, I've just reported what I think to be a bug on Ubuntu but it's present in pure bash 4.2 built from source too so I'm reporting it here. I'm not subscribed to the list so please CC me. I take it there's no public bug-tracker for bash and it's all done privately based on this list? If any Ubuntu bashers have the odd spare minute then https://bugs.launchpad.net/ubuntu/+source/bash/+bugs could do with a bit of love and attention. I've just marked invalid or fixed a few of the bugs on there but there's plenty left. Cheers, Ralph. Report follows. https://bugs.launchpad.net/ubuntu/+source/bash/+bug/791263 This is on 10.04 and also with upstream's pure bash 4.2 with no Debian or Ubuntu patches. $ echo bar >foo $ >bar $ ls `cat foo` bar $ wc !$ wc `cat foo` 0 0 0 bar $ ls $(cat foo) bar $ wc !$ wc ) -bash: syntax error near unexpected token `)' $ ls $(
Built-in printf Sits Awkwardly with UDP.
Hi, (I'm not subscribed to the list so please keep me CC'd.) Consider printf 'foo bar\n' >/dev/udp/localhost/4242 with bash's printf built-in, it works fine. One packet is sent. However, make that two lines printf 'foo\nbar\n' >/dev/udp/localhost/4242 and two packets are sent, one per line. If the aim is to send a string of data in one packet this causes problems. Using /usr/bin/printf from coreutils gives one write(2), so that's a possible workaround. Is this happening because the built-in printf is using putchar(3) in the PC() macro and stdio thinks file descriptor 1 is still to a tty so it's persisting in line buffering? It would seem nicer if fewer write(2)s were done when stdout isn't a tty, and not just for UDP use. Cheers, Ralph.
Re: Built-in printf Sits Awkwardly with UDP.
Hi Chet, > > On 7/18/2011 10:14 AM, Ralph Corderoy wrote: > > > Is this happening because the built-in printf is using putchar(3) > > > in the PC() macro and stdio thinks file descriptor 1 is still to a > > > tty so it's persisting in line buffering? It would seem nicer if > > > fewer write(2)s were done when stdout isn't a tty, and not just > > > for UDP use. > > Almost exactly right, except replace "unbuffered" with "line-buffered > output". Bash uses stdio and sets stdout and stderr to line-buffered. > The advantage of fully-buffered mode shows itself when writing large > amounts of data, which the shell does not do very often. The > advantage of line-buffered mode is better response. I see why it's line-buffered when writing to a terminal, but when bash changes where stdout points it has the option to setvbuf(3) or similar too based on what it knows about the destination, e.g. /dev/pts/3 versus /tmp/foo versus /dev/udp/0x7f01/4242. Does it never do this then, and just leave things as line-buffered all the time? Cheers, Ralph.
Re: Built-in printf Sits Awkwardly with UDP.
Hi Chet, > > I see why it's line-buffered when writing to a terminal, but when > > bash changes where stdout points it has the option to setvbuf(3) or > > similar too based on what it knows about the destination, e.g. > > /dev/pts/3 versus /tmp/foo versus /dev/udp/0x7f01/4242. Does it > > never do this then, and just leave things as line-buffered all the > > time? > > Bash ensures that stdout and stderr are line-buffered, and leaves them > that way. It's a guessing game otherwise. Say sockets and pipes both > present themselves the same way. Do you fully buffer, which reduces > pipe throughput and concurrency, unbuffer, which cuts performance > dramatically, or assume that line buffering is the right choice the > majority of the time? Thanks for the reply. I understand it's behaviour now and agree sockets and pipes could look the same. But a regular file ./foo on disk does look different and it still seems odd that printf '\n\n\n\n\n\n\n\n\n\n\n\n' >foo does a dozen one-byte write(2)s. Still, at least it explains the UDP behaviour. Thanks again, Ralph.
Re: Built-in printf Sits Awkwardly with UDP.
Hi Bob, > Ralph Corderoy wrote: > > ... But a regular file ./foo on disk does look different and it > > still seems odd that > > printf '\n\n\n\n\n\n\n\n\n\n\n\n' >foo > > does a dozen one-byte write(2)s. > > But the only reason you know that there is a long string of newlines > is that your eye is looking over the entire string. It has all of the > data all at once. You have a non-causal relationship with the data > because you are looking over all of the past history of the data. By > the time we see it here it has all already happened. You are not > looking at the data as it arrives. This is true, but has no bearing on the issue. :-) > In other words... Are you trying to suggest that a program should try > to look-ahead at future characters? No. Here's my understanding of how it currently works. The built-in printf works out a character to print at a time; no foresight needed or used. It asks the C library to print it with putchar(3). The C library can implement buffering to avoid too many expensive write(2) system calls. bash has the C library buffer stdout by lines. Each time putchar(3) is called the character is appended to the buffer. If the character was '\n' or the buffer is now full then write(2) is called to pass the buffer to the kernel and the buffer then treated as empty. When printf is finished the buffer is flushed, e.g. if there's anything in it then write(2) is called. The C library provides two other modes of buffering, see setvbuf(3). bash could easily detect that stdout is a regular file, have a new stream be duplicated from file descriptor 1, and set it to block-buffered. (POSIX doesn't permit a stream's buffering to change once the stream has been written to; but that's a downside of using the C library's I/O model.) That would lessen the number of write(2)s and allow a single UDP packet to be sent with print containing a '\n'. No change will be made, I see that. Just didn't want to leave the false suggestion that premonition is in any way involved. :-) BTW, the code for the built-in printf has a bug. For negative field-widths it negates a negative integer without checking it will fit. E.g. on this 64-bit machine $ printf '%-9223372036854775808s.\n' foo foo. $ Cheers, Ralph.
Re: Built-in printf Sits Awkwardly with UDP.
Hi Eric, > > $ printf '%-9223372036854775808s.\n' foo > > foo. > > $ > > Coreutils' printf shares this misfortune. Sadly, it might even be a > bug in the underlying glibc printf(), although I haven't tried to > write a test program to check that, yet. OK, well for %b and %q bash's built-in printf calls it's own printstr() and that does do things like `fw = -fw' without checking if fw was already the largest negative. Cheers, Ralph.
Re: Built-in printf Sits Awkwardly with UDP.
Hi André, > > When printf is finished the buffer is flushed, e.g. if there's > > anything in it then write(2) is called. > > If standard output is a log file, log entries could remain latent for > a very long time. I don't see why. I'm not asking for buffering across multiple calls to the built-in printf but for non-line buffering *within* the processing of one printf. printf 'foo\nbar\n' # One write(2) preferred, not the current two. printf foo # One write(1), printf bar # the second write(1). The stdio buffer should always be flushed at the end of each printf command, to do otherwise would break too many things and be non-sensical. It also wouldn't match the behaviour of calling the /usr/bin printf. Cheers, Ralph. P.S. Please keep me CC'd.
Re: Built-in printf Sits Awkwardly with UDP.
> OK, well for %b and %q bash's built-in printf calls it's own > printstr() and that does do things like `fw = -fw' without checking if > fw was already the largest negative. On a related note, I can't interrupt this, e.g. Ctrl-C. printf '%-92233720368547758q.\n' foo Cheers, Ralph. P.S. Please keep me CC'd.
Re: bug#9129: Built-in printf Sits Awkwardly with UDP.
Hi Jim, > > On 07/20/2011 07:34 AM, Ralph Corderoy wrote: > > > BTW, the code for the built-in printf has a bug. For negative > > > field-widths it negates a negative integer without checking it > > > will fit. E.g. on this 64-bit machine > > > > > > $ printf '%-9223372036854775808s.\n' foo > > > foo. > > > $ > > > > Coreutils' printf shares this misfortune. Sadly, it might even be a > > bug in the underlying glibc printf(), although I haven't tried to > > write a test program to check that, yet. > > This is not about a negative field width. Just to ensure there's no confusion, as far as bash's built-in printf is concerned one aspect of it is about a negative field width since $ printf '.%*q.\n' -10 foo .foo . is implemented by converting the "-10" into a number, spotting it's negative, setting the left-justified flag, and then negating the number. Coreutils' printf doesn't have that particular issue since it passes the negative number to the C library as a negative and has it deal with the justification change. For %b, which libc doesn't support, coreutils says /* FIXME: Field width and precision are not supported for %b, even though POSIX requires it. */ so there's no code to go wrong. :-) Cheers, Ralph.
Re: Built-in printf Sits Awkwardly with UDP.
Hi Chet, > On 7/22/11 10:38 AM, Ralph Corderoy wrote: > > On a related note, I can't interrupt this, e.g. Ctrl-C. > > > > printf '%-92233720368547758q.\n' foo > > That's interesting, since the fieldwidth (and precision) end up > getting set to 0 if they overflow INT_MAX, at least on my machine, and > the result is more or less instantaneous. I agree that bash should do > the same thing in printstr when presented with an overflowing field > width that it does when using getint(). I will look at making that > change. I also get, with vanilla 4.2 built from source, $ ./install/bin/bash -c "printf '%*q.\n' 10 foo -20 bar 0x8000" foo. bar . Segmentation fault (core dumped) $ Cheers, Ralph.
Enabling History Expansion with `set -H'.
Hi, Please keep me CC'd. bash package 4.4.012-2 on Arch Linux, `version 4.4.12(1)-release (x86_64-unknown-linux-gnu)'. I'm trying to enable history expansion in a non-interactive bash with `set -H'. $ printf '%s\n' ': foo' 'echo !!' 'set -H' ': bar' 'echo !!' | > bash !! !! $ I'd expect the second `!!' to be `: bar'. What am I misunderstanding? -- Cheers, Ralph. https://plus.google.com/+RalphCorderoy
bash Fails Glob Though No Glob Present.
Hi, bash 5.0.0-1 on Arch Linux. I'm manipulating a variable that's intended to hold a regular expression for vim(1), but bash thinks it fails pathname expansion even though I see no glob meta-character within it. $ shopt -s failglob $ ls -d foo ls: cannot access 'foo': No such file or directory $ ls foo/bar ls: cannot access 'foo/bar': No such file or directory $ ls foo\/bar ls: cannot access 'foo/bar': No such file or directory That's all as expected. I can escape the / and it remains a /. $ s='foo\/bar' Single quotes ensure $s contains the two characters \ then /. $ ls "$s" ls: cannot access 'foo\/bar': No such file or directory The \/ is preserved, as shown by ls's error. $ ls $s -bash: no match: foo\/bar $ Without the protection of the double-quotes, bash does pathname expansion, but it doesn't contain a *, ?, or [ as listed under `Pathname Expansion' in bash(1). -- Cheers, Ralph.
cd's Extraneous Arguments.
Hi, 4.3.42(1)-release `cd foo bar xyzzy' changes directory to foo, complaining if it doesn't exist. It doesn't complain about the unwanted extra arguments. This can cause problems; `cd *04' ends up in 2015-04, not 2016-04. Keith Thompson raised this in 2010, https://lists.gnu.org/archive/html/bug-bash/2010-12/msg00072.html and the list discussed that pushd, etc., also have this behaviour, but the thread quickly went way off topic and I didn't spot a resolution. I think cd should match the `help cd' output and insist on at most one `dir'. Why doesn't it do this? If the behaviour isn't going to change then the documentation should be altered to match. Please keep me CC'd; I'm not subscribed. Cheers, Ralph.
Re: cd's Extraneous Arguments.
Hi Chet, > > `cd foo bar xyzzy' changes directory to foo, complaining if it > > doesn't exist. It doesn't complain about the unwanted extra > > arguments. This can cause problems; `cd *04' ends up in 2015-04, > > not 2016-04. > > If you want this behavior, you can build bash with CD_COMPLAINS > defined (adding it to CFLAGS is the easiest way). Thanks, I didn't know of CD_COMPLAINS. But I'm not trying to get this fixed for just me, deviating from my distribution's bash, I think it should be fixed in bash's normal configuration so other users aren't caught out by it, and the damage that can follow. What's the argument for why CD_COMPLAINS behaviour isn't the default? The documentation doesn't match the current behaviour, matching instead POSIX in saying there's zero or one directories. Cheers, Ralph.
