printf builtin: '*' broken in conversion spec field width and precision (e.g., "%.*s")

[Rob Robason]

Sorry I couldn't use bugbash - I don't work with source.

I've attached a complete description of the bug.

Regards,

Rob Robason
Driving Breakthrough Performance in Software Product Quality and
Development Practices

   Phone: 
925-825-1512
Home: 
Concord, CA 94518
  Mobile: 
925-348-1512
 Website: 
http://rob.robason.net
   Email: 
r...@robason.net

View Rob Robason's
profile on
LinkedIn

printf: '*' broken in conversion spec field width and precision (e.g., "%.*s")

GNU bash, version 4.0.23(1)-release (i386-redhat-linux-gnu) on Fedora 11
(additional configuration info appears at the bottom of the page)


The printf builtin is broken in many variations of use of '*' (e.g., "%.*s") 
in a printf conversion spec to set field width and precision from argument 
values.

The following (bash builtin) command returns an empty line. E.g.:
  $ printf "%.*s\n" 5 "Sample string"

  $

Compare to correct behavior of printf(1) (/usr/bin/printf) with same args
  $ /usr/bin/printf "%.*s\n" 5 "Sample string"
  Sampl
  $

The builtin works only with either "*.*" or "m.m", (where m is an int), but 
not with any of the following legal usages: ".*", "*.", "m.*", "*.m", which 
are all legal variations. I haven't checked the positional "*m$" variants. 
Although I encountered this working with strings, the same problem exists for 
specifiers other than 's'. See "Other examples", below.

The builtin should work as defined for printf(1) and printf(3) (plus of course 
its own added features defined in bash(1)). 

Defined Behavior (man pages, printf(3) gives the most complete picture):
bash(1):
  [in section SHELL BUILTIN COMMANDS, printf]
"printf [-v var] format [arguments]"
"The format is a character string which contains ... format specifications, 
  each of which causes printing of the next successive argument. In 
  addition to the standard printf(1) formats, ..." 
[no relevant format specifier detail provided]
printf(1): 
  "Interpreted sequences are: ...
  "and all C format specifications ending with one of diouxXfeEgGcs, ... 
  "Variable widths are handled."
  
printf(3):
  [in section "Format of the format string""]
"Each conversion specification is introduced by the character %, and 
  ends with a conversion specifier. In between there may be (in this order) 
  zero or more flags, an _optional_ minimum field width, an _optional_ 
  precision and an optional length modifier."
  [in section "The precision"]:
"An optional precision, in the form of a period ('.') followed by an 
  optional decimal digit string. 
"Instead of a decimal digit string one may write "*" or "*m$" (for some 
  decimal integer m) to specify that the precision is given in the next 
  argument, or in the m-th argument, ...
"This gives the ... the maximum number of characters to be printed from 
  a string for s and S conversions.
  [in section "The conversion specifier, s"]:
"If a precision is specified, no more bytes than the number specified are 
  written, ...

Other examples:
  Other specifiers (f shown here) seem to share the bug:
  $ printf "%*.*f\n" 8 2 123.4567890 # "*.*" OK
123.46
  $ printf "%8.*f\n" 2 123.4567890   # "m.*" Broken
   123


Configuration info:
running Fedora 11 with all available updates installed

$ bash --version
GNU bash, version 4.0.23(1)-release (i386-redhat-linux-gnu)

$ uname -a
Linux [hostname masked for privacy].net 2.6.30.10-105.2.23.fc11.i686.PAE #1 SMP 
Thu Feb 11 07:05:37 UTC 2010 i686 athlon i386 GNU/Linux

$ sudo lshw
[hostname masked for privacy].net
description: Desktop Computer
product: GA-MA78GM-US2H
vendor: Gigabyte Technology Co., Ltd.
width: 32 bits
capabilities: smbios-2.4 dmi-2.4 smp-1.4 smp
configuration: boot=normal chassis=desktop cpus=2 uuid=30303234-[masked for 
privacy]
  *-core
   description: Motherboard
   product: GA-MA78GM-US2H
   vendor: Gigabyte Technology Co., Ltd.
   physical id: 0
   version: x.x
 *-firmware
  description: BIOS
  vendor: Award Software International, Inc.
  physical id: 0
  version: F3 (03/11/2009)
  size: 128KiB
  capacity: 960KiB
  capabilities: isa pci pnp apm upgrade shadowing cdboot bootselect 
socketedrom edd int13floppy360 int13floppy1200 int13floppy720 int13floppy2880 
int5printscreen int9keyboard int14serial int17printer int10video acpi usb agp 
ls120boot zipboot biosbootspecification
 *-cpu:0
  description: CPU
  product: AMD Athlon(tm) 7750 Dual-Core Processor
  vendor: Advanced Micro Devices [AMD]
  physical id: 4
  bus info: c...@0
  version: 15.2.3
  slot: Socket M2
  size: 1350MHz
  capacity: 3GHz
  width: 64 bits
  clock: 200MHz
... [more output available if needed]

echo interrupted by SIGCHLD from a dying coprocess

[Tomáš Trnka]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -
DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu' -
DCONF_VENDOR='unknown' -DLOCALEDIR='/home/trnka/opt/share/locale' -
DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -g -
O2
uname output: Linux a324-2 2.6.24.2 #1 SMP Wed Feb 20 12:36:17 CET 2008 x86_64 
GNU/Linux
Machine Type: x86_64-unknown-linux-gnu

Bash Version: 4.1
Patch Level: 2
Release Status: release

Description:
I've started using coprocesses heavily and I've found a nasty problem related 
(but not limited) to them: After the coprocess finishes its job, the resultant 
SIGCHLD is not properly blocked by bash signal processing logic and interferes 
with script I/O. In my case, I've been using something like:

read var1 var2 < <( a | long | pipeline | here)
echo "var1=$var1"
echo "var2=$var2"

Sometimes, the SIGCHLD arrived just when one of the echos were doing output 
and the result was:
echo: write error: Interrupted system call
As this is a bit of a race, it occurs only when the stars are right, i.e. 
during normal usage the probability of the SIGCHLD hitting exactly the echo is 
quite low. However, as soon as anything causes the I/O to take significantly 
longer, the bug appears. I've been hitting quite often (30%?) when running the 
script over SSH.

This bug has probably been reported years ago here:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=382798

Repeat-By:
I've reduced one of my scripts to this (nothing exceptionally intelligent, but 
it does the job):

#!/bin/bash

while [[ 1 ]]; do
set +e
read tmp tmp2 < <( echo "blabla" | wc | tr -s " " "\n" | tail -n 2 | tr "\n" " 
")
set -e
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
echo $tmp
echo $tmp2
done

Using this script I can reliably reproduce the bug (i.e. get a Interrupted 
system call error) using bash 4.1.2 (compiled myself from vanilla tarball) and 
3.1.17 (Debian lenny) over SSH and 4.0.35 (stock Fedora 12) under strace.

Fix:
Applying the following simple patch (against 4.1.2) fixes the bug:

--- builtins/echo.def.orig  2010-03-24 19:40:54.0 +0100
+++ builtins/echo.def   2010-03-24 19:47:07.0 +0100
@@ -27,6 +27,7 @@
 
 #include "../bashansi.h"
 
+#include 
 #include 
 #include "../shell.h"
 
@@ -108,6 +109,7 @@
 {
   int display_return, do_v9, i, len;
   char *temp, *s;
+  sigset_t nmask, omask;
 
   do_v9 = xpg_echo;
   display_return = 1;
@@ -159,6 +161,10 @@
 
   clearerr (stdout);   /* clear error before writing and testing success */
 
+  sigemptyset(&nmask);
+  sigaddset(&nmask, SIGCHLD);
+  sigprocmask(SIG_BLOCK, &nmask, &omask);
+
   terminate_immediately++;
   while (list)
 {
@@ -193,6 +199,8 @@
   if (display_return)
 putchar ('\n');
 
+  sigprocmask(SIG_SETMASK, &omask, NULL);
+
   terminate_immediately--;
   return (sh_chkwrite (EXECUTION_SUCCESS));
 }

pwd or shell prompt doesn't update after renaming a parent dir

[Ali Ghanavatian]

Configuration Information [Automatically generated, do not change]:
Machine: i486
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i486'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i486-pc-linux-gnu'
-DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H   -I.  -I../bash -I../bash/include -I../bash/lib   -g -O2
-Wall
uname output: Linux dark-star 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16
14:04:26 UTC 2009 i686 GNU/Linux
Machine Type: i486-pc-linux-gnu

Bash Version: 4.0
Patch Level: 33
Release Status: release

Description:
suppose there is a dir called "1" when i'm in it and i use "mv ../1
../44" command for example, mv does the job. but nor pwd or shell prompt
does not show the change until i go back to the parent of "1" directory.

Repeat-By:
it happens every time.

Fix:
i don't know about the source code at all, but if "pwd" or shell prompt
get their information more interactively from the file system, the bug
should be gone.

-- 
Sincerely
A. Ghanavatian

Re: printf builtin: '*' broken in conversion spec field width and precision (e.g., "%.*s")

[Greg Wooledge]

On Thu, Mar 25, 2010 at 03:32:25AM -0700, Rob Robason wrote:
> The printf builtin is broken in many variations of use of '*' (e.g., "%.*s") 
> in a printf conversion spec to set field width and precision from argument 
> values.

The use of '*' as a field width is not required by POSIX.  According to
:

"No provision is made in this volume of POSIX.1-2008 which allows
field widths and precisions to be specified as '*' since the '*'
can be replaced directly in the format operand using shell variable
substitution. Implementations can also provide this feature as an
extension if they so choose."

So, what you're asking for is a new feature; its absence is not a bug.

> Compare to correct behavior of printf(1) (/usr/bin/printf) with same args
>   $ /usr/bin/printf "%.*s\n" 5 "Sample string"
>   Sampl

On HP-UX 10.20:

imadev:~$ /usr/bin/printf "%.*s\n" 5 "Sample string"
printf:  Error printing 5

printf:  Error printing Sample string

> printf(1): 
>   "Interpreted sequences are: ...
>   "and all C format specifications ending with one of diouxXfeEgGcs, ... 
>   "Variable widths are handled."

That's the GNU implementation, which has apparently added the variable
width feature.  Bash hasn't added it (yet?).

Re: pwd or shell prompt doesn't update after renaming a parent dir

[Chet Ramey]

> Bash Version: 4.0
> Patch Level: 33
> Release Status: release
> 
> Description:
> suppose there is a dir called "1" when i'm in it and i use "mv ../1
> ../44" command for example, mv does the job. but nor pwd or shell prompt
> does not show the change until i go back to the parent of "1" directory.

By default, bash uses a logical view of the file system, in which $PWD
and pwd report the current directory using the path used to get there.
If you don't want bash to do that by default, use `set -o physical' or
`pwd -P'.  `pwd -P' will show the true physical path, but not modify
$PWD.

If you want bash to re-evaluate its idea of the current working
directory, which is reflected in $PWD, you have to force that
reevaluation.  Bash doesn't and can't know that the directory path has
changed underneath it.  You can use `cd .' to force bash to recompute
the current directory path if you change parent directories. 

Chet

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: printf builtin: '*' broken in conversion spec field width and precision (e.g., "%.*s")

[Chet Ramey]

On 3/25/10 6:32 AM, Rob Robason wrote:
> Sorry I couldn't use bugbash - I don't work with source.
> 
> I've attached a complete description of the bug.

Thanks for the report.  It's a simple bug, with a simple fix.  I've
attached a patch, but it may not be useful to non-source users.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
*** ../bash-4.1-patched/builtins/printf.def 2010-01-18 10:50:22.0 
-0500
--- builtins/printf.def 2010-03-25 09:40:56.0 -0400
***
*** 118,122 
nw = vflag ? vbprintf (f, fieldwidth, func) : printf (f, fieldwidth, 
func); \
  else if (have_precision) \
!   nw = vflag ? vbprintf (f, precision, func) : printf (f, fieldwidth, 
func); \
  else \
nw = vflag ? vbprintf (f, func) : printf (f, func); \
--- 118,122 
nw = vflag ? vbprintf (f, fieldwidth, func) : printf (f, fieldwidth, 
func); \
  else if (have_precision) \
!   nw = vflag ? vbprintf (f, precision, func) : printf (f, precision, 
func); \
  else \
nw = vflag ? vbprintf (f, func) : printf (f, func); \

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Eric Blake]

[please keep the list in the loop]

On 03/25/2010 12:20 AM, Siddhesh Poyarekar wrote:
> On Tue, Mar 23, 2010 at 9:30 PM, Siddhesh Poyarekar
>  wrote:
>>> I see no bug - bash is within its rights to pretend that an inherited
>>> ignored SIGINT has no trap setting, seeing as how the user cannot modify
>>> that through any use of trap.  That is, 'trap -p' is designed to output
>>> the text that will restore traps to their normal state, but since there
>>> is no way to change the state of SIGINT from being ignored, there is
>>> nothing needed in the output.
>>>
>>
>> Thanks for that explanation. I guess we can also detect if a signal is
>> being ignored by trying to set a trap for that signal and verify that
>> it has not been set. This was my original problem and I went about
>> trying to fix it in code rather than reading the docs carefully :)
>>
> 
> I was wondering if this could be added as a feature to the trap
> behaviour, perhaps a non-standard extension like 'trap -i' which lists
> ignored signals. If not, we could modify the manpage contents to read
> as:
> 
> "Signals ignored upon entry to the shell cannot be trapped, reset or listed."
> 
> so as to make this even clearer.

Technically, this is only a POSIX requirement on non-interactive shells.
 Interactive shells may reset ignored signals on entry, effectively
starting life as if nothing had been inherited ignored.  But I don't
know (without looking at the sources) whether bash explicitly modifies
inherited ignored signals in interactive shells.

> 
> I'm asking this because I have a query from a customer (I work in Red
> Hat Support: ) for this and I had originally
> written the patch for RHEL-5. I've already let him know that the patch
> in the original form has been rejected and he has reverted with these
> questions.

I'm not the bash maintainer.  It's up to Chet if he'd like to provide
such an extension in the upstream sources.  But the beauty of open
source is that you can use your patch, whether or not Chet applies it
upstream; it's just that it becomes more of a maintenance burden on you
if you have to maintain it as a fork.

-- 
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org

-- 
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Siddhesh Poyarekar]

On Thu, Mar 25, 2010 at 7:19 PM, Eric Blake  wrote:
> Technically, this is only a POSIX requirement on non-interactive shells.
>  Interactive shells may reset ignored signals on entry, effectively
> starting life as if nothing had been inherited ignored.  But I don't
> know (without looking at the sources) whether bash explicitly modifies
> inherited ignored signals in interactive shells.

Bash does not do it; ignored signals remain ignored in the subshell. I
wanted to know if it would be possible to explicitly document it in
the manpage or provide a non-standard extension to display signals
that are 'not available to play with'.

-- 
Siddhesh Poyarekar
http://siddhesh.in

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Eric Blake]

On 03/25/2010 08:39 AM, Siddhesh Poyarekar wrote:
> On Thu, Mar 25, 2010 at 7:19 PM, Eric Blake  wrote:
>> Technically, this is only a POSIX requirement on non-interactive shells.
>>  Interactive shells may reset ignored signals on entry, effectively
>> starting life as if nothing had been inherited ignored.  But I don't
>> know (without looking at the sources) whether bash explicitly modifies
>> inherited ignored signals in interactive shells.
> 
> Bash does not do it; ignored signals remain ignored in the subshell. I
> wanted to know if it would be possible to explicitly document it in
> the manpage or provide a non-standard extension to display signals
> that are 'not available to play with'.

Subshells are different than new shells on what POSIX requires for
signal manipulations.  Again, quoting POSIX:

"When a subshell is entered, traps that are not being ignored are set to
the default actions. This does not imply that the trap command cannot be
used within the subshell to set new traps."

That is, if a non-interactive parent inherited a signal ignored, then
neither the parent nor the subshell can change it.  But if a
non-interactive parent inherits a default signal, then explicitly
ignores it, then subshells will inherit it ignored but can still reset
it back to default.

-- 
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Siddhesh Poyarekar]

On Thu, Mar 25, 2010 at 8:13 PM, Eric Blake  wrote:
> Subshells are different than new shells on what POSIX requires for
> signal manipulations.  Again, quoting POSIX:
>
> "When a subshell is entered, traps that are not being ignored are set to
> the default actions. This does not imply that the trap command cannot be
> used within the subshell to set new traps."
>
> That is, if a non-interactive parent inherited a signal ignored, then
> neither the parent nor the subshell can change it.  But if a
> non-interactive parent inherits a default signal, then explicitly
> ignores it, then subshells will inherit it ignored but can still reset
> it back to default.
>

I do this:

In a terminal, execute a script which does basically this:

trap '' SIGTERM
...
bash

According to the above explanation, the resultant bash prompt I get on
execution of this script will allow me to override the ignored SIGTERM
since it was set in its non-interactive parent. But that is not what
happens. I cannot override SIGTERM regardless of whether the parent
shell is interactive or not. But if I replace SIGTERM with SIGHUP, it
works the way you mentioned. The reason for this seems to be the way
the GETORIGSIG macro is defined:

#define GETORIGSIG(sig) \
  do { \
original_signals[sig] = (SigHandler *)set_signal_handler (sig, SIG_DFL); \
set_signal_handler (sig, original_signals[sig]); \
if (original_signals[sig] == SIG_IGN) \
  sigmodes[sig] |= SIG_HARD_IGNORE; \
  } while (0)

So any signal that GETORIGSIG is called for and is inherited as
ignored from parent will automatically be hard ignored. Due to this,
we get automatically hard ignored SIGTERM, SIGQUIT, SIGINT and SIGCHLD
whenever these signals are inherited as ignored.

It doesn't look as though the hard ignore belongs in GETORIGSIG.

-- 
Siddhesh Poyarekar
http://siddhesh.in

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Eric Blake]

On 03/25/2010 01:32 PM, Siddhesh Poyarekar wrote:
> In a terminal, execute a script which does basically this:
> 
> trap '' SIGTERM
> ...
> bash
> 
> According to the above explanation, the resultant bash prompt I get on
> execution of this script will allow me to override the ignored SIGTERM
> since it was set in its non-interactive parent.

No.  In that case, you ended up with an interactive shell, where POSIX
is silent on the expected behavior.  Or, put another way, by stating
nothing explicit about interactive shells, POSIX has implicitly stated
that the resulting bash prompt MAY, but not MUST, allow you to override
the ignored SIGTERM, and that you, as the interactive user, have no
guarantees on which behavior was chosen by bash.

On the other hand, change that last line to 'bash ./program', where
'./program' is a bash script, and then POSIX is clear that ./program
MUST be prohibited from changing SIGTERM because it is non-interactive
and inherited an ignored SIGTERM.

> So any signal that GETORIGSIG is called for and is inherited as
> ignored from parent will automatically be hard ignored.

Yes.  POSIX requires this for non-interactive shells, and does not
forbid it for interactive shells.  Technically, this code could be
modified to exempt interactive shells from hard-ignores, or left as is
by treating both classes of shells identically in keeping things
hard-ignored; either approach still meets POSIX.

-- 
Eric Blake   ebl...@redhat.com+1-801-349-2682
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Siddhesh Poyarekar]

On Fri, Mar 26, 2010 at 2:40 AM, Eric Blake  wrote:
> Yes.  POSIX requires this for non-interactive shells, and does not
> forbid it for interactive shells.  Technically, this code could be
> modified to exempt interactive shells from hard-ignores, or left as is
> by treating both classes of shells identically in keeping things
> hard-ignored; either approach still meets POSIX.
>

The code does explicitly hard ignore signals ignored when entering a
non-interactive shell; see initialize_terminating_signals() in sig.c.
The problem is that in addition to this, it ends up doing a hard
ignore for the special signals in GETORIGSIG() -- not for all
terminating signals, just the special signals.

I know POSIX does not say anything about it, but neither does the man
page. The result of this is confusing behaviour. It should either hard
ignore _all_ signals or none. I really don't see any reason for a hard
ignore for special signals via GETORIGSIG when we already have the
stuff in place in initialize_terminating_signals() that does the right
thing.

What's more, BeOS gets special treatment in all this; its hard ignore
flag is removed after GETORIGSIG() for SIGINT because BeOS sets SIGINT
to SIG_IGN by default:

#if defined (__BEOS__)
  /* BeOS sets SIGINT to SIG_IGN! */
  original_signals[SIGINT] = SIG_DFL;
  sigmodes[SIGINT] &= ~SIG_HARD_IGNORE;
#endif


-- 
Siddhesh Poyarekar
http://siddhesh.in

Re: [PATCH] trap -p does not display ignored signals inherited from parent by a bash process

[Siddhesh Poyarekar]

On Fri, Mar 26, 2010 at 11:20 AM, Siddhesh Poyarekar
 wrote:
> The code does explicitly hard ignore signals ignored when entering a
> non-interactive shell; see initialize_terminating_signals() in sig.c.
> The problem is that in addition to this, it ends up doing a hard
> ignore for the special signals in GETORIGSIG() -- not for all
> terminating signals, just the special signals.
>

Oops, forgot to add that GETORIGSIG is called for interactive as well
as non-interactive shells.


-- 
Siddhesh Poyarekar
http://siddhesh.in