Re: Severe Bash Bug with Arrays

[Chet Ramey]

On 4/25/12 1:23 AM, Clark Wang wrote:

>> Bash Version: 4.2
>> Patch Level: 0
>> Release Status: release
>>
>> Description:
>>Cannot redirect into an array from multiline variable
>>
>> Does not work:
>> echo "$mydata" | while read line; do myarray+=( "$line" ); done
>>
> 
> Here the whole *while* statement runs in a subshell. See
> http://mywiki.wooledge.org/BashPitfalls#grep_foo_bar_.7C_while_read_-r.3B_do_.28.28count.2B-.2B-.29.29.3B_done.

Since you're using bash-4.2, you can experiment with the `lastpipe' shopt
option.  If that's enabled, bash runs the last element of a pipeline in
the parent shell context without creating a subshell.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 08:40:11AM -0400, Chet Ramey wrote:
> Since you're using bash-4.2, you can experiment with the `lastpipe' shopt
> option.  If that's enabled, bash runs the last element of a pipeline in
> the parent shell context without creating a subshell.

I actually started writing a response that mentioned that, but when I
was testing it, it didn't seem to work.

The manual says job control has to be off, so I did it as a script:

imadev:~$ cat foo
#!/bin/bash
set +m
shopt -s lastpipe
echo hi | read foo
echo "<$foo>"
printf '%s\n' some words | while read -r line; do myarray+=("$line"); done
printf '<%s> ' "${myarray[@]}"; echo
imadev:~$ ./foo
<>
<> 
imadev:~$ /bin/bash -c 'echo $BASH_VERSION'
4.2.20(8)-release

This is the first time I've ever attempted to use lastpipe, so I'm not
sure what's wrong.

Re: Severe Bash Bug with Arrays

[Chet Ramey]

On 4/25/12 10:55 AM, Greg Wooledge wrote:
> On Wed, Apr 25, 2012 at 08:40:11AM -0400, Chet Ramey wrote:
>> Since you're using bash-4.2, you can experiment with the `lastpipe' shopt
>> option.  If that's enabled, bash runs the last element of a pipeline in
>> the parent shell context without creating a subshell.
> 
> I actually started writing a response that mentioned that, but when I
> was testing it, it didn't seem to work.
> 
> The manual says job control has to be off, so I did it as a script:
> 
> imadev:~$ cat foo
> #!/bin/bash
> set +m
> shopt -s lastpipe
> echo hi | read foo
> echo "<$foo>"

z4.local(1)$ cat x3
#! ../bash-4.2-patched/bash
shopt -s lastpipe

echo foo | read bar
echo $bar
z4.local(1)$ ../bash-4.2-patched/bash -c 'echo $BASH_VERSION'
4.2.24(9)-release
z4.local(1)$ ../bash-4.2-patched/bash ./x3
foo
z4.local(1)$ ./x3
foo

The set +m doesn't matter, since scripts do not have job control enabled
by default.  Adding it doesn't change the behavior.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 11:07:57AM -0400, Chet Ramey wrote:
> z4.local(1)$ ../bash-4.2-patched/bash -c 'echo $BASH_VERSION'
> 4.2.24(9)-release
> z4.local(1)$ ../bash-4.2-patched/bash ./x3
> foo
> z4.local(1)$ ./x3
> foo

I grabbed patches 21-24 and applied them.  No change.

imadev:/var/tmp/bash/bash-4.2$ cat ~/foo
#!/bin/bash
shopt -s lastpipe
echo hi | read foo
echo "<$foo>"
imadev:/var/tmp/bash/bash-4.2$ ./bash -c 'echo $BASH_VERSION'
4.2.24(9)-release
imadev:/var/tmp/bash/bash-4.2$ ./bash ~/foo
<>

Re: Severe Bash Bug with Arrays

[Chet Ramey]

On 4/25/12 11:27 AM, Greg Wooledge wrote:
> On Wed, Apr 25, 2012 at 11:07:57AM -0400, Chet Ramey wrote:
>> z4.local(1)$ ../bash-4.2-patched/bash -c 'echo $BASH_VERSION'
>> 4.2.24(9)-release
>> z4.local(1)$ ../bash-4.2-patched/bash ./x3
>> foo
>> z4.local(1)$ ./x3
>> foo
> 
> I grabbed patches 21-24 and applied them.  No change.

OK.  I get the same working behavior on Mac OS X, Solaris 8/10, RHEL 5,
BSD/OS (old), and Fedora 15.  I'm not sure what to tell you.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Roman Rakus]

On 04/25/2012 06:15 PM, Chet Ramey wrote:

On 4/25/12 11:27 AM, Greg Wooledge wrote:

On Wed, Apr 25, 2012 at 11:07:57AM -0400, Chet Ramey wrote:

z4.local(1)$ ../bash-4.2-patched/bash -c 'echo $BASH_VERSION'
4.2.24(9)-release
z4.local(1)$ ../bash-4.2-patched/bash ./x3
foo
z4.local(1)$ ./x3
foo

I grabbed patches 21-24 and applied them.  No change.

OK.  I get the same working behavior on Mac OS X, Solaris 8/10, RHEL 5,
BSD/OS (old), and Fedora 15.  I'm not sure what to tell you.

Chet

On Fedora 16 it works as expected.
$ rpm -q bash
bash-4.2.24-1.fc16.x86_64

RR

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 06:26:18PM +0200, Roman Rakus wrote:
> On 04/25/2012 06:15 PM, Chet Ramey wrote:
> >OK.  I get the same working behavior on Mac OS X, Solaris 8/10, RHEL 5,
> >BSD/OS (old), and Fedora 15.  I'm not sure what to tell you.

> On Fedora 16 it works as expected.
> $ rpm -q bash
> bash-4.2.24-1.fc16.x86_64

It fails on HP-UX 10.20 in bash 4.2.20 and 4.2.24 (from source).
It also fails on OpenBSD 5.0 using bash 4.2.10 from ports.  Since those
are the first two cases I tried, I was quite puzzled.

It works on Debian 6.0 in bash 4.2.10 (from source), which I just happened
to have lying around.  So far everyone has reported success on any Linux
system, so that doesn't surprise me.

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 12:55:06PM -0400, Greg Wooledge wrote:
> It also fails on OpenBSD 5.0 using bash 4.2.10 from ports.

And it also fails on OpenBSD 5.0 using bash 4.2.24 straight from the
original sources.

cyclops:/var/tmp/bash/bash-4.2$ ./bash -c 'echo $BASH_VERSION'
4.2.24(1)-release
cyclops:/var/tmp/bash/bash-4.2$ cat ~/foo
#!/bin/bash
shopt -s lastpipe
echo hi | read foo
echo "<$foo>"
cyclops:/var/tmp/bash/bash-4.2$ ./bash ~/foo
<>
cyclops:/var/tmp/bash/bash-4.2$ uname -a
OpenBSD cyclops.wooledge.org 5.0 GENERIC.MP#63 amd64

Re: Directory completion problems

[Chet Ramey]

On 4/23/12 3:57 AM, Artur Rataj wrote:
> To be exact, symlink is in the directory ~/a, and it points to ~/test and
> there is also a directory ~/.java. So, after cd ~/a/symlink, there are two
> ../.java files -- one in the physical parent, and one in the logical
> parent. The former is a directory, the latter is a plain file.
> 
> ~/a/symlink$ cd ../
> 
> produces one alternative: .java, but only if the plain file .java exists.

OK, we're back to completion again.  My previous post was intended to
illustrate the effect of a logical view of the file system.

I can't reproduce very much of what you're seeing.  I can only conclude
that there is a completion specification for `cd' that is getting in the
way and complicating things.  For instance, after a `cd symlink', hitting
TAB after "cd ../" gives me the alternatives `i' and `symlink'.  Hitting
TAB after "cd symlink/../" in `a' gives me the same two alternatives.
You're not seeing that, so I have to think you're using some other function
to generate the completions.

One thing, however, could make a difference.  If, for some reason, you
get .java as a completion and there is a file named .java in the logical
parent and a directory named .java in the physical parent, cd will work.
This is a case of cd trying to be helpful: directory canonicalization
fails, so the shell tries to use the argument as supplied as a last resort.
In this case, that succeeds.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Chet Ramey]

On 4/25/12 1:10 PM, Greg Wooledge wrote:
> On Wed, Apr 25, 2012 at 12:55:06PM -0400, Greg Wooledge wrote:
>> It also fails on OpenBSD 5.0 using bash 4.2.10 from ports.
> 
> And it also fails on OpenBSD 5.0 using bash 4.2.24 straight from the
> original sources.

OK.  Maybe someone with OpenBSD can debug the appropriate code in
execute_cmd.c:execute_pipeline.  Maybe the call to move_to_high_fd
fails.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Ted Okuzumi]

Please note that if I use a different command in the while loop it works.

So instead of
echo "$mydata" | while read -r line; do myarray+=( "$line" ); done

if I say:
echo "$mydata" | while read -r line; do echo "$line" ; done

Then it works.  To pipe output into a while statement works, normally,
however it fails when the array+=( )  is used.  It doesn't fail on other
commands that I've seen.

This is why I am saying it is definitely a bug.  Do you not agree?

Thanks,
Ted

On Tue, Apr 24, 2012 at 10:23 PM, Clark Wang  wrote:

> On Wed, Apr 25, 2012 at 07:21, Ted Okuzumi  wrote:
>
>> I am writing this e-mai to report to report a bug in bash.
>>
>> Configuration Information [Automatically generated, do not change]:
>> Machine: x86_64
>> OS: linux-gnu
>> Compiler: gcc
>> Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
>> -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-unknown-linux-gnu'
>> -DCONF_VENDOR='unknown' -DLOCALEDIR='/usr/local/share/locale'
>> -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib
>> -g \
>> -O2
>> uname output: Linux concour5 2.6.18-274.3.1.el5xen #1 SMP Tue Sep 6
>> 20:57:11 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
>> Machine Type: x86_64-unknown-linux-gnu
>>
>> Bash Version: 4.2
>> Patch Level: 0
>> Release Status: release
>>
>> Description:
>>Cannot redirect into an array from multiline variable
>>
>> Does not work:
>> echo "$mydata" | while read line; do myarray+=( "$line" ); done
>>
>
> Here the whole *while* statement runs in a subshell. See
> http://mywiki.wooledge.org/BashPitfalls#grep_foo_bar_.7C_while_read_-r.3B_do_.28.28count.2B-.2B-.29.29.3B_done.
>
>>
>>
>> Works:
>> while read -r line; do myarray+=( "$line" ); done < <(echo "$mydata")
>>
>>
>

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 10:36:53AM -0700, Ted Okuzumi wrote:
> Please note that if I use a different command in the while loop it works.
> 
> So instead of
> echo "$mydata" | while read -r line; do myarray+=( "$line" ); done
> 
> if I say:
> echo "$mydata" | while read -r line; do echo "$line" ; done

You do not appear to understand the issue here.  Each command in a pipeline
is run in a separate subshell.  That means "echo" runs in its own subshell,
and the entire "while" loop runs in its own subshell.

In the first example, you are populating a variable inside this subshell.

After the pipeline ends, that subshell is gone, and so is the variable
that you populated.

In the second example, you are simply writing things to stdout.  You are
not populating variables that need to persist beyond the lifespan of the
subshell.

> Then it works.  To pipe output into a while statement works, normally,
> however it fails when the array+=( )  is used.  It doesn't fail on other
> commands that I've seen.
> 
> This is why I am saying it is definitely a bug.  Do you not agree?

I do not agree.

Re: Severe Bash Bug with Arrays

[DJ Mills]

On Wed, Apr 25, 2012 at 1:36 PM, Ted Okuzumi  wrote:
> Please note that if I use a different command in the while loop it works.
>
> So instead of
> echo "$mydata" | while read -r line; do myarray+=( "$line" ); done
>
> if I say:
> echo "$mydata" | while read -r line; do echo "$line" ; done
>
> Then it works.  To pipe output into a while statement works, normally,
> however it fails when the array+=( )  is used.  It doesn't fail on other
> commands that I've seen.
>
> This is why I am saying it is definitely a bug.  Do you not agree?
>
> Thanks,
> Ted



It is NOT a bug. POSIX specifies this behavior. The echo is still run
in a subshell...

The issue is that commands in a pipeline are run in a subshell. This
means that assignments and changes in PWD do not persist afterwards.
See http://mywiki.wooledge.org/BashFAQ/024.

The lastpipe shopt will work, as discussed before. The other options
are process substitution, or (POSIXly) a fifo.

Examples:

# Process substitution:
while read -r line; do
  array+=("$line")
done < <(some_command)


# FIFO:
mkfifo myfifo || exit
trap 'rm -f myfifo' EXIT

some_command > myfifo &
while read -r line; do
  array+=("$line")
done < myfifo


# Or, since it's a string you're using in your example, you could use
a here string or here document
while read -r line; do array+=("$line"); done <<<"$mydata"

while read -r line; do array+=("$line"); done <

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 01:31:11PM -0400, Chet Ramey wrote:
> OK.  Maybe someone with OpenBSD can debug the appropriate code in
> execute_cmd.c:execute_pipeline.  Maybe the call to move_to_high_fd
> fails.

Breakpoint 1 at 0x41d5fc: file execute_cmd.c, line 2107.
Breakpoint 2 at 0x414aa3: file general.c, line 444.
(gdb) run ~/foo
...
Breakpoint 1, execute_pipeline (command=0x202665680, asynchronous=0,
pipe_in=-1, pipe_out=-1, fds_to_close=0x20e5bd660) at execute_cmd.c:2107
2107  BLOCK_CHILD (set, oset);
(gdb) cont
Continuing.

Breakpoint 2, move_to_high_fd (fd=0, check_new=0, maxfd=255) at general.c:444
444   if (maxfd < 20)
(gdb) n
453 nfds = maxfd;
(gdb) n
455   for (nfds--; check_new && nfds > 3; nfds--)
(gdb) n
459   if (nfds > 3 && fd != nfds && (script_fd = dup2 (fd, nfds)) != -1)
(gdb) n
468   return (fd);
(gdb) print fd
$1 = 0
(gdb) n
469 }

Is that saying it's searching for an available file descriptor starting
at or near 255?

cyclops:/var/tmp/bash/bash-4.2$ ulimit -a
core file size  (blocks, -c) unlimited
data seg size   (kbytes, -d) 524288
file size   (blocks, -f) unlimited
max locked memory   (kbytes, -l) 2631052
max memory size (kbytes, -m) 7888636
open files  (-n) 128
pipe size(512 bytes, -p) 1
stack size  (kbytes, -s) 4096
cpu time   (seconds, -t) unlimited
max user processes  (-u) 128
virtual memory  (kbytes, -v) 528384

That's the OpenBSD box.  The HP-UX box is even more extreme:

$ ulimit -a
core file size  (blocks, -c) 2097151
data seg size   (kbytes, -d) 65536
file size   (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files  (-n) 60
pipe size(512 bytes, -p) 16
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 76
virtual memory  (kbytes, -v) unlimited

Maybe it only works on boxes with a very large "open files" resource
limit.  Perhaps the hard-coded 255 should be replaced with something
involving getrlimit() or similar?

Re: Severe Bash Bug with Arrays

[Chet Ramey]

On 4/25/12 2:04 PM, Greg Wooledge wrote:

> Maybe it only works on boxes with a very large "open files" resource
> limit.  Perhaps the hard-coded 255 should be replaced with something
> involving getrlimit() or similar?

I see two potential fixes here.  First, try changing the 255 to -1 and
see what happens.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 02:54:07PM -0400, Chet Ramey wrote:
> On 4/25/12 2:04 PM, Greg Wooledge wrote:
> 
> > Maybe it only works on boxes with a very large "open files" resource
> > limit.  Perhaps the hard-coded 255 should be replaced with something
> > involving getrlimit() or similar?
> 
> I see two potential fixes here.  First, try changing the 255 to -1 and
> see what happens.

That broke something pretty badly.

cyclops:/var/tmp/bash/bash-4.2$ cat ~/foo
#!/bin/bash
shopt -s lastpipe
echo hi | read foo
echo "<$foo>"
cyclops:/var/tmp/bash/bash-4.2$ ./bash ~/foo
cyclops:/var/tmp/bash/bash-4.2$ ./bash -c 'shopt -s lastpipe; echo hi | read 
foo; echo "<$foo>"'


So it seems to work with -c but not when reading from a file.  I'm guessing
the file descriptor move somehow made bash lose track of the file it was
reading commands from.

The move_to_high_fd part looks happier:

Breakpoint 2, move_to_high_fd (fd=0, check_new=0, maxfd=-1) at general.c:444
444   if (maxfd < 20)
(gdb) n
446   nfds = getdtablesize ();
(gdb) n
447   if (nfds <= 0)
(gdb) n
449   if (nfds > HIGH_FD_MAX)
(gdb) n
455   for (nfds--; check_new && nfds > 3; nfds--)
(gdb) n
459   if (nfds > 3 && fd != nfds && (script_fd = dup2 (fd, nfds)) != -1)
(gdb) n
461   if (check_new == 0 || fd != fileno (stderr))  /* don't close 
stderr */
(gdb) n
462 close (fd);
(gdb) n
463   return (script_fd);
(gdb) print script_fd
$1 = 127
(gdb) n
469 }
(gdb) n

But after that I have no idea what's happening (or failing to happen):

execute_pipeline (command=0x209d2c260, asynchronous=0, pipe_in=-1,
pipe_out=-1, fds_to_close=0x202b8ca10) at execute_cmd.c:2209
2209  if (lstdin > 0)
(gdb) n
2211  do_piping (prev, pipe_out);
(gdb) n
2212  prev = NO_PIPE;
(gdb) n
2213  add_unwind_protect (restore_stdin, lstdin);
(gdb) n
2214  lastpipe_flag = 1;
(gdb) n
2215  freeze_jobs_list ();
(gdb) n
2216  lastpipe_jid = stop_pipeline (0, (COMMAND *)NULL);/* XXX 
*/
(gdb) n
2217  add_unwind_protect (lastpipe_cleanup, lastpipe_jid);
f (cmd)
(gdb) n
2220cmd->flags |= CMD_LASTPIPE;
(gdb) n
  if (prev >= 0)
(gdb) n
2226  exec_result = execute_command_internal (cmd, asynchronous, prev, 
pipe_out, fds_to_close);
(gdb) n
2229  if (lstdin > 0)
(gdb) n
2230restore_stdin (lstdin);
(gdb) n
2233  if (prev >= 0)
(gdb) n
2237  UNBLOCK_CHILD (oset);
(gdb) n
2240  QUIT;
(gdb) n
2242  if (lastpipe_flag)
(gdb) n
2245  append_process (savestring (the_printed_command), 
dollar_dollar_pid, exec_result, lastpipe_jid);
(gdb) n
2247  lstdin = wait_for (lastpid);
(gdb) n
2249  exec_result = job_exit_status (lastpipe_jid);
(gdb) n
2251  unfreeze_jobs_list ();
(gdb) n
2254  discard_unwind_frame ("lastpipe-exec");
(gdb) n
2256  return (exec_result);
(gdb) n
2257}
(gdb) n
execute_connection (command=0x209d2c260, asynchronous=0, pipe_in=-1, 
pipe_out=-1, fds_to_close=0x202b8ca10) at execute_cmd.c:2338
2338  if (was_error_trap && ignore_return == 0 && invert == 0 && 
exec_result != EXECUTION_SUCCESS)
(gdb) cont
Continuing.

Program exited normally.
(gdb) 

Re: Severe Bash Bug with Arrays

[Chet Ramey]

On 4/25/12 3:09 PM, Greg Wooledge wrote:
> On Wed, Apr 25, 2012 at 02:54:07PM -0400, Chet Ramey wrote:
>> On 4/25/12 2:04 PM, Greg Wooledge wrote:
>>
>>> Maybe it only works on boxes with a very large "open files" resource
>>> limit.  Perhaps the hard-coded 255 should be replaced with something
>>> involving getrlimit() or similar?
>>
>> I see two potential fixes here.  First, try changing the 255 to -1 and
>> see what happens.
> 
> That broke something pretty badly.

OK.  Now make the call move_to_high_fd(0, 1, -1) and see if that fixes it.


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: Severe Bash Bug with Arrays

[Greg Wooledge]

On Wed, Apr 25, 2012 at 04:05:15PM -0400, Chet Ramey wrote:
> OK.  Now make the call move_to_high_fd(0, 1, -1) and see if that fixes it.

cyclops:/var/tmp/bash/bash-4.2$ ./bash ~/foo

cyclops:/var/tmp/bash/bash-4.2$ ./bash -c 'shopt -s lastpipe; echo hi | read 
foo; echo "<$foo>"'

cyclops:/var/tmp/bash/bash-4.2$ ./bash < ~/foo


So those three cases work.  I tried one other test for grins (remember
that ulimit -n is 128):

cyclops:/var/tmp/bash/bash-4.2$ ./bash -c 'shopt -s lastpipe; exec 3"'


Looks good at first glance.

Re: Severe Bash Bug with Arrays

[Ted Okuzumi]

Thank you for that detailed information.  I really appreciate the 2
workarounds you provided.

I can confirm by adding in this line to my script:

shopt -s lastpipe

fixes the problem, confirming that it was indeed a subshell issue.

Best regards,
Ted

On Wed, Apr 25, 2012 at 10:50 AM, DJ Mills  wrote:

> On Wed, Apr 25, 2012 at 1:36 PM, Ted Okuzumi  wrote:
> > Please note that if I use a different command in the while loop it works.
> >
> > So instead of
> > echo "$mydata" | while read -r line; do myarray+=( "$line" ); done
> >
> > if I say:
> > echo "$mydata" | while read -r line; do echo "$line" ; done
> >
> > Then it works.  To pipe output into a while statement works, normally,
> > however it fails when the array+=( )  is used.  It doesn't fail on other
> > commands that I've seen.
> >
> > This is why I am saying it is definitely a bug.  Do you not agree?
> >
> > Thanks,
> > Ted
>
>
>
> It is NOT a bug. POSIX specifies this behavior. The echo is still run
> in a subshell...
>
> The issue is that commands in a pipeline are run in a subshell. This
> means that assignments and changes in PWD do not persist afterwards.
> See http://mywiki.wooledge.org/BashFAQ/024.
>
> The lastpipe shopt will work, as discussed before. The other options
> are process substitution, or (POSIXly) a fifo.
>
> Examples:
>
> # Process substitution:
> while read -r line; do
>   array+=("$line")
> done < <(some_command)
>
>
> # FIFO:
> mkfifo myfifo || exit
> trap 'rm -f myfifo' EXIT
>
> some_command > myfifo &
> while read -r line; do
>   array+=("$line")
> done < myfifo
>
>
> # Or, since it's a string you're using in your example, you could use
> a here string or here document
> while read -r line; do array+=("$line"); done <<<"$mydata"
>
> while read -r line; do array+=("$line"); done < $mydata
> EOF
>
> mapfile, with bash4, is also an option for this particular case.
>

Re: Severe Bash Bug with Arrays

[Linda Walsh]

Ted Okuzumi wrote:


I am writing this e-mai to report to report a bug in bash

Description:
Cannot redirect into an array from multiline variable

Does not work:
echo "$mydata" | while read line; do myarray+=( "$line" ); done



Have you tried this?


Ishtar:> a='this
 is
  a
   test'
Ishtar:> echo "$a"
this
 is
  a
   test
Ishtar:> readarray b < <(echo "$a")
Ishtar:> echo ${#b[*]}
4
Ishtar:> echo ${b[*]}
this is a test
Ishtar:> echo "${b[*]}"
this
  is
   a
test

Ishtar:> echo -n "${b[*]}"
this
  is
   a
test
Ishtar:> readarray -t b < <(echo "$a")
Ishtar:> echo "${b[*]}"
this  is   atest
Ishtar:> echo ${b[*]}
this is a test
Ishtar:>

-

Seeems like you can have it anyway you want it...?

Does the above work for you?

Re: Passing variables by reference conflicts with local

[Linda Walsh]

Clark Wang wrote:


On Wed, May 5, 2010 at 01:57, Freddy Vulto  wrote:


It appears that `unset' is capable of traversing down the call-stack and
unsetting variables repeatedly:

   a=0 b=0 c=0 d=0 e=0
   _unset() { unset -v b c c d d d e; }
   t1() {
   local a=1 b=1 c=1 d=1
   t2
   }
   t2() {
   local a=2 b=2 c=2 d=2 e=2
   _unset
   echo a:$a b:$b c:$c d:$d e:$e
   }
   t1  # Outputs: a:2 b:1 c:0 d: e:0
   #^   ^   ^   ^  ^-- unset once (skipped t1)
   #|   |   |   +- unset thrice to global
   #|   |   +- unset twice till global
   #|   +- unset once till t1
   #+- unset not

It seems to work on bash-3.0, 3.2, 4.0 and 4.1.
Is this a bug or a feature?



When I revisit this 2 years old thread I don't understand why following
foo() function does not output the global var:

$ cat foo.sh
var=global
foo()
{
local var=foo
unset var
echo foo: $var
}
bar_unset()
{
unset var
}
bar()
{
local var=bar
bar_unset
echo bar: $var
}
foo
bar
$ bash foo.sh
foo:
bar: global
$

In foo() it unsets its local var so why doesn't the subsequent $var refer
to the global var?



The above is pretty deep stuff, but I would guess from this:


Ishtar:> foo() { local var=foo; echo -n "type=\""; typeset -p var;echo -n "\""; 
unset var; typeset -p var ;echo foo: $var; }

Ishtar:> foo
type="declare -- var="foo"
"-bash: typeset: var: not found
foo:

I would guess (without looking at the codE), that
when you do a local, it creates another copy of 'var' at the end of an array.
like var[0][1]
so global is at 0, and local is at 1.
So let #=the index of the last position in the array so at
global var[#]='global',
local var[#]=foo, but 'local adds one to the var array, so it's going into 
var[1].
Now if you unset var[1], function foo still is using it's incremented value
of '#' (1) to tell you it has no value.

However, if you call a second function 'bar_unset', to unset the var
it deletes var[1], and sets # to 0,
so now when control returns to original
foo,
foo Can't assume # is unchanged (you've called another function,
it might have done something to # (the count of copies of var -- which it
did!)...
so it knows it did a "local", so it's copy of 'var' would be "last"
on the "var" value array --
but # has been reset to 0,
so now foo picks up var[0]...

That's just a guess based on how I might implement such behavior if I wanted
it to behave that way.

As to whether or not it is "correct" by some arbitrarily set standard or 
definition of "correct" -- that's a matter for nit-pickers and POSIXies.  :-)


I'm sure someone more qualified with the internals can give a more accurate 
description of what happened, but that's my high level, "black-box" view.

Re: Shouldn't "type" and "command" complain if there are no parameters?

[Linda Walsh]

Victor Engmark wrote:


Re-post from 
:

[T]hese commands print nothing and return exit code 0 if no parameters are 
provided. But [...] `help command` and `help type` both state that at least one 
parameter is mandatory. Is this a bug, a feature, or did I misunderstand 
something?




command executes the named argument as a command.

It is perfectly valid to type in nothing at the command line and expect
no error.

A similar argument like can be proposed for type.

You give it nothing as input, why would you expect output?

Conversely, would you want it to give you an error every time you
hit 'Enter' at the prompt?

it's similar to the annoyance caused by getting an error for unexpected ";"
when ";" is meant, in such context, to be taken for a line separator.
;-/

Is it possible or RFE to expand ranges of *arrays*

[Linda Walsh]

I know I can get
a="abcdef" echo "${a[2:4]}" = cde

 how do I do:
typeset -a a=(apple berry cherry date); then get:

echo ${a[1:2]} = "berry" "cherry"  ( non-grouped args)

I tried to do it in a function and hurt myself.

Re: Is it possible or RFE to expand ranges of *arrays*

[John Kearney]

Am 26.04.2012 06:26, schrieb Linda Walsh:

I know I can get
a="abcdef" echo "${a[2:4]}" = cde

 how do I do:
typeset -a a=(apple berry cherry date); then get:

echo ${a[1:2]} = "berry" "cherry"  ( non-grouped args)

I tried to do it in a function and hurt myself.




echo ${a[@]:1:2}

Re: Shouldn't "type" and "command" complain if there are no parameters?

[Victor Engmark]

On Thu, Apr 26, 2012 at 3:50 AM, Linda Walsh  wrote:

> Victor Engmark wrote:
>
>> Re-post from > parameters-to-bashs-command-**and-type-optional
>> >:
>>
>> [T]hese commands print nothing and return exit code 0 if no parameters
>> are provided. But [...] `help command` and `help type` both state that at
>> least one parameter is mandatory. Is this a bug, a feature, or did I
>> misunderstand something?
>>
>
> command executes the named argument as a command.
>
> It is perfectly valid to type in nothing at the command line and expect
> no error.
>

Actually, if you type in a command, and then press Enter more than once,
the exit code is *kept*, as if you had pressed Enter only once. So pressing
Enter on an empty command line is *not* the same as saying `:`, or `true`.

A similar argument like can be proposed for type.
>
> You give it nothing as input, why would you expect output?
>

The issue is rather that the synopsis is *inconsistent* with the
functioning of `type` and `command`. For example, the `watch` command
requires a command string as its last parameter, and without it, it prints
its own synopsis and returns with exit code 1, as expected.

My suggestion is simply that unless someone can show that somehow the
synopsis *is* consistent with the functioning of these builtins, the
synopsis should be corrected *or* the functioning of these commands should
be changed, in order to be consistent with most other *nix tools.

Cheers,
V