Backslashed space in variable/argument expansion does not prevent splitting.
From: deanw
To: bug-bash@gnu.org
Subject: Backslashed space in variable expansion does not prevent splitting.
Configuration Information [Automatically generated, do not change]:
Machine: i686 OS: cygwin Compiler: gcc Compilation CFLAGS: -
DPROGRAM='bash.exe' -DCONF_HOSTTYPE='i686' -DCONF_OSTYPE='cygwin'
-DCONF_MACHTYPE='i686-pc-
cygwin' -DCONF_VENDOR='pc' -DLOCALEDIR='/usr/share/locale' -
DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -DRECYCLES_PIDS -I.
-I/usr/src/bash-4.3.42-4.i686/src/bash-
4.3 -I/usr/src/bash-4.3.42-4.i686/src/bash-4.3/include
-I/usr/src/bash-4.3.42-4.i686/src/bash-
4.3/lib -DWORDEXP_OPTION -ggdb -O2 -pipe -Wimplicit-function-
declaration -fdebug-prefix-map=/usr/src/bash-4.3.42-
4.i686/build=/usr/src/debug/bash-4.3.42-4
-fdebug-prefix-map=/usr/src/bash-4.3.42-4.i686/src/bash-
4.3=/usr/src/debug/bash-4.3.42-4 uname output: CYGWIN_NT-6.1-WOW
DW03PC02 2.5.1(0.297/5/3) 2016-04-21 22:12 i686 Cygwin Machine Type:
i686-pc-cygwin
Bash Version: 4.3 Patch Level: 42 Release Status: release
Description:
The expansion of a variable containing spaces splits on all spaces including
escaped/backslashed space.
I ran into this problem when trying to buiild a TCL extension. Building
the extension requires pulling in with source tclConfig.sh which
provides various settings. TCL_DEFS contains space separated -D
arguments to pass to gcc. Some of these arguements contains space
which are backslashed to prevent splitting.
Argument expansion is not distinguishing between space and backslased
space. I have tried 3.00.16(1)-release on FedoraCore 4, 4.2.10(1)-
release on FedoraCore 16, 4.3.42(4)-release on Cygwin 2.5.1 with the
same outcome.
Repeat-By:
Run the ./bad_exp.sh script. It explains the desired behaviour and
demonstrates issue. I include below the contents of two script and my output
log.
=Start File ./bad_exp.sh
#!/usr/bin/env bash
# Issue: Backslash space is not preventing argument splitting
echo $BASH_VERSION
# TCL_DEFS is actually source-d from TCL config script so have no
# control over it.
TCL_DEFS='-DPACKAGE_NAME=\"tcl\" -DPACKAGE_TARNAME=\"tcl\" -
DPACKAGE_VERSION=\"8.5\" -DPACKAGE_STRING=\"tcl\ 8.5\" -
DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -
DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -
DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -
DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_LIMITS_H=1 -
DHAVE_SYS_PARAM_H=1 -DTCL_CFGVAL_ENCODING=\"iso8859-1\" -
DMODULE_SCOPE=extern\ __attribute__\(\(__visibility__\(\"hidden\"\)\)\)
-DTCL_CFG_DO64BIT=1 -DHAVE_CAST_TO_UNION=1 -DTCL_SHLIB_EXT=\".so\" -
DNDEBUG=1 -DTCL_CFG_OPTIMIZED=1 -DTCL_TOMMATH=1 -DMP_PREC=4 -
D_LARGEFILE64_SOURCE=1 -DTCL_WIDE_INT_IS_LONG=1 -DHAVE_GETCWD=1 -
DHAVE_OPENDIR=1 -DHAVE_STRTOL=1 -DHAVE_WAITPID=1 -DHAVE_GETADDRINFO=1
-DUSE_TERMIOS=1 -DHAVE_SYS_TIME_H=1 -DTIME_WITH_SYS_TIME=1 -
DHAVE_GMTIME_R=1 -DHAVE_LOCALTIME_R=1 -DHAVE_MKTIME=1 -
DHAVE_TM_GMTOFF=1 -DHAVE_TIMEZONE_VAR=1 -DHAVE_STRUCT_STAT_ST_BLOCKS=1
-DHAVE_STRUCT_STAT_ST_BLKSIZE=1 -DHAVE_BLKCNT_T=1 -DHAVE_INTPTR_T=1 -
DHAVE_UINTPTR_T=1 -DHAVE_SIGNED_CHAR=1 -DHAVE_LANGINFO=1 -
DHAVE_SYS_IOCTL_H=1 -DTCL_UNLOAD_DLLS=1 -DHAVE_CPUID=1 '
./echo_args.sh ${TCL_DEFS}
echo 'Reduction of problem' echo 'The desired result is echo_args should
receive three arguments arg0, arg1 and "arg 2"' T='arg0 arg1 arg\ 2'
echo 'This receives four arguments, wrongly splitting on the backslashed
space' ./echo_args.sh $T
echo 'This correctly receives one argument but is not the desired
result' ./echo_args.sh "$T"
=End File ./bad_exp.sh =Start File echo_args.sh
#!/usr/bin/env bash
# Script to display command line arguments
echo "${#} arguments" i=0 for arg in "${@}" do echo "${i}:$arg"
((i++)) done
=End File echo_args.sh =Start output log
4.2.10(1)-release 53 arguments :-DPACKAGE_NAME=\"tcl\"
1:-DPACKAGE_TARNAME=\"tcl\"
2:-DPACKAGE_VERSION=\"8.5\"
3:-DPACKAGE_STRING=\"tcl\
4:8.5\"
5:-DPACKAGE_BUGREPORT=\"\"
6:-DSTDC_HEADERS=1
7:-DHAVE_SYS_TYPES_H=1
8:-DHAVE_SYS_STAT_H=1
9:-DHAVE_STDLIB_H=1
10:-DHAVE_STRING_H=1
11:-DHAVE_MEMORY_H=1
12:-DHAVE_STRINGS_H=1
13:-DHAVE_INTTYPES_H=1
14:-DHAVE_STDINT_H=1
15:-DHAVE_UNISTD_H=1
16:-DHAVE_LIMITS_H=1
17:-DHAVE_SYS_PARAM_H=1
18:-DTCL_CFGVAL_ENCODING=\"iso8859-1\"
19:-DMODULE_SCOPE=extern\
20:__attribute__\(\(__visibility__\(\"hidden\"\)\)\)
21:-DTCL_CFG_DO64BIT=1
22:-DHAVE_CAST_TO_UNION=1
23:-DTCL_SHLIB_EXT=\".so\"
24:-DNDEBUG=1
25:-DTCL_CFG_OPTIMIZED=1
26:-DTCL_TOMMATH=1
27:-DMP_PREC=4
28:-D_LARGEFILE64_SOURCE=1
29:-DTCL_WIDE_INT_IS_LONG=1
30:-DHAVE_GETCWD=1
31:-DHAVE_OPENDIR=1
32:-DHAVE_STRTOL=1
33:-DHAVE_WAITPID=1
34:-DHAVE_GETADDRINFO=1
35:-DUSE_TERMIOS=1
36:-DHAVE_SYS_TIME_H=1
37:-DTIME_WITH_SYS_TIME=1
38:-DHAVE_GMTIME_R=1
39:-DHAVE_LOCALTIME_R=1
40:-DHAVE_MKTIME=1
41:-DHAVE_TM_GMTOFF=1
42:-DHAVE_TIMEZONE_VAR=1
43:-DHAVE_STRUCT_STAT_ST_BLOCKS=1
44:-DHAVE_STRUCT_STAT_ST_BLKSIZE=1
45:-DHAVE_BLKCNT_
Feature request: Support Linux packetized pipes
Hi bug-bash, The Linux kernel supports creating pipes in a mode (enabled by passing the O_DIRECT flag, and known sometimes as a "packetized pipe") that respects message boundaries, to an extent. An excerpt from the Linux manpage pipe(2): O_DIRECT (since Linux 3.4) Create a pipe that performs I/O in "packet" mode. Each write(2) to the pipe is dealt with as a separate packet, and read(2)s from the pipe will read one packet at a time. Note the following points: * Writes of greater than PIPE_BUF bytes (see pipe(7)) will be split into multiple packets. The constant PIPE_BUF is defined in . * If a read(2) specifies a buffer size that is smaller than the next packet, then the requested number of bytes are read, and the excess bytes in the packet are discarded. Specifying a buffer size of PIPE_BUF will be sufficient to read the largest possible packets (see the previous point). * Zero-length packets are not supported. (A read(2) that specifies a buffer size of zero is a no-op, and returns 0.) Older kernels that do not support this flag will indicate this via an EINVAL error. I would like to use a packetized pipe in a conventional shell pipeline to support parallelism. Multiple processes can read from the same pipe at the same time and not receive any partial lines, as long as the pipe is being written to one-line-at-a-time, with lines under PIPE_BUF size. (Parallelism in shell pipelines is an interest of mine; I wrote about one technique here: https://catern.com/posts/pipes.html but bash support of packetized pipes would make this much less painful) So, it would be nice if it was possible in bash to somehow explicitly request that a pipe between two processes be packetized (on systems that support this behavior). I can go into more detail about my use cases, and I can work on adding such support. Before I do so, though, does this sound like a good feature to add to bash? Thanks!
Re: Backslashed space in variable/argument expansion does not prevent splitting.
On Fri, 17 Jun 2016 11:21:42 +0100, Dean Wakerley wrote:
> Description: The expansion of a variable containing spaces splits on all
> spaces including escaped/backslashed space.
It's not a bug, you probably want
eval ./echo_args.sh ${TCL_DEFS}
but you're sitting on very thin ice here.
--
D.
Re: Backslashed space in variable/argument expansion does not prevent splitting.
On 6/17/16 6:21 AM, Dean Wakerley wrote: > Bash Version: 4.3 > Patch Level: 42 > Release Status: release > > Description: > The expansion of a variable containing spaces splits on all spaces > including escaped/backslashed space. This analysis isn't correct, and the description above describes the correct behavior. A backslash in the expansion of a variable does not have any special meaning to word splitting. Backslashes are special to the shell parser; if you want to have them interpreted, you need to run the text through the parser. This is usually accomplished using `eval'. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Backslashed space in variable/argument expansion does not prevent splitting.
On Fri, Jun 17, 2016 at 11:21:42AM +0100, Dean Wakerley wrote:
> I ran into this problem when trying to buiild a TCL extension. Building
> the extension requires pulling in with source tclConfig.sh which
> provides various settings. TCL_DEFS contains space separated -D
> arguments to pass to gcc. Some of these arguements contains space
> which are backslashed to prevent splitting.
I've never seen a Tcl extension that requires sourcing a .sh file.
Granted, the Tcl extension space is a royal mess, but *sane* ones
are supposed to follow a standard known as TEA which is basically GNU
autoconf.
I'm guessing you're stuck with an insane one. It would help to know
which one it is. Maybe you're simply doing something wrong.
> # TCL_DEFS is actually source-d from TCL config script so have no
> # control over it.
> TCL_DEFS='-DPACKAGE_NAME=\"tcl\" -DPACKAGE_TARNAME=\"tcl\" -
> DPACKAGE_VERSION=\"8.5\" -DPACKAGE_STRING=\"tcl\ 8.5\" -
> DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -
What you have here is from a file that is installed as part of Tcl
itself. I have no idea why your extension thinks you should be sourcing
this file, or what it wants you to do with it.
> ./echo_args.sh ${TCL_DEFS}
I have my own args program. This is what mine shows, if I pass it an
unquoted $TCL_DEFS to perform the word-splitting that you seem to want:
wooledg@wooledg:~$ source /opt/ebase/lib/tclConfig.sh
wooledg@wooledg:~$ args $TCL_DEFS
64 args: <-DPACKAGE_NAME=\"tcl\"> <-DPACKAGE_TARNAME=\"tcl\">
<-DPACKAGE_VERSION=\"8.6\"> <-DPACKAGE_STRING=\"tcl\> <8.6\"> [[...]]
I've snipped it, because it's quite long.
Now bear in mind, these are not the ACTUAL arguments you would want
to pass to gcc. These are options that have been spelled out with extra
backslashes in them because they are meant to be dropped into a Makefile,
or some other shell environment. The shell environment will strip out
the backslashes as it interprets the command.
See how -DPACKAGE_STRING (fourth and fifth args) has been split into
two words. This is because TCL_DEFS is NOT actually an array of arguments
that you can pass directly to a program. It's a single string that has to
be copied out, into a Makefile, or a shell script, and then interpreted.
In other words, this will not work:
$ gcc $TCL_DEFS
It needs to be this:
$ eval gcc "$TCL_DEFS"
Here's what that looks like, using my args prorgam:
wooledg@wooledg:~$ eval args gcc "$TCL_DEFS"
63 args: <-DPACKAGE_NAME="tcl"> <-DPACKAGE_TARNAME="tcl">
<-DPACKAGE_VERSION="8.6"> <-DPACKAGE_STRING="tcl 8.6"> [[...]]
Again, I've snipped it.
Either your Tcl extension's documentation is incorrect, or you are doing
something wrong. That's all I can come up with. There is no bash bug
here.
You might be better served by asking for help on this extension's
mailing list, if it has one. If it doesn't (wouldn't be a surprise),
then on a general Tcl mailing list or newsgroup.
help with rsync and Missing trailing-" in remote-shell command.
I have three servers. nginx00 (master nginx server) and nginx01 and nginx02
(slave nginx servers).
nginx00 is a staging environment that I do testing on. Once it goes live I run
a bash script that syncs the config files and restarts the nginx daemon. It's
working fine as root. However, I am bound by rules in my company to run
commands as users and not the root user.
I have setup keys for ssh for all my sysadmin users, and it works fine if I
just us it.
rsync -q -avz --delete -e 'ssh -i /home/dpich/.ssh/id_rsa' --rsync-path='sudo
rsync' /etc/nginx/ dpich@balancer01:/etc/nginx/
If I run it in this bash script...
#!/bin/bash
currentuser=$(whoami)
OPTS=(-q -avz --delete -e \'ssh -i /home/$currentuser/.ssh/id_rsa\'
--rsync-path=\'sudo rsync\')
SRC=(/etc/nginx/)
RSYNC_CMD=(rsync)
for h in nginx0{1..2} ; do
DST=$currentuser'@'$h':/etc/nginx/'
rsync ${OPTS[@]} ${SRC} ${DST}
done
I keep getting this error:
dpich@nginx00:~$ /usr/sbin/deploy2.sh
Missing trailing-" in remote-shell command.
rsync error: syntax or usage error (code 1) at main.c(430) [sender=3.1.1]
Missing trailing-" in remote-shell command.
rsync error: syntax or usage error (code 1) at main.c(430) [sender=3.1.1]
dpich@nginx00:~$
I'm 1000% convinced it has to do with quotes, but I haven't been able to solve
this.
Any help would be appreciated.
