History file clobbered by multiple simultaneous exits

[geoff]

Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc -I/home/abuild/rpmbuild/BUILD/bash-4.2 
-L/home/abuild/rpmbuild/BUILD/bash-4.2/../readline-6.2
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-suse-linux-gnu' 
-DCONF_VENDOR='suse' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
-DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -fmessage-length=0 -O2 -Wall 
-D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables 
-fasynchronous-unwind-tables -g  -D_GNU_SOURCE -DRECYCLES_PIDS -Wall -g 
-std=gnu89 -Wuninitialized -Wextra -Wno-unprototyped-calls -Wno-switch-enum 
-Wno-unused-variable -Wno-unused-parameter -ftree-loop-linear -pipe 
-fprofile-use
uname output: Linux bow 3.7.10-1.16-desktop #1 SMP PREEMPT Fri May 31 20:21:23 
UTC 2013 (97c14ba) x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-suse-linux-gnu

Bash Version: 4.2
Patch Level: 42
Release Status: release

Description:
When the history file is written, no locking is used.  The
result is that when multiple bash instances exit (nearly)
simultaneously, the history file can be replaced by a
zero-length file.

Repeat-By:
Because it's a race condition, the problem can't be reliably
reproduced.  The way I stumble across it is to create multiple
bash sessions 6 or more in a number of terminal windows, each
with HISTFILE set and histappend unset, and then shut the X
server down.  I imagine that you could also create multiple
shells and then kill them all simultaneously.

Fix:
Locking should be used when truncating and writing the history
file.  (Yes, I know it's a pain in a portable program like
bash.)

Strictly speaking, locking is only half a solution, because
the net result will be that the saved history is taken from
a randomly chosen one of the multiple exiting shells.  But
that's better than the current situation where all history is lost.

What might be cooler would be to merge all the history lines
from all shells, in timestamp order.  But given the current
history file format, that seems...hard.

CDPATH reports to stdout and even non-interactively

[geoff]

Configuration Information [Automatically generated, do not change]:
Machine: i586
OS: linux-gnu
Compiler: gcc -I/usr/src/packages/BUILD/bash-3.2 
-L/usr/src/packages/BUILD/bash-3.2/../readline-5.2
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i586' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i586-suse-linux-gnu' 
-DCONF_VENDOR='suse' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
-DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -O2 -march=i586 -mtune=i686 
-fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -g -D_GNU_SOURCE 
-DRECYCLES_PIDS -Wall -pipe -g -fPIE -fprofile-use
uname output: Linux bow 2.6.22.13-0.3-default #1 SMP 2007/11/19 15:02:58 UTC 
i686 i686 i386 GNU/Linux
Machine Type: i586-suse-linux-gnu

Bash Version: 3.2
Patch Level: 25
Release Status: release

Description:
If CDPATH is set, whenever bash changes directories to a
non-absolute path it reports the new directory to stdout.
This is done even if bash is running in non-interactive mode,
such as in a script.  That breaks scripts that do things like
this:

(cd /foo/bar; do-something; cd foo; echo Important information) > bar

or, more commmonly, ones that do this:

cd /foo/bar; do-something; cd foo; echo Important information

when their output is redirected.

Repeat-By:
export CDPATH=.:..
cd /
(cd etc) > /tmp/bashcdpath
cat /tmp/bashcdpath
# /etc is shown
cat > /tmp/bashcdpathbug << EOF
#!/bin/bash
cd /etc
echo This should be the only output
cd tmp
EOF
chmod +x /tmp/bashcdpathbug
/tmp/bashcdpathbug > /tmp/bashcdpath2
wc -l /tmp/bashcdpath2
# File contains two lines when it should have only one
rm -f /tmp/bashcdpath*

Fix:
Bash should only report CDPATH status in interactive mode.
Furthermore, CDPATH status (like all such human-interest
information) should be reported to stderr, not stdout.

I would also argue that CDPATH should only be respected in
interactive mode.  Otherwise, there's a potential security
hole with respect to scripts that use relative directory
changes (and potential bugs in other scripts, which is how
this problem as discovered).

Perhaps more than CDPATH is risky

[geoff]

Configuration Information [Automatically generated, do not change]:
Machine: i586
OS: linux-gnu
Compiler: gcc -I/usr/src/packages/BUILD/bash-3.2 
-L/usr/src/packages/BUILD/bash-3.2/../readline-5.2
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i586' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i586-suse-linux-gnu' 
-DCONF_VENDOR='suse' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL 
-DHAVE_CONFIG_H   -I.  -I. -I./include -I./lib   -O2 -march=i586 -mtune=i686 
-fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -g -D_GNU_SOURCE 
-DRECYCLES_PIDS -Wall -pipe -g -fPIE -fprofile-use
uname output: Linux bow 2.6.22.13-0.3-default #1 SMP 2007/11/19 15:02:58 UTC 
i686 i686 i386 GNU/Linux
Machine Type: i586-suse-linux-gnu

Bash Version: 3.2
Patch Level: 25
Release Status: release

Description:
With respect to my previous report on CDPATH, Werner Fink of
SuSE (now Novell) has some additional comments:

Maybe there are some more variables which are normaly used in
interactive mode but also available in script mode.  The question
rises how many scripts are out there relying on such variables
like CDPATH and others:

 #!/bin/bash 
 CDPATH=/tmp
 tmp=$(mktemp -d ${CDPATH}/${0##*/}.XX) || exit 1
 wrk=${tmp##*/}
 cd $wrk

Repeat-By:
I think somebody needs to audit bash to ask the following questions:

(1) If environment variable X is imported to a script, what
are the bad effects?

(2) What is the probability that a script writer will remember
to reset X?  (Heck, how many scripts even reset IFS?  And
that's been well known about sh for 20 years.)

(3) If X is disabled in non-interactive mode, what are the bad
results?

(4) Is it sufficient to null out X on startup, and then let
the script reset it if it chooses?

Without doing the audit, I'd guess that the answer to #4 is
nearly always (or, more likely, always) yes.

Fix:
[Description of how to fix the problem.  If you don't know a
fix for the problem, don't include this section.]

Re: History file clobbered by multiple simultaneous exits

[Geoff Kuenning]

> ge...@cs.hmc.edu wrote:
>>  Locking should be used when truncating and writing the history
>>  file.  (Yes, I know it's a pain in a portable program like
>>  bash.)
>>
>>  Strictly speaking, locking is only half a solution, because
>>  the net result will be that the saved history is taken from
>>  a randomly chosen one of the multiple exiting shells.  But
>>  that's better than the current situation where all history is lost.
>>
>>  What might be cooler would be to merge all the history lines
>>  from all shells, in timestamp order.  But given the current
>>  history file format, that seems...hard.
> 
> You shouldn't write to the same history file from multiple
> sessions.  Encode the session name (or ttyname) in the history file name
> Then they won't collide as you can only have 1 person logged in / tty.

Sigh.

Bug reporter: "Program X doesn't work correctly when used in perfectly
reasonable fashion Y--which happens to be the default."

Maintainer: "You shouldn't do Y.  I don't do Y, therefore nobody else
should do Y either, they should go out of their way and do Z instead.  I
don't want to fix the incorrect behavior even though it can happen in
many situations."

It's normal to open multiple windows.  By default, bash writes to
~/.history.  Furthermore, for perfectly good reasons I want all my
sessions to write to--and more importantly, to read from--the same
history file.

And in this the case the suggested workaround is loaded with flaws.  For
example, it's quite easy to be logged in to the same filesystem multiple
times under the same tty name.  (Think NFS.)

The correct response to a valid bug report isn't "go away and change
your behavior."  It's to fix the bug.
-- 
Geoff Kuenning   ge...@cs.hmc.edu   http://www.cs.hmc.edu/~geoff/

McDonald's, which does not wait on your table, does not cook your food
to order, and does not clear your table, came up with the slogan ``We
Do It All For You.''
-- Dave Barry

Re: History file clobbered by multiple simultaneous exits

[Geoff Kuenning]

I have to apologize; it's clear that we're not connecting.

I must also apologize for misunderstanding your (Linda's) role; I DO
appreciate getting suggestions and help from a bash fan regardless of
whether you're personally responsible for the code.

It's perhaps also worth mentioning that the underlying cause of the
problem isn't so much multiple windows as multiple cores; I've been
running the same multi-windowed setup for decades without trouble but
only started encountering problems when multicore machines arrived,
because it was only then that true simultaneity occurred.

Now, to clarify: the difficulty isn't that bash overwrites the history
file.  That's the default behavior, and it's to be expected.  If a user
opens three shells (in any fashion) and then successively types "exit"
in each, it's to be expected that only the last one's history would be
written to HISTFILE.  And that's what I, personally, want to happen.

But right now, if all three of those shells exit simultaneously--for
whatever reason--there is a significant probability that the history
file will end up zero-length.  That's not theoretical; I've experienced
it multiple times.  And that's a bug, plain and simple.  And I suspect
that it can be fixed on 99+% of all deployed systems by just adding
O_EXCL to the open.

It's certainly helpful to be offered 10+ line solutions to the problem
that can be adapted to my personal situation.  (Special thanks for the
pointer to HISTTIMEFORMAT; I hadn't realized that it affects the
history file and that's a _really_ cool feature!)

But it's still true that the default behavior should be reasonable, and
it's not reasonable to wipe the history file completely when there is
actually tons of history that could be saved.  Nor is it reasonable to
expect millions of bash users to edit their .bashrc to work around the
problem.

(I'll also mention that I'm not swayed by arguments of the general form
"it's really hard to do."  That argument can be justification for
putting a bug into the database and then, in extreme cases, deciding to
indefinitely postpone the fix.  But it's still a bug, and in this
situation the 99% fix is easy.)

>   Loaded with flaws?  You've mentioned one that my default
> system wouldn't have a problem with.  I just simplified my mention
> in a quick note...

As did I.  As another example, we have 50+ machines in our labs.  I
might log in from pine and ten minutes later log in from oak.  I want
the history from the first to be available on the second.

This falls into the general misunderstanding I alluded to in a previous
message, i.e., assuming that one person's solution is applicable to
another's situation.  As I mentioned, I have good reasons for the setup
I use.  That's why I reported a bug rather than asking the
Google-answerable "How can I have different history for different
windows?"

> You are asking for new behavior -- that's asking for a new feature.
> If you can find where bash says it *won't* overwrite files if you
> write to the same name, they you'd have a case for calling it a bug,
> but bash has never had such a feature, thus calling it a bug is
> specious at best.  New feature, yet, bug... stretching the word.

This would be where I am glaringly guilty of bad writing, because if I
had written clearly you wouldn't have said that.  I am most definitely
NOT asking for "no overwrite" behavior.  I'm just asking for "no
simultaneous opens for write"--i.e., O_EXCL.
-- 
Geoff Kuenning   ge...@cs.hmc.edu   http://www.cs.hmc.edu/~geoff/

It's is not, it isn't ain't, and it's it's, not its, if you mean it
is.  If you don't, it's its.  Then too, it's hers.  It isn't her's.  It
isn't our's either.  It's ours, and likewise yours and theirs.
-- Oxford University Press, Edpress News

Re: History file clobbered by multiple simultaneous exits

[Geoff Kuenning]

Linda:

I actually don't use histappend; I like the fact that history eventually
disappears.

Chet:

I have to confess that I'm equally confused.  The probability of a
malloc failure seems vanishingly small; my bashes are only about 20M
virtual, my history size is a measly 500, and my current history is just
over 8K..

I can't think of an interleaving in either 4.2 or 4.3 that would result
in this behavior.  No matter what the interleaving, one of the instances
ought to finish last.

I wrote two test programs to try to reproduce the behavior, and neither
failed.  This makes me wonder if it has to do with how bash deals with
signals (I'm assuming a SIGHUP is involved, though it might be another).
Perhaps bash is getting a second signal during the brief period when the
buffer is being malloc'ed and the history file is being formatted, and
that second signal is killing it before it can do the write?  That seems
unlikely, but sometimes computers surprise us.

> Let's think about why and how this could happen. Maybe that will give some
> insight into how to solve the problem.
>
> The current (bash-4.3-devel) code works something like this, assuming no
> errors (lib/readline/histfile.c:history_do_write()):
>
> rename (histfile, histfile~)
> open file with O_CREAT|O_TRUNC
> malloc buffer large enough to hold all history data
> write all of the history entries in one write(2) call
> close file
> unlink (histfile~)
>
> The bash-4.2 code works the same way except that it does not back up the
> history  file.  Each shell does the same thing when it exits, assuming
> histappend is not set, as in your configuration.
>
> There are a couple of ways the history file can end up zero-length: the
> malloc can fail, or the write can fail.  In bash-4.2, it's too late to do
> anything about the truncated history file at that point.  In bash-4.3, the
> previous history file will be restored.
>
> The O_EXCL solution could work, to some extent, in bash-4.3.  It won't do
> any good in bash-4.2 because it will cause open() to fail under the
> perfectly reasonable circumstance of an existing history file.
>
> What have I missed that could cause file truncation due to a race condition?
>
> Chet
> -- 
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>    ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
>

-- 
Geoff Kuenning   ge...@cs.hmc.edu   http://www.cs.hmc.edu/~geoff/

If we knew what we were doing, it wouldn't be research, would it?
-- Albert Einstein

Re: History file clobbered by multiple simultaneous exits

[Geoff Kuenning]

>   As for the problem...the fact that you using 4.2, would seem
> to make the algorith
> open()
> write(whatever we have as history)
> close(set eof to where we are).
>
> What file system are you are?  is it local or networked?

Local, ext3.

> one way for it to be zero is if the last bash exiting had no history,
> cuz the zero/truncate of each open can zero the file from any previous
> bash being left.

I thought of that too, but it's not the case for me.  Even after the
failure has wiped the old history, my new shells have at least 1-2
commands kicking around.  So I could imagine my nice 500-line history
turning into a 2-line one, but not zero-length.

> I can also see the possibility of some kernel or file system routine
> waiting after you issue the close call so that it doesn't have to zero
> the area where data is arriving.  I.e. it might only zero the file beyond
> the valid text AFTER some delay (5 seconds?) OR might wait until the file
> is closed, so if you completely overwrite the old file with text, the
> kernel won't have to zero anything out.

If so, that would be a big bug.  When you're truncating a file to a
shorter length, some filesystems do indeed delay freeing the blocks in
hopes of reusing them.  But the length is set to zero when the O_TRUNC
happens, and likewise if you write n bytes, the length is immediately
increased by n.  There are certain races on some filesystems that could
cause the n bytes to be incorrect (e.g., garbage), but that generally
happens only on system crashes.  There's a paper on this from a few
years back; I'd have to review it to be sure but my recollection is that
you can't get zero-length files in the absence of system or hardware
failures.  (However, I'm not sure whether they used SMPs...)

Still, I suppose it could be a kernel bug.  Maybe I'll have to write a
better test program and let it run overnight.

> in the case of write...close to non-pre-zeroed record, the operation
> becomes a read-modify-write.  Thing is, if proc 3 goes out for the
> partial buffer
> (~4k is likely), it may have been completely zeroed from proc2 closing
> where proc3
> wants to write.

No generic Linux filesystem that I'm aware of zeroes discarded data at
any time; it's too expensive.  And the partial buffer would be in-memory
at that point, since the processes can exit much faster than the buffer
could be written to disk.  So I don't think that's it.

> (multi-threaded ops on real multiple execution units do the darnest things).

Ain't that the truth!
-- 
Geoff Kuenning   ge...@cs.hmc.edu   http://www.cs.hmc.edu/~geoff/

An Internet that is not Open represents a potentially grave risk to
freedoms of many sorts -- freedom of speech and other civil liberties,
freedom of commerce, and more -- and that openness is what we must so
diligently work to both preserve and expand.
-- Lauren Weinstein

Re: History file clobbered by multiple simultaneous exits

[Geoff Kuenning]

>   Instead of "junk", secure file systems mark it as needing to be
> zeroed.  Perhaps instead of zeroing it ext3 simply marks it of zero
> length?  Imagine, embedded in the junk are credit cards and passwords
> and you'll begin to understand why zero pages are kept "in-stock" (in
> memory) in the kernel so they can rapidly issue a fresh page.

Perhaps I should mention that file systems are my field.

While you're right that secure-deletion file systems clobber deleted
data, that's the exception rather than the rule.  So your guess about
ext3 is also correct: when you truncate a file, ext3 marks it as zero
length but never overwrites the data.  This is easy to prove on any
machine that has a way of monitoring disk activity (either an LED or via
programs like gkrellm or iotop).  Simply create a multi-gigabyte file,
sync the disk, and cp /dev/null to the file.  The disk will be only
briefly active while the free list is updated, not nearly long enough to
rewrite zeros to the freed blocks.

The difference from zero pages in memory is that the API for memory is
designed around the "give me a block and I'll do things with it later"
model.  So asking for a page doesn't inherently clobber it, and the
kernel has to do that for you.  By contrast, the only way to allocate a
page in a file system is to either write to it or to lseek over it.  The
first case automatically clobbers it; in the second it gets marked as
"demand zero if anybody ever reads it" but the disk never gets
written--in fact, no block is allocated unless you write to it.

(As a historical note, Unix at least through V7 didn't demand-zero newly
allocated memory.  Eventually somebody noticed the bug and a wrote
program to steal passwords that way, prompting a change in the kernel
code.  I also believe that in V6 if you wrote a partial block at the end
of a file, you could get leftover data from another file in the
remainder of the block; since you couldn't read it back--it was past
EOF--it was considered OK to have garbage there, and zeroing the memory
before the write was thought to be too expensive.  Unfortunately my
treasured photocopy of the Lions book is in storage while my office is
being moved, so I can't check the V6 code to be sure.)

> It's an edge case usually only seen during some system crash as you mentioned,
> so I can't see how it would cause the symptom you are seeing, but it's only
> seen in crashes in more mature file systems.  Can you reproduce it on another
> file system like xfs?

Good question.  Unfortunately that's not a practical experiment for me
to run.  :-(
-- 
Geoff Kuenning   ge...@cs.hmc.edu   http://www.cs.hmc.edu/~geoff/

Paymasters come in only two sizes: one sort shows you where the book
says that you can't have what you've got coming to you; the second
sort digs through the book until he finds a paragraph that lets you
have what you need even if you don't rate it.  Doughty was the second
sort.
-- Robert A. Heinlein, "The Door Into Summer"

Re: History file clobbered by multiple simultaneous exits

[Geoff Kuenning]

> If you are using whatver `close' button your window manager provides to
> exit the shell rather than exit or ^D, it's possible that there is a signal
> race condition involved here.  Many window managers send SIGHUP/SIGKILL to
> running processes on window close.

That's pretty much what happens.  I use xinit, so when I exit it kills
the X server.  That closes the connections to the (typically 3-4)
xterms, which IIRC then send SIGHUP to their child shells.

> Bash-4.2 runs its handlers for terminating signals like SIGHUP immediately,
> interrupting whatever is in progress.  That, coupled with attempting to
> save the history from interactive shells killed this way, could result in
> zero-length files if the signals arrive at the right time.

True, but "the right time" is pretty skinny.  My typical behavior is to
type "sync" (ancient habit that I doubt I'll ever break, dating back to
the days of major unreliability) and then kill X.  The net is that all
my bash instances are idle at that point, sitting at the command
prompt.  So even if the signal handlers are calling the history-writing
code directly, life should be OK unless a second SIGHUP arrives.  (Does
bash re-enable signals when the handler gets called?  Seems unlikely,
but...)
-- 
Geoff Kuenning   ge...@cs.hmc.edu   http://www.cs.hmc.edu/~geoff/

The "M" in XML stands for _markup_.  If you don't have anything
outside the angle brackets, you probably shouldn't be using XML.

Re: [sr #108491] Is `set -o histexpand` meant to trump `set -o posix`? [eg, echo "#!/"] (Because it does.)

[Geoff Nixon]

So having done some more playing around with this, I've found that when 
bash is configured and compiled with '--enable-strict-posix-default', 
this doesn't occur.  I see this is documented in the change log for 
4.3alpha, as:


  k.  When compiled for strict Posix conformance, history expansion is 
disabled by default.


But why is this a compile-time-only behavior?

There's nothing else in the documentation about any differences between 
bash compiled with -DSTRICT_POSIX v.s. enabling this mode with `bash 
--posix`. It just says that POSIX mode will become a default option. 
And reverse also seems to be true; if you compile with -DSTRICT_POSIX, 
invoke the shell, then 'set +o posix', history expansion remains 
disabled? This definitely seems like it could cause some trouble.


URL: http://savannah.gnu.org/support/?108491

Summary: Is `set -o histexpand` meant to trump `set -o posix`? [eg, 
echo "#!/"] (Because it does.)

Project: The GNU Bourne-Again SHell
Submitted by: None
Submitted on: Sat 01 Feb 2014 01:32:34 PM UTC
Category: None
Priority: 5 - Normal
Severity: 5 - Blocker
Status: None
Privacy: Public
Assigned to: None
Originator Email: geoff.ni...@aol.com
Open/Closed: Open
Discussion Lock: Any
Operating System: *BSD

   ___

Details:

From: geoff.ni...@aol.com
To:  c...@po.cwru.edu, chet.ra...@case.edu, 
c...@cwru.edu,bash-test...@case.edu, bash-test...@cwru.edu


Subject: Is `set -o histexpand` meant to trump `set -o posix`? [eg, 
echo "#!/"] (It does.)


Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: darwin13.1.0
Compiler: /usr/bin/clang
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='darwin13.1.0' 
-DCONF_MACHTYPE='x86_64-apple-darwin13.1.0' -DCONF_VENDOR='apple'
-DLOCALEDIR='/Users/Shared/usr/local/opt/bash/share/locale' 
-DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -DMACOSX -DDEBUG -DMALLOC_DEBUG -I.  -I. 
-I./include -I./lib -g -O2
uname output: Darwin air.local 13.1.0 Darwin Kernel Version 13.1.0: Thu 
Jan 16

19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64
Machine Type: x86_64-apple-darwin13.1.0

Bash Version: 4.3
Patch Level: 0
Release Status: rc2

Description:

Please see these:

 
http://stackoverflow.com/questions/21494213/why-cant-i-echo-the-characters-in-a-string-in-bash

http://stackoverflow.com/questions/21496049/what-is-the-intended-effective-ordering-of-set-o-options-in-bash-does-histe

Its still not at all clear to me if I'm correct or not; and I kind of 
got a
serious beatdown for calling it a bug on Stack Overflow. So I'm not 
sure if
this is a 'blocker' per se or even something you'll consider a bug. But 
it
certainly blocks me from wanting to using bash to execute a 
POSIX-portable

script!

(Note that I now fully realize my statement there to the effect of 
'bash never
sees any updates so use a different shell' is erroneous. I made this 
claim
because a keep a git checkout on my hard drive which gets pulled daily 
- but
your master branch is configured to track 4.2 rather than devel, so 
that's
what I thought was indeed the case. I was completely aghast to find 
that 4.3
is in rc2 having just made that statement, and I intend to redact it a 
soon as

there's some resolution to this "bug", be it a bug or not.)

I apologize if this appears in your inbox two dozen times. I don't know 
if

sendmail even works at all on my system; and there are three slightly
different emails in the bashbug script, in git, and the top hit in a web
search for your name. I'm trying to make sure this gets seen before the
'release' button gets pushed for 4.3.

Repeat-By:
set -o histexpand && set -o posix && echo "#!/"
   set -o histexpand && set -o posix && echo "#!/bin/bash"
echo $?

Fix:
Disable `set -o histexpand` while parsing and expanding double-quoted
strings, at least for lines which begin with a hash symbol, 
when `set -o posix` is set.


   ___

Reply to this item at:

 

Fix for HISTCMD when HISTFILE is defined, but HISTSIZE is not

[Geoff Odhner]

Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' 
-DCONF_VENDOR='pc' -DLOCALEDIR='/home/geoff/i686/share/locale' -DPACKAGE='bash' 
-DSHELL -DHAVE_CONFIG_H  -I.  -I.. -I../include -I../lib   -g -O2
uname output: Linux monster 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 
i686 i686 i386 GNU/Linux
Machine Type: i686-pc-linux-gnu

Bash Version: 3.0
Patch Level: 16
Release Status: release

Description:
Unsetting HISTSIZE allows an unlimited number of entries in the 
history, but it unfortunately causes $HISTCMD to evaluate as 1.
This should only happen if HISTFILE is unset.

Repeat-By:
execute the following two command lines:

% unset HISTSIZE
% echo HISTCMD=$HISTCMD
HISTCMD=1
% 

Fix:
The following patch corrects the problem, making HISTCMD
available whenever there is a history file:

==
diff -Naur bash-3.0/bashhist.c bash-3.0-fixed/bashhist.c
--- bash-3.0/bashhist.c   2004-03-22 08:27:59.0 -0500
+++ bash-3.0-fixed/bashhist.c 2005-09-29 09:53:19.903722088 -0400
@@ -706,7 +706,7 @@
 history_number ()
 {
   using_history ();
-  return (get_string_value ("HISTSIZE") ? history_base + where_history () : 1);
+  return (get_string_value ("HISTFILE") ? history_base + where_history () : 1);
 }
 
 static int
==


___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash

Fix for HISTCMD when HISTFILE is defined, but HISTSIZE is not

[Geoff Odhner]

Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' 
-DCONF_VENDOR='pc' -DLOCALEDIR='/home/geoff/i686/share/locale' -DPACKAGE='bash' 
-DSHELL -DHAVE_CONFIG_H  -I.  -I.. -I../include -I../lib   -g -O2
uname output: Linux monster 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 
i686 i686 i386 GNU/Linux
Machine Type: i686-pc-linux-gnu

Bash Version: 3.0
Patch Level: 16
Release Status: release

Description:
Unsetting HISTSIZE allows an unlimited number of entries in the 
history, but it unfortunately causes $HISTCMD to evaluate as 1.
This should only happen if HISTFILE is unset.

Repeat-By:
execute the following two command lines:

% unset HISTSIZE
% echo HISTCMD=$HISTCMD
HISTCMD=1
% 

Fix:
The following patch corrects the problem, making HISTCMD
available whenever there is a history file:

==
diff -Naur bash-3.0/bashhist.c bash-3.0-fixed/bashhist.c
--- bash-3.0/bashhist.c   2004-03-22 08:27:59.0 -0500
+++ bash-3.0-fixed/bashhist.c 2005-09-29 09:53:19.903722088 -0400
@@ -706,7 +706,7 @@
 history_number ()
 {
   using_history ();
-  return (get_string_value ("HISTSIZE") ? history_base + where_history () : 1);
+  return (get_string_value ("HISTFILE") ? history_base + where_history () : 1);
 }
 
 static int
==


___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash

Re: CDPATH reports to stdout and even non-interactively

[Geoff Kuenning]

> And in general, it is a bad idea to export CDPATH.  Using it in
> interactive shells as a non-environment shell variable is fine, but
> exporting it affects non-interactive shells.

There are good reasons to export CDPATH.  In particular, CDPATH is
inherited by interactive subshells.  For precisely that reason, my
startup files are careful to avoid resetting CDPATH, so that if I want
to change it and then invoke an interactive subshell I won't have to
change it a second time.

In general, it's a bad idea to assume that one's own usage patterns
match those of all other users.
-- 
Geoff Kuenning   [EMAIL PROTECTED]   http://www.cs.hmc.edu/~geoff/

The P in "PDF" is a lie.

Re: CDPATH reports to stdout and even non-interactively

[Geoff Kuenning]

ffects should spend some
time thinking about how to crack a script by
manipulating it.

TMPDIR  This one is pretty well known.  It's abusable, but
removing it would be pretty bad for backwards
compatibility.
-- 
Geoff Kuenning   [EMAIL PROTECTED]   http://www.cs.hmc.edu/~geoff/

The P in "PDF" is a lie.

Re: CDPATH reports to stdout and even non-interactively

[Geoff Kuenning]

> Nope.  Look at the -p option for set.  BASH_ENV can be used to cause
> scripts to go haywire, but only with your own account.

Thanks, I missed that section of the manual.  BASH_ENV is already
ignored when euid != ruid.

> Not necessarily.  In some cases, it may be that a script relies on the
> inherited CDPATH as a way for the user to tell it where to operate -
> that is, a script might treat it just like HOME, PATH, and TMPDIR.  In
> cases like that, the script can't know what value the user wants
> CDPATH to be set to.  (Though it's still true that many other scripts
> don't treat it that way and may be vulnerable to abuse.)

It would be interesting to know how many scripts use this behavior.
Unfortunately, I can't think of a way to research it.

> CDPATH and GLOBIGNORE could at least be added to the -p handling, I'd
> say.

I'd argue for that at a minimum.  It wouldn't fix the situation I
encountered, but it would help.

> The trouble with unsetting an inherited variable in non-interactive
> shells is that it screws up the situation where an interactive shell
> invokes a non-interactive shell, which then invokes another
> interactive shell.

Good point.

Chet Ramey writes:

>> GLOBIGNORE  Should be unset in non-interactive mode.  I can't come
>> up with a crack in 10 seconds, but I'm confident that
>> within 30 minutes I could figure out a way to abuse a
>> script by controlling its globbing.
>
>I'd be interested in seeing that exploited to do something malicious.

Well, I don't really have 30 spare minutes to work something out.  But
I might start by arranging for a script that creates and then removes
temp files to fail to remove them due to globbing.  Then I might look
for situations where I could cause a denial of service by causing lock
files to not get removed.
-- 
Geoff Kuenning   [EMAIL PROTECTED]   http://www.cs.hmc.edu/~geoff/

The most exciting phrase to hear in science, the one that heralds new
discoveries, is not "Eureka!" (I found it!) but "That's funny ..."
-- Isaac Asimov

Bash-3.0.16 hangs when running this pipeline: set | grep BASH

[Geoff Odhner]

Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' 
-DCONF_VENDOR='pc' -DLOCALEDIR='/home/geoff/i686/share/locale' -DPACKAGE='bash' 
-DSHELL -DHAVE_CONFIG_H  -I.  -I.. -I../include -I../lib   -g -O2
uname output: Linux mica 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686 
i686 i386 GNU/Linux
Machine Type: i686-pc-linux-gnu

Bash Version: 3.0
Patch Level: 16
Release Status: release

Description:
When I run a pipeline such as "set | grep BASH", bash will hang.
I did not have this problem at patch level 3.0.1.  (Is this a 
side effect of the fix in patch bash30-012?)

In gdb I see that control is looping from line 1869 through line
1935 in wait_for() in jobs.c.  Both the "set" process and the "grep"
process are defunct:

geoff22692  0.0  0.0 00 pts/2Z+   17:22   0:00 [bash] 

geoff22693  0.0  0.0 00 pts/2Z+   17:22   0:00 [grep] 


Unfortunately I do not have the time to learn study the system
long enough to understand why bash did not mark the processes as
dead.

Repeat-By:
%  set | grep BASH

-- 
Geoff Odhner <[EMAIL PROTECTED]>
Franklin Electronic Publishers


___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash

Re: Bash-3.0.16 hangs when running this pipeline: set | grep BASH

[Geoff Odhner]

Thanks very much.  This does beautifully.

Geoff Odhner


Chet Ramey <[EMAIL PROTECTED]> wrote on 05/06/2005 09:35:18 PM:

> Geoff Odhner wrote:
> > Configuration Information [Automatically generated, do not change]:
> > Machine: i686
> > OS: linux-gnu
> > Compiler: gcc
> > Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' -
> DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' -
> DCONF_VENDOR='pc' -DLOCALEDIR='/home/geoff/i686/share/locale' -
> DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H  -I.  -I.. -I../include -I..
> /lib   -g -O2
> > uname output: Linux mica 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 
> EST 2005 i686 i686 i386 GNU/Linux
> > Machine Type: i686-pc-linux-gnu
> > 
> > Bash Version: 3.0
> > Patch Level: 16
> > Release Status: release
> > 
> > Description:
> >When I run a pipeline such as "set | grep BASH", bash will hang.
> >I did not have this problem at patch level 3.0.1.  (Is this a 
> >side effect of the fix in patch bash30-012?)
> 
> A patch is available here:
> 
> http://lists.gnu.org/archive/html/bug-bash/2004-09/msg00081.html
> 
> Chet
> -- 
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ( ``Discere est Dolere'' -- chet )
>   Live...Laugh...Love
> Chet Ramey, ITS, CWRU[EMAIL PROTECTED] http://cnswww.cns.cwru.edu/~chet/



___
Bug-bash mailing list
Bug-bash@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-bash

Wildcard expansion can fail with nonprinting characters

[Geoff Kuenning]

Version:

GNU bash, version 4.4.23(1)-release (x86_64-suse-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>


Behavior:

If a pathname contains nonprinting characters, and is expanded 
from a variable name, wildcard expansion can sometimes fail.


Example (with actual nonprinting characters):

$ mkdir /tmp/test$'\361'dir
$ touch /tmp/test�dir/foo
$ ls /tmp/test�dir/f*
/tmp/test?dir/foo
$ x=/tmp/test�dir
$ echo "$x" | cat -v
/tmp/testM-qdir
$ ls "$x"/f*
ls: cannot access '/tmp/test'$'\361''dir/f*': No such file or 
directory


(Note: in the above example, I completed the "ls" line and 
generated x by using tab completion.  However, the bug happens 
even if the variable is set by other means, such as from the 
output of a script that generates the nonprinting pathname.)


Example repeated, with nonprinting character replaced with M-q in 
case mailers mangle them:


$ mkdir /tmp/test$'\361'dir
$ touch /tmp/testM-qdir/foo
$ ls /tmp/testM-qdir/f*
/tmp/test?dir/foo
$ x=/tmp/testM-qdir
$ echo "$x" | cat -v
/tmp/testM-qdir
$ ls "$x"/f*
ls: cannot access '/tmp/test'$'\361''dir/f*': No such file or 
directory


Expected behavior: ls with "$x/*" should list foo in both cases.

Observed behavior: when foo is accessed via variable substitution, 
the wildcard is not expanded.


Other comments: I've observed similar behavior with tab expansion 
and complex pathnames that contain "surprising" characters such as 
parentheses, spaces, and nonprinting characters, but I'm not sure 
whether it's related.

--
   Geoff Kuenning   ge...@cs.hmc.edu 
   http://www.cs.hmc.edu/~geoff/


The major difference between a thing that might go wrong and a 
thing

that cannot possibly go wrong is that when a thing that cannot
possibly go wrong goes wrong it usually turns out to be impossible 
to

get at or repair.
   -- Douglas Adams

Re: Wildcard expansion can fail with nonprinting characters

[Geoff Kuenning]

$'\361' is a valid character in Latin-1, which is how it happened 
to arise in my case.  Also, I tested with the C locale, which 
should be agnostic to character encodings, and got the same 
result.


The general Unix philosophy, which in this case says "I'm not 
going to pass judgment on the weird things you do even though I 
don't understand them", argues for being able to handle any 
arbitrary sequence of bytes, at least on Linux.  That's one of the 
things that makes the Unix paradigm so powerful.  So I appreciate 
your willingness to fix this.



On 9/27/19 7:52 PM, Geoff Kuenning wrote:

Version:

GNU bash, version 4.4.23(1)-release (x86_64-suse-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>


Behavior:

If a pathname contains nonprinting characters, and is expanded 
from a

variable name, wildcard expansion can sometimes fail.


This is an interesting report. The $'\361' is a unicode 
combining
character, which ends up making the entire sequence of 
characters an

invalid wide character string in a bunch of different locales.

Some file systems (Mac OS X APFS) don't allow you to create 
files with
invalid characters or character sequences in their names. Others 
(Linux)

don't have a problem with it.

The code to dequote filenames that's needed for "$x" tries to 
fall back to
single-byte character operations in the presence of invalid 
character or
byte sequences, but that means you can't use any of the standard 
wide
character functions to check for valid and invalid wide 
character strings.


The change between bash-4.4 and bash-5.0 is that the globbing 
code doesn't
bother to try and convert to wide characters to do the dequoting 
if there
aren't any valid multibyte characters in the pathname, but uses 
the single
byte character code instead. That works for this case, but 
doesn't work for
pathnames that have both valid and invalid wide character 
sequences.


A better fix is to write a symmetric function that will take the 
output of
xdupmbstowcs2 (bash's replacement for mbstowcs that handles 
zero-length
wide character strings that aren't null wide characters) and 
handle the
invalid wide character strings that may result from it. I'll 
make that fix

for the next release.

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.edu 
http://tiswww.cwru.edu/~chet/




--
   Geoff Kuenning   ge...@cs.hmc.edu 
   http://www.cs.hmc.edu/~geoff/


Orchestra retrospectively extremely satisfied with symphony 
[No. 1] as

result of barrel of free beer.
-- Gustav Mahler, post-premiere letter to Arnold Berliner

"Variation" in Command Substitution Behaviour

[Geoff Hull]

I think the following is a bug, but I'm not really sure. I've checked the
documentation for differences between old-style command substitution (
`...` ) and new-style ( $(...) ), and couldn't see anything that related to
this problem.

*Background*

At my place of work, I use some loops with variables to generate a lot of
aliases and functions. Our employees use these to modify variables,
including their PATH, to change their environment for testing different
versions of our Cobol products. When I made a recent change, some of the
PATH modifications stopped working. (Before diving in to investigate the
issue, I changed the aliases to functions to get things working again.)

*Test Case*

I have created a much simpler set of test aliases and functions to
illustrate the issue and I have to admit it's a bit obscure.

If I "source" the attached file (i.e. ". test_aliases") in a bash session,
then run the following:

assemble_fam1
assemble_fam2
say_families

I see the following output:

Flintstones=wilma:bam-bam:fred
Rubbles=barney

I think I should be seeing this output:

Flintstones=wilma:bam-bam:fred
Rubbles=betty:pebbles:barney

Interestingly, if I run the function 'run_aliases' to run the assemble_famX
aliases, then it all works fine.

Another change that also makes the assemble_fam2 alias work properly (apart
from using backticks) is to write the assemble_fam2 alias on one line as:

alias assemble_fam2="barney; RUBBLES=betty:pebbles:\$RUBBLES"

Hopefully, you can use this information to see if there is a problem.

Cheers,
Geoff


test_aliases
Description: Binary data

Re: "Variation" in Command Substitution Behaviour

[Geoff Hull]

Hi Reuti, Andreas,

Yes, forgot to mention the bash versions I have tried: 4.3.46 on Arch Linux
and 4.1.2 on CentOS 6.8.

This is getting weirder: I've found I can replicate the behaviour you are
getting Reuti, if I first do this command:

unset COMMAND_PROMPT

If I want to go back to the behaviour I first experienced (i.e.
Rubbles=barney after running assemble_fam1 then assemble_fam2), then I put
the default COMMAND_PROMPT back:

PROMPT_COMMAND='printf "\033]0;%s@%s:%s\007" "${USER}" "${HOSTNAME%%.*}"
"${PWD/#$HOME/~}"'

Does this work the same for you?

Geoff


On 1 March 2017 at 02:21, Reuti  wrote:

> Hi,
>
> > Am 28.02.2017 um 11:00 schrieb Andreas Schwab :
> >
> > On Feb 28 2017, Geoff Hull  wrote:
> >
> >> If I "source" the attached file (i.e. ". test_aliases") in a bash
> session,
> >> then run the following:
> >>
> >> assemble_fam1
> >> assemble_fam2
> >> say_families
> >>
> >> I see the following output:
> >>
> >> Flintstones=wilma:bam-bam:fred
> >> Rubbles=barney
>
> In which version of Bash are you observing this? For me it's working using
> this order, but when I swap assemble_fam1 and assemble_fam2 I observe a
> strange behavior, that RUBBLES gets reset and I get your output:
>
> $ assemble_fam1
> $ assemble_fam2
> $ say_families
> Flintstones=wilma:bam-bam:fred
> Rubbles=betty:pebbles:barney
> $ reset_vars
> $ assemble_fam2
> $ say_families
> Flintstones=
> Rubbles=betty:pebbles:barney
> $ assemble_fam1
> $ say_families
> Flintstones=wilma:bam-bam:fred
> Rubbles=barney
>
>
> This I have in 4.2 and 4.4.
>
> $ reset_vars
> + reset_vars
> + unset FSTONES RUBBLES
> $ assemble_fam2
> + barney
> ++ echo barney
> + RUBBLES=barney
> $ say_families
> + RUBBLES=betty:pebbles:barney
> + say_families
> + echo Flintstones=
> Flintstones=
> + echo Rubbles=betty:pebbles:barney
> Rubbles=betty:pebbles:barney
> $ assemble_fam1
> + fred
> ++ echo fred
> + FSTONES=fred
> + FSTONES=wilma:bam-bam:fred
> $ say_families
> + say_families
> + echo Flintstones=wilma:bam-bam:fred
> Flintstones=wilma:bam-bam:fred
> + echo Rubbles=barney
> Rubbles=barney
>
>
> > It seems like the shell is eating an extra newline.  Try typing an empty
> > line after assemble_fam2:
>
> The extra newline doesn't help for the observation above. While defining
> assemble_famX in one line for both does as mentioned by Geoff.
>
> -- Reuti
>

Re: "Variation" in Command Substitution Behaviour

[Geoff Hull]

Aha! That's it. Case closed, I think.

Thanks,
Geoff

On 2 March 2017 at 05:07, Reuti  wrote:

>
> > Am 01.03.2017 um 16:53 schrieb Grisha Levit :
> >
> > On Mar 1, 2017 1:04 AM, "Geoff Hull"  wrote:
> > If I want to go back to the behaviour I first experienced then I put the
> default COMMAND_PROMPT back
> >
> > I reported the multi-line alias and PROMPT_COMMAND interaction last
> June. The behavior you note should be fixed in 4.4.
> >
> > https://lists.gnu.org/archive/html/bug-bash/2016-06/msg00020.html
>
> Aha, typo - yes: setting PROMPT_COMMAND gives your reported behavior
> Grisha.
>
> -- Reuti
>