Re: "here strings" and tmpfiles
On Mon, Mar 18, 2019 at 05:18:10PM -0400, Daniel Kahn Gillmor wrote: > strace -o tmp/bash.herestring.strace -f bash -c 'cat <<<"hello there"' > It turns out that this creates a temporary file, actually touching the > underlying filesystem: Yes, just like here documents do. And have always done, in all shells. > For example, sending a password or secret key material from the > environment to stdin would be a typical way to use a herestring. Note that the environment may also be visible to users on the system, via "ps eww aux" or similar commands. The availability of the BSD ps(1) options and the details about who can see what are OS-specific. > A few possible options for trying to improve the situation: Don't put sensitive data in shell scripts? In general: 1) Do not pass secrets as arguments to external commands. The arguments of a command are generally visible in ps(1). 2) Do not pass secrets as environment variables. The initial environment of a process is generally visible in ps(1). 3) Read the documentation of the thing you're trying to authenticate against. Find out the various ways it can accept authentication secrets/tokens and choose the most appropriate. This may mean using ssh keys stored in an ssh-agent, etc. 4) If something requires a password, let it prompt the user for the password by itself. Just run it inside a terminal so that it can launch a dialog with the end user if required. Do not try to "help" it by storing the password in your script and then trying to figure out how to circumvent its security in order to pass the password to it. 5) If you absolutely MUST store a password somewhere on disk, don't store it inside the shell script. Shell scripts must have read permissions in order to be used. Store the password in a separate file that doesn't have universal read permission, and let the appropriate process read that file. The appropriate process may be your script in rare cases, but more often it'll be whatever program is actually going to use that password. Thus, there is absolutely no reason you should ever have a secret password inside a here document or here string. That would mean the password is hard-coded inside a shell script, which violates several of my points. If the password has been read from a file and is now inside a shell variable (NOT environment variable) in memory, and you want to pass it on stdin to a process, you do that by running something like printf %s\\n "$secret" | program No here strings are wanted. printf is a builtin, so you aren't violating point 1. $secret is not exported, so you aren't violating point 2. https://mywiki.wooledge.org/BashFAQ/069 and https://mywiki.wooledge.org/BashFAQ/078 also touch on this, but they could both use some expansion/rewriting, I see.
cd "" return value
Good morning, In GNU bash, version 5.0.2(1)-release (x86_64-pc-linux-gnu): The command cd "" is successful but cd -P "" fails. This is inconsistent and not evident in man bash. Expected is that cd "$(mktemp -d)" returns error if mktemp fails. BASH(1) cd ... The return value is true if the directory was successfully changed; false otherwise.
Re: "here strings" and tmpfiles
On Mon 2019-03-18 17:18:10 -0400, Daniel Kahn Gillmor wrote:
> A few possible options for trying to improve the situation:
>
> a) use socketpair(2) or pipe(2) instead of making a tmpfile. this has
> the potential downside that the semantics of access to the remaining
> file descriptor would be subtly different from "regular file"
> semantics.
>
> b) On systems that support O_TMPFILE, try something like
> open("/dev/shm", O_RDWR|O_CREAT|O_EXCL|O_TMPFILE). /dev/shm tends
> to be a globally-writable tmpfs, so that avoids touching any disk,
> and O_TMPFILE avoids tmpfile-style race conditions. This might need
> a fallback (to the current tmpdir selection mechanics?) in case
> /dev/shm isn't available.
>
> c) Just use O_TMPFILE with the current tmpdir selection mechanics, if
> it's supported. This isn't quite as clever as trying to use
> /dev/shm first, and it won't fix the herestrings hitting the disk,
> but it at least avoids tmpfile races.
>
> d) If none of the above can be done, at the very least, bash(1)'s
> section on here docs and here strings should warn that the contents
> of these documents are likely to get written to the disk
> unprotected.
One more possibility for an implementation fix occurs to me (at least on
systems with Linux >= 3.17 and glibc >= 2.27):
e) bash could use use memfd_create(2) for heredocs and herestrings --
that should preserve "regular file" semantics, avoid tmpfile races,
and avoid hitting the disks.
--dkg
signature.asc
Description: PGP signature
Re: "here strings" and tmpfiles
Thanks for the feedback, Eduardo-- On Mon 2019-03-18 17:40:17 -0700, Eduardo A. Bustamante López wrote: > I don't think the implementation details of herestrings are documented > anywhere, > and I'm not too sure if they should (i.e. IMO if you need that degree of > control > over the implementation details, then you should use something other than > shell). I hear you in general -- i also don't want the documentation to be as detailed as the source code. But casually sending ephemeral data to disk is a risk that i think ought to be avoided or at least avoidable. If bash was in the habit of writing the environment to disk, i think users would rightly complain. > Having said that, have you tried process substitution as an option? sure, that's an option (as long as process substitution is enabled on the platform -- apparently that's not universal either). Also possible (for stdin in particular) is sending data via a pipeline using bash builtins. Both of these require users of bash to rewrite their scripts though. It seems like it'd be preferable for the shell itself to avoid these problems automatically, at least on platforms where it's possible to do so. Otherwise, we *require* users of the shell to know which things are "safe" and which things aren't before they can use the shell safely. --dkg
Re: "here strings" and tmpfiles
On Tue, Mar 19, 2019 at 09:20:33AM -0400, Daniel Kahn Gillmor wrote: > On Tue 2019-03-19 08:25:50 -0400, Greg Wooledge wrote: > > On Mon, Mar 18, 2019 at 05:18:10PM -0400, Daniel Kahn Gillmor wrote: > >> strace -o tmp/bash.herestring.strace -f bash -c 'cat <<<"hello there"' > >> It turns out that this creates a temporary file, actually touching the > >> underlying filesystem: > > > > Yes, just like here documents do. And have always done, in all shells. > > Apologies for being unaware of the history. It looks like there are a > handful of possible approaches today that minimize these fixes, which > may not have been possible on older systems, which i listed upthread. > And they work on arbitrary file descriptors, not just stdin. > > Do you think that bash should not improve the situation, at least on > platforms that support these other approaches? There are scripts that *rely* on the seekability of the temporary files created by here-documents and here-strings. "Improving" the "situation" would break backward compatibility. I already showed how to use a pipeline to send information from a shell variable to another process's stdin without using a here-string (which is a bashism that is much less portable than the pipeline, in addition to being less "safe" in the incredibly narrow niche situation being described in this thread). There is simply NO valid reason to write <<<"$secret" in a script, and thus there is no need to "improve" anything other than the scripts that are doing that. Use a pipe instead.
Re: "here strings" and tmpfiles
On Tue 2019-03-19 08:25:50 -0400, Greg Wooledge wrote: > On Mon, Mar 18, 2019 at 05:18:10PM -0400, Daniel Kahn Gillmor wrote: >> strace -o tmp/bash.herestring.strace -f bash -c 'cat <<<"hello there"' >> It turns out that this creates a temporary file, actually touching the >> underlying filesystem: > > Yes, just like here documents do. And have always done, in all shells. Apologies for being unaware of the history. It looks like there are a handful of possible approaches today that minimize these fixes, which may not have been possible on older systems, which i listed upthread. And they work on arbitrary file descriptors, not just stdin. Do you think that bash should not improve the situation, at least on platforms that support these other approaches? --dkg signature.asc Description: PGP signature
Re: cd "" return value
On 3/19/19 4:21 AM, furrymc...@lippydanger.jumpingcrab.com wrote: > > Good morning, > > In GNU bash, version 5.0.2(1)-release (x86_64-pc-linux-gnu): > The command cd "" is successful but cd -P "" fails. > This is inconsistent and not evident in man bash. > Expected is that cd "$(mktemp -d)" returns error if mktemp fails. This is a consequence of the pathname canonicalization that takes place using the supplied argument, because the -L option is the default. Pretty much every POSIX-conformant shell behaves like this. > The return value is true if the directory was successfully > changed; false otherwise. After canonicalization, it's equivalent to cd $PWD, which is successful. The -P option forces an attempt to cd to "" directly, which fails because chdir(2) fails with ENOENT when its argument is the empty string. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: "here strings" and tmpfiles
Date:Tue, 19 Mar 2019 08:25:50 -0400 From:Greg Wooledge Message-ID: <20190319122550.khv5jp66iobjo...@eeg.ccf.org> | Yes, just like here documents do. And have always done, in all shells. That's not correct. There are shells that don't use files for here docs. Any application that relies on stdin being seekable is broken (unless it makes that happen for itself) - the most obvious example which is not seekable is when stdin is a terminal. The same applies to any other file descriptor which is opened when an application is started. POSIX (XCU section 2.7.4) explicitly says (of here docs): It is unspecified whether the file descriptor is opened as a regular file, a special file, or a pipe. Portable applications cannot rely on the file descriptor being seekable (see XSH lseek( )). Note: I am not suggesting bash should change - using files for here docs is the way they were originally implemented (in the Bourne sh) (though it had bugs, which could leave the files lying around in some cases). However, using files for here docs makes here docs unusable in a shell running in single user mode with no writable filesystems (whatever is mounted is read only, until after file system checks are finished). kre
