Re: Suggestions for improving Bash
On Mon, Apr 20, 2020 at 10:25:30PM -0400, Dale R. Worley wrote: > Is this the proper place to make suggestions for improving Bash? Yes. It should also be noted that there is no public bug tracker, other than the archives of this mailing list. So, it can be difficult to tell which suggestions have already been made.
Re: Suggestions for improving Bash
On 4/20/20 10:25 PM, Dale R. Worley wrote: > Is this the proper place to make suggestions for improving Bash? Yes, you can post bug reports and feature suggestions here. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Suggestions for improving Bash
On 4/21/20 7:17 AM, Greg Wooledge wrote: > On Mon, Apr 20, 2020 at 10:25:30PM -0400, Dale R. Worley wrote: >> Is this the proper place to make suggestions for improving Bash? > > Yes. It should also be noted that there is no public bug tracker, That's not quite true. There is a bug tracker on savannah (https://savannah.gnu.org/support/index.php). It's just not as frequently used. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: process substitution fd lifetime race condition
On 4/21/20 12:29 AM, Jason A. Donenfeld wrote: > Were you planning on committing this to Savannah? Yes, this will most likely end up as patch 17, unless a higher-priority thing comes up first. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: pasting in bracketed-paste mode breaks reverse-i-search
On 4/20/20 9:32 PM, gentoo_eshoes--- via Bug reports for the GNU Bourne Again SHell wrote: > pasting while in reverse-i-search undoes it > Repeat-By: > $ bind 'set enable-bracketed-paste on' > select character 'c' with the mouse, so it can be pasted via pressing MMB or > shift+insert > press Ctrl+R to enter (reverse-i-search)`': > paste, (either by pressing MMB or shift+insert) > the reverse-i-search prompt goes away and you're back to regular prompt, > having just 'c' pasted there: $ c It's the ESC prefix. You can use it if you remove ESC from the list of characters that terminate an incremental search. The problem is that most of the key sequences that begin with ESC (e.g., arrow keys) are intended to terminate the isearch and use the ESC as the first character of a key sequence. You have to look farther ahead in the input stream than incremental search presently does to differentiate between an ESC-prefixed key sequence you want to use for searching and one you want to terminate the search and use as a command. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [PATCH] operate-and-get-next skips command after a multi-line command
On 4/19/20 7:58 PM, Greg Price wrote: > Bash Version: 5.0 > Patch Level: 3 > Release Status: release > > Description: > > When the function operate-and-get-next (C-o) is used on a multi-line > command, if the history is full, the next prompt shows not the next > command but the one after that. Thanks for the report and patch. It will be in the next devel branch push. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: pasting in bracketed-paste mode breaks reverse-i-search
Apr 21, 2020, 16:14 by chet.ra...@case.edu: > On 4/20/20 9:32 PM, gentoo_eshoes--- via Bug reports for the GNU Bourne > Again SHell wrote: > >> pasting while in reverse-i-search undoes it >> Repeat-By: >> $ bind 'set enable-bracketed-paste on' >> select character 'c' with the mouse, so it can be pasted via pressing MMB or >> shift+insert >> press Ctrl+R to enter (reverse-i-search)`': >> paste, (either by pressing MMB or shift+insert) >> the reverse-i-search prompt goes away and you're back to regular prompt, >> having just 'c' pasted there: $ c >> > > It's the ESC prefix. You can use it if you remove ESC from the list of > characters that terminate an incremental search. The problem is that most > of the key sequences that begin with ESC (e.g., arrow keys) are intended to > terminate the isearch and use the ESC as the first character of a key > sequence. You have to look farther ahead in the input stream than > incremental search presently does to differentiate between an ESC-prefixed > key sequence you want to use for searching and one you want to terminate > the search and use as a command. > Did you mean to say that bracketed-mode pastes ESC, perhaps as a way to highlight(ie. that white background) ? interesting static char * const default_isearch_terminators = "\033\012"; ESC and newline(ie. \n) hmm... oh hey, I found out how I know that Ctrl+R is the key that invokes reverse-i-search: $ bind -q reverse-search-history reverse-search-history can be invoked via "\C-r". ok, so to translate what you said... the default is this: $ bind "set isearch-terminators '\033\012'" even though `bind -v | grep isearch-terminators` shows nothing before I run that command because it's not set. So if I run this: $ bind "set isearch-terminators '\012'" then the issue is gone and reverse-i-search works, with the caveat that it breaks when you try to rpess left/right arrows by it typing in "[D" respectively "[C" , but it doesn't happen the first time right after paste, only if I just do Ctrl+R type some string, press arrow... I see that left/right arrow keys yield ESC[D and ESC[C respectively. So it eats the ESC, it still terminates the i-search, then continues to type what's left: "[D" or "[C". Confused as to why. I thought ESC shouldn't terminate i-search anymore, given that I've set it to just newline(\012 aka \x0a). At this point I'm questioning my sanity, my (mis)understanding of things thus far. I'm still not sure where the ESC that breaks it comes from, is it from the highlight? like something similar to $ echo -e '\033[47mPASTED' to get white-ish background? I'm confused whether or not this is fixable by bash, or should I have to use that \012 terminator workaround? Please advise :) Thx.
Re: pasting in bracketed-paste mode breaks reverse-i-search
On 4/21/20 4:25 PM, gentoo_esh...@tutanota.com wrote: >> It's the ESC prefix. You can use it if you remove ESC from the list of >> characters that terminate an incremental search. The problem is that most >> of the key sequences that begin with ESC (e.g., arrow keys) are intended to >> terminate the isearch and use the ESC as the first character of a key >> sequence. You have to look farther ahead in the input stream than >> incremental search presently does to differentiate between an ESC-prefixed >> key sequence you want to use for searching and one you want to terminate >> the search and use as a command. >> > Did you mean to say that bracketed-mode pastes ESC, perhaps as a way to > highlight(ie. that white background) ? > interesting Not exactly. Bracketed paste mode is called that because it `brackets' the pasted text with special escape sequences. That's how you can recognize it as a paste -- otherwise it just shows up in the input stream as if the text had been typed. The escape sequences, like just about everything else with terminals, start with ESC. $ grep BRACK_P rlprivate.h #define BRACK_PASTE_PREF"\033[200~" #define BRACK_PASTE_SUFF"\033[201~" #define BRACK_PASTE_LAST'~' #define BRACK_PASTE_SLEN6 #define BRACK_PASTE_INIT"\033[?2004h" #define BRACK_PASTE_FINI"\033[?2004l\r" The last two are what you send when you want to start and stop using bracketed paste mode, respectively. The bracketed paste prefix and suffix serve only to delimit the text. They allow readline to recognize that a paste took place and mark the boundaries of the pasted text so the redisplay can highlight them later. > > static char * const default_isearch_terminators = "\033\012"; > > ESC and newline(ie. \n) hmm... > > oh hey, I found out how I know that Ctrl+R is the key that invokes > reverse-i-search: > $ bind -q reverse-search-history > reverse-search-history can be invoked via "\C-r". > > ok, so to translate what you said... the default is this: > $ bind "set isearch-terminators '\033\012'" > even though `bind -v | grep isearch-terminators` shows nothing before I run > that command because it's not set. You haven't set it, so it gets the default. > > So if I run this: > $ bind "set isearch-terminators '\012'" > then the issue is gone and reverse-i-search works, with the caveat that it > breaks when you try to rpess left/right arrows by it typing in "[D" > respectively "[C" , but it doesn't happen the first time right after paste, > only if I just do Ctrl+R type some string, press arrow... > I see that left/right arrow keys yield ESC[D and ESC[C respectively. So it > eats the ESC, it still terminates the i-search, then continues to type what's > left: "[D" or "[C". Confused as to why. I thought ESC shouldn't terminate > i-search It doesn't really. What happens is that it reads enough characters to resolve to a bound key sequence, finds the function (backward-char), and assumes that it should break the search and execute that function (this is standard incremental search behavior). The problem is that it doesn't save enough of the key sequence to do that -- it just pushes the current and previous characters back onto the input. I should do something about that sometime, but it really hasn't been a problem until now. > I'm still not sure where the ESC that breaks it comes from, is it from the > highlight? See above. > like something similar to $ echo -e '\033[47mPASTED' to get white-ish > background? Similar. > I'm confused whether or not this is fixable by bash, or should I have to use > that \012 terminator workaround? Please advise :) Thx. I'll have to add something to change this. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Proposed new feature for bash: unbuffered pipes, part 1: overview
It may seem intemperate to propose that a program with an 82-page manual "page" could benefit from yet another feature, but that is what this is doing: Bash's users would benefit from a feature that suppresses buffering of output sent to particular pipes (designated by the user). At the bottom of this message is a list of a dozen references spanning 20 years requesting a general mechanism to obtain this behavior. At least the following programs have options or default behaviors to suppress buffering stdout in situations where a generic program would buffer: python -u (or env. var. PHTHONUNBUFFERED) grep --line-buffered sed -u cat -u (unbuffered output by default in Gnu) tcpdump -l tee (unbuffered output by default) tail -f awk/mawk (fflush function or unbuffered output by default) The following commands have been devised to suppress buffering: unbuffer (probably == expect_unbuffer) stdbuf -oL script -q env. var. NSUnbufferedIO=YES (on MacOS) Each runs into one or more of these issues: does not work with statically-linked executables security issues involving env. var. LD_PRELOAD allocation of a pty combines stderr and stdout the effect does not apply to subprocesses of command, particularly commands within shell scripts, or applies to subprocesses whose output is not the pipe This message describes a proposed solution to this problem which is easier to invoke and works in a much broader range of circumstances than the previous solutions. The associated messages provide a proof-of-concept demonstration implementation of this proposal. This command demonstrates the issue: for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A | cat Currently, when this command is run, after 10 seconds it outputs 10 lines reading "ABCDE". That is because the "grep" command's stdout is to a pipe, and so is buffered. The buffer accumulates all 70 characters of output and outputs them to the pipe when "grep" exits. The demo proposed change defines: for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A >|> cat The "unbuffered pipe" symbol ">|>" causes Bash to set in the environment of the "grep" process a variable "STDOUT_UNBUFFERED" with a value that contains the dev and ino values for the pipe which the "grep" process sees as fd 1. The stdio library of the "grep" process, when it initializes processing of stdout (== fd 1), observes that the fd has the dev and ino values specified in the env. var. STDOUT_UNBUFFERED, and sets stdout's buffering state to unbuffered. (This can be overridden later if the process calls setvbuf.) The result is that the above command outputs one line "ABCDE" each second for 10 seconds. In addition to ">|>" for an unbuffered pipe, the demo also defines ">|>&", which is the unbuffered parallel of "|&", i.e., it is equivalent to "2>&1 >|>" and redirects both stdout and stderr from one command into an unbuffered pipe. There are a large number of open issues regarding this proposal, and in particular, the code only works for the simplest usages. But the first issue to be addressed is whether this functionality is worth pursuing. Dale -- References 14 Sep 1999 https://marc.info/?l=glibc-bug&m=98313957306295&w=4 "[REMINDER] stdio buffer flushing control environment variable" 26 May 2006 http://www.pixelbeat.org/programming/stdio_buffering/ "buffering in standard streams" 19 May 2009 https://stackoverflow.com/questions/881696/unbuffered-stdout-in-python-as-in-python-u-from-within-the-program "unbuffered stdout in python (as in python -u) from within the program [duplicate]" upvoted 57 times 16 Jun 2009 https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe "Turn off buffering in pipe" upvoted 415 times 20 Jan 2010 https://lists.gnu.org/archive/html/bug-bash/2010-01/msg00069.html "weird bash pipe behavior" 26 Jul 2010 https://stackoverflow.com/questions/3332045/bash-force-execd-process-to-have-unbuffered-stdout "bash: force exec'd process to have unbuffered stdout" upvoted 21 times 12 Aug 2010 https://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered "How to make output of any shell command unbuffered?" upvoted 66 times 23 Jun 2011 https://perkin.org.uk/posts/how-to-fix-stdio-buffering.html "How to fix stdio buffering" 26 Jul 2011 https://serverfault.com/questions/294218/is-there-a-way-to-redirect-output-to-a-file-without-buffering-on-unix-linux/589614#589614 "Is there a way to redirect output to a file without buffering on unix/linux?" upvoted 50 times 30 Nov 2012 https://stackoverflow.com/questions/13644024/stdbuf-with-setuid-capabilities/18624182 "stdbuf with setuid/capabilities" upvoted 3 times 2 Nov 2013 https://blog.jpalardy.com/posts/grep-and-output-buffering/ "Grep and Output Buffering" 31 Jan 2015 https://unix.stackexchange.com/questions/18
Proposed new feature for bash: unbuffered pipes, part 2: demo implementation
The baseline behavior is that this command, after 10 seconds, outputs 10 lines of output: $ bash -c 'for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | grep A | cat' To build and run a demo of the "unbuffered pipes" feature: Create and cd to a scratch directory. Build a modified Bash: $ wget http://ftp.wayne.edu/gnu/bash/bash-5.0.tar.gz $ tar -xf bash-5.0.tar.gz $ pushd bash-5.0 $ patch -p1 https://gnu.askapache.com/grep/grep-3.4.tar.xz $ tar -xf grep-3.4.tar.xz $ pushd grep-3.4 $ ./configure $ make $ # On Fedora 32, this is the command that "make" above used to create $ # the executable "grep", modified to statically link it with $LIBC. $ # I had to install the "pcre-static" and "glibc-static" packages to make $ # it work. $ pushd src $ gcc -static -g -O2 -o grep dfasearch.o grep.o kwsearch.o kwset.o searchutils.o pcresearch.o ../lib/libgreputils.a ../lib/libgreputils.a -lpcre -lpthread $LIBC $ export GREP=$( pwd )/grep $ ls -l $GREP $ file $GREP The desired behavior is that this command outputs one line of output each second for 10 seconds: $ $BASH -c 'for I in $( seq 10 ) ; do echo ABCDE ; sleep 1 ; done | $GREP A >|> cat'
Proposed new feature for bash: unbuffered pipes, part 3: bash.diff
diff --git a/command.h b/command.h
index 3249516..ef611a4 100644
--- a/command.h
+++ b/command.h
@@ -186,6 +186,7 @@ typedef struct element {
#define CMD_COPROC_SUBSHELL 0x1000
#define CMD_LASTPIPE 0x2000
#define CMD_STDPATH0x4000 /* use standard path for command lookup
*/
+#define CMD_STDOUT_UNBUFFERED 0x8000 /* Do not buffer stdout. */
/* What a command looks like. */
typedef struct command {
diff --git a/execute_cmd.c b/execute_cmd.c
index 8b3c83a..61d3647 100644
--- a/execute_cmd.c
+++ b/execute_cmd.c
@@ -567,6 +567,11 @@ execute_command_internal (command, asynchronous, pipe_in,
pipe_out,
volatile char *ofifo_list;
#endif
+#ifdef DRW
+ fprintf(stderr, "execute_command_internal %p %X\n", command, (command ?
command->flags : 0));
+ fprintf(stderr, "%s\n", make_command_string(command));
+#endif
+
if (breaking || continuing)
return (last_command_exit_value);
if (command == 0 || read_but_dont_execute)
@@ -836,6 +841,8 @@ execute_command_internal (command, asynchronous, pipe_in,
pipe_out,
command->value.Simple->flags |= CMD_IGNORE_RETURN;
if (command->flags & CMD_STDIN_REDIR)
command->value.Simple->flags |= CMD_STDIN_REDIR;
+ if (command->flags & CMD_STDOUT_UNBUFFERED)
+ command->value.Simple->flags |= CMD_STDOUT_UNBUFFERED;
line_number_for_err_trap = line_number = command->value.Simple->line;
exec_result =
@@ -2466,6 +2473,11 @@ execute_pipeline (command, asynchronous, pipe_in,
pipe_out, fds_to_close)
struct fd_bitmap *fd_bitmap;
pid_t lastpid;
+#ifdef DRW
+ fprintf(stderr, "execute_pipeline %p %X\n", command, command->flags);
+ fprintf(stderr, "%s\n", make_command_string(command));
+#endif
+
#if defined (JOB_CONTROL)
sigset_t set, oset;
BLOCK_CHILD (set, oset);
@@ -2477,7 +2489,9 @@ execute_pipeline (command, asynchronous, pipe_in,
pipe_out, fds_to_close)
cmd = command;
while (cmd && cmd->type == cm_connection &&
-cmd->value.Connection && cmd->value.Connection->connector == '|')
+cmd->value.Connection &&
+(cmd->value.Connection->connector == '|' ||
+ cmd->value.Connection->connector == GREATER_BAR_GREATER))
{
/* Make a pipeline between the two commands. */
if (pipe (fildes) < 0)
@@ -2551,6 +2565,10 @@ execute_pipeline (command, asynchronous, pipe_in,
pipe_out, fds_to_close)
dispose_fd_bitmap (fd_bitmap);
discard_unwind_frame ("pipe-file-descriptors");
+ if (cmd->flags & CMD_STDOUT_UNBUFFERED)
+ /* If cmd should not buffer stdout, then cmd...->second should
+ not buffer stdout. */
+ cmd->value.Connection->second->flags |= CMD_STDOUT_UNBUFFERED;
cmd = cmd->value.Connection->second;
}
@@ -2644,6 +2662,11 @@ execute_connection (command, asynchronous, pipe_in,
pipe_out, fds_to_close)
int ignore_return, exec_result, was_error_trap, invert;
volatile int save_line_number;
+#ifdef DRW
+ fprintf(stderr, "execute_connection %p %X\n", command, (command ?
command->flags : 0));
+ fprintf(stderr, "%s\n", make_command_string(command));
+#endif
+
ignore_return = (command->flags & CMD_IGNORE_RETURN) != 0;
switch (command->value.Connection->connector)
@@ -4158,6 +4181,16 @@ execute_simple_command (simple_command, pipe_in,
pipe_out, async, fds_to_close)
SHELL_VAR *func;
volatile int old_builtin, old_command_builtin;
+#ifdef DRW
+ fprintf(stderr, "execute_simple_command %p %X\n", simple_command,
(simple_command ? simple_command->flags : 0));
+ {
+WORD_LIST *w;
+for (w = simple_command->words; w; w = w->next)
+ fprintf(stderr, "%s ", w->word->word);
+fprintf(stderr, "\n");
+ }
+#endif
+
result = EXECUTION_SUCCESS;
special_builtin_failed = builtin_is_special = 0;
command_line = (char *)0;
@@ -5425,6 +5458,25 @@ execute_disk_command (words, redirects, command_line,
pipe_in, pipe_out,
exit (EXECUTION_FAILURE);
}
+#ifdef DRW
+ fprintf(stderr, "execute_disk_command %X '%s'\n", cmdflags,
command_line);
+#endif /* DRW */
+ if (cmdflags & CMD_STDOUT_UNBUFFERED)
+ {
+ /* Set the environment to request unbuffered stdout from the new
+program. */
+ struct stat buf;
+ char value[40];
+ fstat(1, &buf);
+ sprintf(value, "STDOUT_UNBUFFERED=%lu:%lu", (unsigned long)
buf.st_dev,
+ (unsigned long) buf.st_ino);
+ maybe_make_export_env ();
+ export_env = add_or_supercede_exported_var (value, 1);
+#ifdef DRW
+ fprintf(stderr, "execute_disk_command set '%s'\n", value);
+#endif /* DRW */
+ }
+
if (async)
interactive = old_interactive;
diff --git a/make_cmd.c b/make_cmd.c
index ecbbfd6..cd8e4c6 100644
--- a/make_cmd.c
+++ b/make_cmd.c
@@ -39,6 +39,7 @@
#include "parser.h"
#include "flags.h"
#include "input.h"
+#include "y.tab.h"
#if defined (JOB_CONTROL)
#include "jobs.h"
@@ -189,11 +190,19 @@ co
Proposed new feature for bash: unbuffered pipes, part 4: glibc.diff
diff --git a/libio/filedoalloc.c b/libio/filedoalloc.c
index 412b8d5937..35044e3512 100644
--- a/libio/filedoalloc.c
+++ b/libio/filedoalloc.c
@@ -58,6 +58,7 @@
#include "libioP.h"
#include
#include
+#include
#include
#include
@@ -71,6 +72,17 @@ local_isatty (int fd)
return res;
}
+struct _IO_stdout_unbuffered
+{
+ int initialized;
+ int present;
+ dev_t dev;
+ ino_t ino;
+} _IO_stdout_unbuffered_values;
+
+static void
+process_stdout_unbuffered (void);
+
/* Allocate a file buffer, or switch to unbuffered I/O. Streams for
TTY devices default to line buffered. */
int
@@ -80,9 +92,23 @@ _IO_file_doallocate (FILE *fp)
char *p;
struct stat64 st;
+ /* Check to see if buffering of stdout has been suppressed. */
+ if (! _IO_stdout_unbuffered_values.initialized)
+{
+ process_stdout_unbuffered();
+}
+
size = BUFSIZ;
if (fp->_fileno >= 0 && __builtin_expect (_IO_SYSSTAT (fp, &st), 0) >= 0)
{
+ // If this fd matched STDOUT_UNBUFFERED ...
+ if (_IO_stdout_unbuffered_values.present &&
+ _IO_stdout_unbuffered_values.dev == st.st_dev &&
+ _IO_stdout_unbuffered_values.ino == st.st_ino)
+ {
+ fp->_flags |= _IO_UNBUFFERED;
+ return 1;
+ }
if (S_ISCHR (st.st_mode))
{
/* Possibly a tty. */
@@ -105,3 +131,26 @@ _IO_file_doallocate (FILE *fp)
return 1;
}
libc_hidden_def (_IO_file_doallocate)
+
+/* Examine the STDOUT_UNBUFFERED environment variable, if any.
+ If it has the format :, parse those numbers into
+ _IP_stdout_unbuffered_values. */
+static void
+process_stdout_unbuffered (void)
+{
+ _IO_stdout_unbuffered_values.initialized = 1;
+ _IO_stdout_unbuffered_values.present = 0;
+ char *su_value = getenv("STDOUT_UNBUFFERED");
+ if (!su_value)
+return;
+ char *p1;
+ _IO_stdout_unbuffered_values.dev = strtoul(su_value, &p1, 10);
+ if (*p1 != ':')
+return;
+ char *p2;
+ _IO_stdout_unbuffered_values.ino = strtoul(p1+1, &p2, 10);
+ if (*p2)
+return;
+ // Enable the values that we have set.
+ _IO_stdout_unbuffered_values.present = 1;
+}
