Re: Proposed new feature for bash: unbuffered pipes, part 1: overview

2020-04-22 Thread Greg Wooledge
On Tue, Apr 21, 2020 at 08:38:41PM -0400, Dale R. Worley wrote:
> The "unbuffered pipe" symbol ">|>" causes Bash to set in the
> environment of the "grep" process a variable "STDOUT_UNBUFFERED" with
> a value that contains the dev and ino values for the pipe which the
> "grep" process sees as fd 1.

Which libc implements this?

> References
> 
> 14 Sep 1999
> https://marc.info/?l=glibc-bug&m=98313957306295&w=4
> "[REMINDER] stdio buffer flushing control environment variable"

The next message in the thread is from Ulrich Drepper, saying:

  I will not implement this since it's completely up to the application
  to do this.  One knows in advance when this is necessary.  If there is
  a problem with the current code to select when line-buffering is used
  (we use as everybody else isatty) then one can talk about this.  I
  think it is correct, though.



PATCH [1/1]: compressedand - escape single ampersand found in URLs

2020-04-22 Thread Gonzalez, Eric (UMKC-Student)
I've written a feature I find useful for bash that escapes a single ampersand 
if it is surrounded on both sides by any non-blank or non-meta characters. This 
logic is toggled by a set option, "compressedand" that can be enabled with `set 
[-|+]c` or `set -o compressedand`.


As an example:

   $ echo this&echo that
   [1] 32324
   that
   this
   [1]+  Doneecho this

   $ set -c
   $ echo this&echo that
   this&echo that

Handling of multi-character metas, such as &>, |&, &&, et al, is still parsed 
by BASH and their functions remain unaffected.

   $ echo this& echo that
   [1] 1373
   that
   this
   [1]+  Doneecho this

   $ echo this &echo that
   [1] 1375
   that
   this
   [1]+  Doneecho this

   $ echo this&&echo that
   this
   that

   $ ps ax&>/tmp/ps&echo
   $ ls /tmp/ps\&echo
   /tmp/ps&echo


The primary use case for the option is to ease using unquoted URLs as arguments 
that have multiple query parameters. Yes, quoting the URL precludes the need 
for this feature, but I forget sometimes (okay, too often):

   $ set +c
   $ echo https://example.com/inputs?v=22&time=now&cp=key
   [1] 32335
   [2] 32336
   https://example.com/inputs?v=22
   [1]-  Doneecho https://example.com/inputs?v=22
   [2]+  Donetime=now

   $ set -c
   $ echo https://example.com/inputs?v=22&time=now&cp=key
   https://example.com/inputs?v=22&time=now&cp=key


I've attached a patch that was written against bash-4.4 and tested to work 
against the most recent DEVEL commit (3235014e5b) as of 04/22. The patch 
provides the logic, set option control, and updates to the man and info pages. 
I'm still working on getting some tests worked up. I wanted to get this 
submitted to get a feel for general likeability and for input on things I need 
to do for a proper contribution.

And, for completeness, here are the bash VERSIONs I've tested this on:

   GNU bash, version 4.4.23(1)-release (x86_64-pc-linux-gnu)
   GNU bash, version 5.0.16(1)-maint (x86_64-pc-linux-gnu)

I haven't looked up to see what POSIX has to say about this, but I'm fairly 
confident that this flies in the face of it. If that's the case and this patch 
is dismissed accordingly, I'll accept that. This was a fun exercise anyway.  .)

- Eric.

---
Fudd's First Law of Opposition:
"If you push something hard enough, it will fall over."
  - Firesign Theater

diff --git a/builtins/set.def b/builtins/set.def
index 8122361..2467e73 100644
--- a/builtins/set.def
+++ b/builtins/set.def
@@ -59,7 +59,7 @@ extern int no_line_editing;
 
 $BUILTIN set
 $FUNCTION set_builtin
-$SHORT_DOC set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
+$SHORT_DOC set [-abcefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
 Set or unset values of shell options and positional parameters.
 
 Change the value of shell attributes and positional parameters, or
@@ -68,6 +68,7 @@ display the names and values of shell variables.
 Options:
   -a  Mark variables which are modified or created for export.
   -b  Notify of job termination immediately.
+  -c  Escape a single ampersand surrounded by non-blank/non-meta text.
   -e  Exit immediately if a command exits with a non-zero status.
   -f  Disable file name generation (globbing).
   -h  Remember the location of commands as they are looked up.
@@ -77,50 +78,51 @@ Options:
   -n  Read commands but do not execute them.
   -o option-name
   Set the variable corresponding to option-name:
-  allexportsame as -a
-  braceexpand  same as -B
+  allexport  same as -a
+  braceexpandsame as -B
+  compressedand  same as -c
 #if defined (READLINE)
-  emacsuse an emacs-style line editing interface
+  emacs  use an emacs-style line editing interface
 #endif /* READLINE */
-  errexit  same as -e
-  errtrace same as -E
-  functracesame as -T
-  hashall  same as -h
+  errexitsame as -e
+  errtrace   same as -E
+  functrace  same as -T
+  hashallsame as -h
 #if defined (BANG_HISTORY)
-  histexpand   same as -H
+  histexpand same as -H
 #endif /* BANG_HISTORY */
 #if defined (HISTORY)
-  history  enable command history
+  historyenable command history
 #endif
-  ignoreeofthe shell will not exit upon reading EOF
+  ignoreeof  the shell will not exit upon reading EOF
   interactive-comments
-   allow comments to appear in interactive commands
-  keyword  same as -k
+ allow comments to appear in interactive commands
+  keywordsame as -k
 #if defined (JOB_CONTROL)
-  monitor  same as -m
+  monitorsame as -m
 #endif
-  noclobbersame as -C
-  noexec   sam

nameref pointing to array element

2020-04-22 Thread Daniel Molina
Hello,

Is the following an intended behavior? Undefined? A bug?

$ a[1]=2; declare -n ref=a[1]; echo $ref $((ref))
2 0

Thanks,

Daniel




Re: Proposed new feature for bash: unbuffered pipes, part 1: overview

2020-04-22 Thread Dale R. Worley
Greg Wooledge  writes:
> On Tue, Apr 21, 2020 at 08:38:41PM -0400, Dale R. Worley wrote:
>> The "unbuffered pipe" symbol ">|>" causes Bash to set in the
>> environment of the "grep" process a variable "STDOUT_UNBUFFERED" with
>> a value that contains the dev and ino values for the pipe which the
>> "grep" process sees as fd 1.
>
> Which libc implements this?

None does yet, obviously.  That's why I provided a patch for it.

>> References
>> 
>> 14 Sep 1999
>> https://marc.info/?l=glibc-bug&m=98313957306295&w=4
>> "[REMINDER] stdio buffer flushing control environment variable"
>
> The next message in the thread is from Ulrich Drepper, saying:
>
>   I will not implement this since it's completely up to the application
>   to do this.  One knows in advance when this is necessary.  If there is
>   a problem with the current code to select when line-buffering is used
>   (we use as everybody else isatty) then one can talk about this.  I
>   think it is correct, though.

That's true, but I don't hink he's looking at it properly.  It was also
from 20 years ago.

Dale



Re: Proposed new feature for bash: unbuffered pipes

2020-04-22 Thread Dale R. Worley
The crux of the problem, IMHO, is to look at it from the right angle:

Occasionally, the user desires that I/O through certain pipes should be
unbuffered, that is, the stdio stream(s) that write into the pipe should
be unbuffered, rather than the default block-buffered.  These are
situations where the user can tell that it is desirable to sacrifice
efficiency to gain low latency, whereas the writer of the program that
the user is running could not predict the needs of this particular
invocation.  (This is where Ulrich Drepper was incorrect; the writer of
the program doesn't know when this is desirable.)

This facility is best activated by a feature in bash, as the need arises
from the relationship between two processes.

The ideal way to implement this would be for the kernel to provide two
flavors of pipe, one designated buffered and one designated unbuffered,
and according to the user's instructions, bash would allocate an
unbuffered pipe when it was needed.  The kernel itself would treat the
two types of pipe in the same way, but stdio would create buffered
streams for fd's that were opened on buffered pipes and it would create
unbuffered streams for fd's that were opened on unbuffered pipes.

However, getting a coordinated change into the kernel seems like it
would be quite difficult.

So an alternative implementation is to have a way for bash to notify
processes it creates that certain pipes (identified by their dev/ino
numbers) should have streams that are unbuffered.  Pretty much the only
way to do this is to have a conventional environment variable for
communicating this information.

Of course, either of these solutions requires changes to glibc, but
then, if you're going to modify buffering behavior of streams, you are
going to have to modify glibc.

Dale