Re: feature request: new builtin `defer`, scope delayed eval
> I don't think this would be a feature worth the time to implement and > the complexity it would introduce. Is there any other use case for this > than cleaning up temporary files on function return? Yeah! I think that's a fair point since it seems like we're landing on traps being the way forward. I think there may be some overlap already here with the RETURN trap, I'm not seeing a huge implementation burden yet, but I admittedly haven't played around with this code much. Is there a particular area you're thinking this will add a lot of weight to that RETURN doesn't add in some form? I think use cases outside of cleanup are relatively sparse, which is part of why originally I had proposed a builtin not adding to trap (something more purpose-focused). I'll think about it though. > This can be achieved by adding a new option to `trap' that causes it to > print the action string for given condition. Let this option be named > `-P', the user would do something like this to update a trap: Yep! This is very similar to what I had in mind, however in a slightly different form. With your syntax you still have to be aware of what's in that trap (whether it ends with a semicolon or not, and things of that nature). I'd really like to just be able to have a parameter like: `trap -P "foo" EXIT` > Exit trap is now: "foo" `trap -P "bar" EXIT` > Exit trap is now: "foo;bar" or whatever we end up using as a separator if it's not a semicolon. That way as a user writing my script I never have to be aware of what's already existing in a trap, I can always safely "add" my code, without stomping over anybody else. - Cynthia On Fri, Oct 7, 2022 at 11:49 PM Oğuz İsmail Uysal wrote: > > On 10/8/22 6:03 AM, Cynthia Coan wrote: > > Otherwise, I think we can perhaps reword this into two smaller > > features: "function local trap signals", and > I don't think this would be a feature worth the time to implement and > the complexity it would introduce. Is there any other use case for this > than cleaning up temporary files on function return? > > > "options to safely append/prepend to a trap" (for some definition of > > 'safe', perhaps this just means separating by ';'? Unsure if there's a > > better way off the top of my head). > This can be achieved by adding a new option to `trap' that causes it to > print the action string for given condition. Let this option be named > `-P', the user would do something like this to update a trap: > > trap "foo;" EXIT > trap "$(trap -P EXIT)bar;" EXIT
Re: feature request: new builtin `defer`, scope delayed eval
On Sat, Oct 8, 2022, at 2:11 AM, Koichi Murase wrote: > 2022年10月8日(土) 12:04 Cynthia Coan : >> [...] >> >> Otherwise, I think we can perhaps reword this into two smaller >> features: "function local trap signals", > > The existing RETURN trap is exactly the trap that can be used to clean > up resources local to functions and is already ``function-local'' > unless the user changes its behavior by setting `declare -ft', `set > -T', or `shopt -s extdebug'. One thing to note is that the RETURN trap > should be cleaned up inside the RETURN trap itself by running `trap - > RETURN', or otherwise the RETURN trap is erroneously considered by > Bash to be set for the caller function. The latter behavior is why I didn't describe RETURN as function-local in my earlier message, but I forgot the workaround of unsetting the trap from within itself. Nice. >> and "options to safely >> append/prepend to a trap" (for some definition of 'safe', perhaps this >> just means separating by ';'? > > Currently, there is no way to register multiple handlers to a trap in > a safe way, but if you can accept the approach by `;' or newline, you > can just do it by getting the current trap string with `trap -p > RETURN' and then overwrite the trap with the adjusted trap string. Another strategy is to construct the complete trap string before setting the trap, using whichever concatenation methods are preferred: f() { local trapcmds=cleanup if something; then trapcmds+='; more_cleanup' else trapcmds+='; so_fresh_and_so_clean' fi trap "$trapcmds; trap - RETURN" RETURN rest_of_function } -- vq
Re: feature request: new builtin `defer`, scope delayed eval
Hi Cynthia, 2022年10月8日(土) 15:56 Cynthia Coan : > I'm well aware that "defer"/"safe trap appending" can be implemented > today as it stands (see the original email in this chain, which links > to an example that I've used in a more reduced form). I did notice your link to the Gist example, but I have to say that it misuses the RETURN trap. As I have written in the previous reply, the RETURN trap is already designed to be function-local so that it suits your use cases. However, the Gist version somehow disables the function-local nature of the RETURN trap by running `set -o functrace' and instead tries to implement the function-local behavior by itself. > I also have in > the past written several helpers for safely appending to a trap I also have implemented a wrapper (and the generalized mechanism) of the trap in my project [1], but I admit that this is overcomplicated for the general advice in the Bash mailing list (the entire code has about 1000 lines). But if you are interested in a larger framework, you might be interested in it. [1] https://github.com/akinomyoga/ble.sh/blob/a144ffa807b40baa2b58f558cdd3b22bca44d38e/src/util.hook.sh#L42-L61 > (yours > in particular can mess up with certain quoting rules, but is nothing a > quick sed can't fix). Could you describe it in more detail or could you provide an example of how this can mess up? For example, the quoting of the trap string in the output of `trap -p' is processed by "eval" in the second line of the function body. I currently don't see any other parts where the quoting is relevant. > I don't see any particular criticisms about either `defer` as a > built-in, or adding functionality to `trap` outlined in particular. I actually don't see sufficient reasons to add a builtin command or add `append/prepend' options to trap. The suggested `builtin' can be implemented by a short shell function of about 10 lines. If we finally conclude that the suggested `defer' is useful for the general Bash scripting, we can just include it in /examples/functions in the Bash source tree. As for the `append/prepend' option, I think they would be very useful, but I think it requires more careful and comprehensive discussions. For example, it is non-trivial to find well-defined semantics for the multiple separate trap handlers containing the commands that affect the control flow outside the trap (such as continue, return, break, etc.). It might also conflict with the existing idioms of trap/signal handlers that assume the single trap handler such as `trap - INT && kill -INT "$BASHPID"' for SIGINT. How to choose a single exit status of the DEBUG trap which would be used to determine the next action by Bash? etc. On the other hand, I think `trap -P' suggested by Oğuz can be safely added without conflicting existing behaviors, but this is more general and can be considered separately from the defer implementation. > In fact some of the things you mentioned you can do today, are also > outlined in the original email as things that can be "harder than they > need to be" (...). I agree the proper implementation of `defer' by the RETURN trap could be tricky (as described in "notes" in my previous reply), but it's definitely not as hard as you initially posted (one 12-line function vs 244-line defer.sh). -- Koichi
Re: feature request: new builtin `defer`, scope delayed eval
Let me update the example implementation of ``defer''. I think this is more readable: function defer { local trap eval "trap=($(trap -p RETURN))" trap -- " if [[ \$FUNCNAME != defer ]]; then trap - RETURN $1 ${trap[2]-} fi" RETURN } declare -ft defer I also tested it with an example to demonstrate the behavior: function f1 { echo allocA defer 'echo freeA' echo allocB defer 'echo freeB' f2 f2 } function f2 { echo f2 echo allocC defer 'echo freeC' echo allocD defer 'echo freeD' } f1 -- result -- allocA allocB f2 allocC allocD freeD freeC f2 allocC allocD freeD freeC freeB freeA -- Koichi
Re: feature request: new builtin `defer`, scope delayed eval
Hi Lawrence, 2022年10月8日(土) 16:04 Lawrence Velázquez : > On Sat, Oct 8, 2022, at 2:11 AM, Koichi Murase wrote: > > [...]. One thing to note is that the RETURN trap > > should be cleaned up inside the RETURN trap itself by running `trap - > > RETURN', or otherwise the RETURN trap is erroneously considered by > > Bash to be set for the caller function. > > The latter behavior is why I didn't describe RETURN as function-local > in my earlier message, but I forgot the workaround of unsetting > the trap from within itself. Nice. Yeah, that is the tricky point. > Another strategy is to construct the complete trap string before > setting the trap, using whichever concatenation methods are preferred: I think the original idea of ``defer'', RAII, or similar features in other programming languages is to avoid that (i.e., requiring cleanup codes in a separate position of the code). -- Koichi
Re: feature request: new builtin `defer`, scope delayed eval
Hi Koichi, > I did notice your link to the Gist example, but I have to say that it > misuses the RETURN trap. As I have written in the previous reply, the > RETURN trap is already designed to be function-local so that it suits > your use cases. However, the Gist version somehow disables the > function-local nature of the RETURN trap by running `set -o functrace' > and instead tries to implement the function-local behavior by itself. > I also have implemented a wrapper (and the generalized mechanism) of > the trap in my project [1], but I admit that this is overcomplicated > for the general advice in the Bash mailing list (the entire code has > about 1000 lines). But if you are interested in a larger framework, > you might be interested in it. > > [1] > https://github.com/akinomyoga/ble.sh/blob/a144ffa807b40baa2b58f558cdd3b22bca44d38e/src/util.hook.sh#L42-L61 > I agree the proper implementation of `defer' by the RETURN trap could > be tricky (as described in "notes" in my previous reply), but it's > definitely not as hard as you initially posted (one 12-line function > vs 244-line defer.sh). This is definitely where the PoC nature comes in. I agree there are many different ways of implementing this. To point out where the lines come from, out of those "224" lines, removing all the comments (and extra functionality that is included in the same file) gets us down to < 60 lines, and the reason it implements the "function local" state itself is because for quickness of a PoC it was copied from an "error mode" example which does need to implement function local checks itself (with functrace), as if the function "sets error mode", and than calls another function which changes the error mode not through a wrapper you'd be in the wrong state for the rest of the function. So you always need to check where you are. That being said, I'm sure those PoC's could be tuned much better and be created in much smaller line amounts! I don't think we need to spam the mailing list with potential implementations of either of these though. I'd prefer we don't focus on any particular implementations, or how many lines they end up being. After all, again what is a local variable if not a variable with a deferred unset/set to old value? That could be just as easily implemented, yet we still found its utility, and brevity worthy enough of adding to bash. I'm not saying defer necessarily meets the same bar as usefulness, but I think it'd be wrong to dismiss new features just because they can be accomplished today, with relatively few lines. If we find something lots of people are doing, it might be worth discussing adding it in (and again: I'm totally willing to admit defer/defer-like might not meet that bar)! > Could you describe it in more detail or could you provide an example > of how this can mess up? For example, the quoting of the trap string > in the output of `trap -p' is processed by "eval" in the second line > of the function body. I currently don't see any other parts where the > quoting is relevant. Yep that's on me, I totally missed the eval. Apologies! > I actually don't see sufficient reasons to add a builtin command or > add `append/prepend' options to trap. > The suggested `builtin' can be implemented by a short shell function > of about 10 lines. If we finally conclude that the suggested `defer' > is useful for the general Bash scripting, we can just include it in > /examples/functions in the Bash source tree. Interesting, is this purely about lines of code? I don't think we'll agree on that point, but I'm curious if you think this is something that is used a lot vs. just easy enough in terms of lines. I'm certainly not opposed to adding things to bash examples, if they're not generally usable! > As for the `append/prepend' option, I think they would be very useful, > but I think it requires more careful and comprehensive discussions. > For example, it is non-trivial to find well-defined semantics for the > multiple separate trap handlers containing the commands that affect > the control flow outside the trap (such as continue, return, break, > etc.). It might also conflict with the existing idioms of trap/signal > handlers that assume the single trap handler such as `trap - INT && > kill -INT "$BASHPID"' for SIGINT. How to choose a single exit status > of the DEBUG trap which would be used to determine the next action by > Bash? etc. > > On the other hand, I think `trap -P' suggested by Oğuz can be safely > added without conflicting existing behaviors, but this is more general > and can be considered separately from the defer implementation. I'm certainly hopeful to have those discussions, I opened this wanting to get feedback/drive discussion, and if it requires more discussion I'd love to do that. I think either way in general the mailing list seems to agree that "defer" as a built-in probably not the way to go, so I think we can drop that mostly (or just add it to the examples like you mentioned)! On t
Re: feature request: new builtin `defer`, scope delayed eval
8 Ekim 2022 Cumartesi tarihinde Cynthia Coan yazdı: > > [...] I think > use cases outside of cleanup are relatively sparse, [...] There. There already are several ways to do cleanup on exit/return using existing features, why add one more? > [...] With your syntax you still have to be aware of what's > in that trap [...] > Not really. If, for some reason, I didn't know what's in that trap and were super paranoiac about it, I could do oldaction=$(trap -P EXIT) trap 'eval "$oldaction"; bar' EXIT I've never been in such a situation though. Don't get me wrong, I just like it when new features align with the old ones and work together nicely. -- Oğuz
Re: feature request: new builtin `defer`, scope delayed eval
Hi Cynthia, thank you for your reply. 2022年10月8日(土) 17:26 Cynthia Coan : > > I agree the proper implementation of `defer' by the RETURN trap could > > be tricky (as described in "notes" in my previous reply), but it's > > definitely not as hard as you initially posted (one 12-line function > > vs 244-line defer.sh). > > [...] the reason it implements the "function local" state > itself is because for quickness of a PoC it was copied from an "error > mode" example (This is just a side note about `local -' but unrelated to the main discussion): I didn't mention it because I thought `set -e' is merely an example to demonstrate the behavior of `defer', but if the handling of `set -e' is one of the main motivations for requesting `defer', I suggest trying Bash 4.4's `local -', which localizes the shell-option changes within the function. If the function doesn't want to change the shell option of the caller, it can just declare `local -' before changing any shell options. [ It is unavailable in Bash 4.3 or below, but the suggested `defer' builtin would neither be available in older versions of Bash, so we can forget about the support of `local -' of older Bash in the discussion of the new feature. ] > which does need to implement function local checks > itself (with functrace), as if the function "sets error mode", and > than calls another function which changes the error mode not through a > wrapper you'd be in the wrong state for the rest of the function. So > you always need to check where you are. I think you might be caring about the cases the callee changes the error mode without restoring it, but I would say that restoring the error mode to the original is the responsibility of the callee function, but it's not what the caller or the entire execution system of Bash should take care. Also, how to distinguish them from the functions that want to intensionally change the error mode of outer/global scope (such as functions similar to `__{push,pop}_error_stack' in other codebases)? Here, I still think we do not need to re-implement the function-local traps other than the already built-in one even for the `set -e' handling. > I'd prefer we don't focus on any particular implementations, or how > many lines they end up being. Obviously, the number of lines itself is not important. I mentioned the number of lines because you seemed to present the idea that "the implementation can be harder than they need to be" mentioning the original email [2]. [2] https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00036.html >>> In fact some of the things you mentioned you can do today, are also >>> outlined in the original email as things that can be "harder than they >>> need to be" But I don't think it is hard to implement it in a shell script as said in the original email, so I just used the number of lines as an ``approximate'' measure of how hard the code is. If both codes had similar numbers of lines, it wouldn't correctly compare the actual complexity of the codes as you think, but I think the number of about 10 lines is clear to show that the implementation is not that hard (as far as one understands the usage of the RETURN trap) compared to what the original email seemed to convince us with the link to 244 lines of code (regardless of the intention). > I'm not saying > defer necessarily meets the same bar as usefulness, but I think it'd > be wrong to dismiss new features just because they can be accomplished > today, with relatively few lines. In my point of view, the suggested "new" feature is *already implemented in Bash as the RETURN trap*, and there is no reason to add the same feature twice. These "relatively few lines" (including `declare -ft' and the FUNCNAME check) are just needed to adjust the usage to your taste as the wrapper `defer'. If we do not use `defer', we could naturally write it like eval "original=($(trap -p RETURN))" trap "trap - RETURN;;${original[2]}" RETURN If `trap -P' suggested by Oğuz would be supported, it can even be simplified as trap "trap - RETURN;;$(trap -P RETURN)" RETURN The remaining complication is actually caused by the lack of support for the multiple trap handlers and is unrelated to `defer'. Of course, if one wants to use the above idiom multiple times, it may not be useful to write it every time. In that case, one can define a function. If it needed to implement the essential parts of the processing as in defer.sh, it might have been better implemented as a built-in feature, but I don't think the wrapper of the above one or two lines should be a separate builtin command. A shell function is sufficient. > Interesting, is this purely about lines of code? Of course, no! See the above response about the number of lines. > On the traps append/prepend point I will have to > do some thinking on that, I'll take the weekend/beginning of the week > to think of what I can, but if someone else has ideas I would love to > hear those too. I think this is a mor
Re: [PATCH] Fix foreground dead jobs in trap handlers reported like background ones in `jobs'
Thank you for your reply and sorry for the late reply. 2022年10月4日(火) 0:56 Chet Ramey : > > I expect the same behavior of `f1' and `f2' as far as there are no > > background jobs. > > Why? f2 calls `jobs', and so requests information in a particular format, > which may or may not be the same format as the default (compact) format > bash uses when it reports job status before printing a prompt. My point is *not* about the format of the output of `jobs', but whether `jobs' should print the entries of foreground dead jobs, to begin with. I think this is related to why the job entries of the foreground jobs are preserved in special contexts such as trap handler, bind -x, PROMPT_COMMAND, etc. (see the discussion later). > Here's how it works. ... I understand all of these because I read the code to create the patches. > When the shell is interactive, you get notified about > dead jobs in two places: before the shell prints a new prompt, and after > the shell waits for a foreground process to exit and discovers that a > job has completed. You'd think the first would be enough, No. I know that that is needed. To be honest, I actually thought that in the opposite way as ``We would primarily check the jobs at the end of a foreground process, but it would be suppressed in the special contexts of trap/bind/PROMPT_COMMAND'.' > but it leaves you > vulnerable to job table explosion when you're running a bunch of jobs in a > loop. The pragmatic solution of notifying when the job completes works well > enough to avoid this. Bash has done this for decades. > > This default compact format prints a message that strsignal() returns, > since it assumes you can easily associate it with the just-completed job. > > So you can see what happens. When run in the foreground, f1 prints the > status from strsignal() after waiting for each process to finish. When > run as a trap handler, f1 defers printing until it's about to print a > prompt, then prints the status from strsignal(). When run in the > foreground, f2 prints the status after waiting for each process to finish > (which means that when you run `jobs' there are no un-notified processes). > When run as a trap handler, f2 tries to defer printing the status until > the trap handler completes, but the function forces status printing with > `jobs' in a specific format, which means there are no un-notified jobs > when the time comes to print the next prompt. These also seem to just describe what is done in the current codebase, which I think I understand. > I think this is reasonable behavior. I wouldn't say the current behavior is unreasonable or inconsistent, which is the reason that I write the suggested behavior to be "more" consistent at the end of my previous reply. As I have written above, I think it is actually related to the reason that the foreground jobs are preserved in the special contexts in trap/bind -x/PROMPT_COMMAND/etc. I just thought that these entries are just kept to defer printing the status, which would have occurred immediately on the termination of foreground jobs if it were outside the special contexts. In this sense, I thought the entries of the foreground dead jobs are just an internal means to make the printing defer, and not the real/practical job entries that are exposed to users. In fact, now in the devel branch, we only preserve the foreground dead jobs only when they need to be notified later (i.e., when they are killed by the signals). If it is just a means to defer the printing to keep the job entries of foreground dead jobs, I think it is "more" consistent that they should only affect the deferred printing but not affect the builtin command `jobs' used by the users to read and handle the job information. Or is there use cases of the job entries of killed foreground dead jobs in the output of `jobs' for the users? Actually, this is related to the usage of the builtin command `jobs' in my code. I would like to collect the information about the currently-running and terminated background jobs using the `jobs' builtin (by doing something like « jobs > tmp; read < tmp »). I wouldn't like to pick up the foreground dead jobs by the `jobs' builtin, but they are indistinguishable from the background dead jobs and ``contaminate'' the result of `jobs'. This is the problem that we were faced with in Ref. [1]. [1] https://github.com/akinomyoga/ble.sh/issues/78#issue-771281403 Although the foreground jobs that terminated without signals now do not appear in the output of `jobs' now in the devel, the killed foreground jobs still contaminate the output of `jobs'. As a workaround, currently, my code discards all the information of jobs that were terminated while evaluating PROMPT_COMMAND regardless of whether they were foreground or background, but I still would like to keep the information of the terminated background jobs if possible. -- Koichi
Re: feature request: new builtin `defer`, scope delayed eval
Hi Koichi, > (This is just a side note about `local -' but unrelated to the main > discussion): I didn't mention it because I thought `set -e' is merely > an example to demonstrate the behavior of `defer', but if the handling > of `set -e' is one of the main motivations for requesting `defer', I > suggest trying Bash 4.4's `local -', which localizes the shell-option > changes within the function. If the function doesn't want to change > the shell option of the caller, it can just declare `local -' before > changing any shell options. [ It is unavailable in Bash 4.3 or below, > but the suggested `defer' builtin would neither be available in older > versions of Bash, so we can forget about the support of `local -' of > older Bash in the discussion of the new feature. ] I appreciate this tip, it is one I was aware of, but I appreciate the callout regardless. I didn't mean to make the error handling mode a "key point" for adopting defer, it was just a good stand in for "global state manipulation" in general. I apologize for my wording errors there, making it seem like a "main motivation" to you. > > which does need to implement function local checks > > itself (with functrace), as if the function "sets error mode", and > > than calls another function which changes the error mode not through a > > wrapper you'd be in the wrong state for the rest of the function. So > > you always need to check where you are. > > I think you might be caring about the cases the callee changes the > error mode without restoring it, but I would say that restoring the > error mode to the original is the responsibility of the callee > function, but it's not what the caller or the entire execution system > of Bash should take care. Also, how to distinguish them from the > functions that want to intensionally change the error mode of > outer/global scope (such as functions similar to > `__{push,pop}_error_stack' in other codebases)? Here, I still think we > do not need to re-implement the function-local traps other than the > already built-in one even for the `set -e' handling. As much as I agree with you here, unfortunately not all shell scripts do what they _should_. From this particular codebase example it was meant to help callers "not worry about the details", as they did have to deal with scripts not always doing what they should. > > I'd prefer we don't focus on any particular implementations, or how > > many lines they end up being. > > Obviously, the number of lines itself is not important. I mentioned > the number of lines because you seemed to present the idea that "the > implementation can be harder than they need to be" mentioning the > original email [2]. > > [2] https://lists.gnu.org/archive/html/bug-bash/2022-10/msg00036.html > But I don't think it is hard to implement it in a shell script as said > in the original email, so I just used the number of lines as an > ``approximate'' measure of how hard the code is. If both codes had > similar numbers of lines, it wouldn't correctly compare the actual > complexity of the codes as you think, but I think the number of about > 10 lines is clear to show that the implementation is not that hard (as > far as one understands the usage of the RETURN trap) compared to what > the original email seemed to convince us with the link to 244 lines of > code (regardless of the intention). > > Interesting, is this purely about lines of code? > > Of course, no! See the above response about the number of lines. I'm sorry for my poor word choices here, I adopted your idioms of using "lines" in place of "complexity", but I like you meant 'complexity'. I also did not mean to imply that defer was in any way hard to implement, in fact at multiple points in this chain I've admitted how easy implementing defer, or appending to a trap could be! I didn't mean to create any sort of impression that creating a defer would be hard, long, or tedious. In fact, much the opposite! The original email at the very very beginning of this thread, I tried to word quite carefully to show how defer could be a more elegant simple solution, to other solutions which can remain at times tedious. Because it is a simple feature that might be generally useful we should just give it to folks. Rather than having it written in a bunch of different places, making it a hopefully fairly easy change to adopt. I truly apologize if I implied in any form that creating "defer" would be complex, or difficult. I am in complete agreement with you on this point. However, I would also like to reiterate how I think "how un-complex something can be" should not be a blocker for inclusion. We might miss something like the next `local var=` change, after all bash is quite powerful which is why it is used. It should be our goal to allow making bash scripts easier to write correctly overall, even if some features may be simple if they're generally useful saving lots of folks from each having their own copy of a thing, or starting a proj
Re: declare -F incorrect line number
On 10/2/22 4:51 AM, Daniel Castro wrote: Bash Version: 5.0 Patch Level: 17 Release Status: release Description: declare -F yields the wrong line number for a function that has nested functions declared within. Instead it gives the line number of the last nested function. This is a consequence of using a bison-generated parser, which basically makes you do things `bottom-up': you know you have a complete function definition only when you have a complete function definition and the production reduces. The lexer keeps track of what looks like the start of a function definition, and if the parser ends up reducing the function def rule, we can use it to set the line number the function starts on. It's pretty easy to put something in to save the last value when you set the start of a function definition, allowing us to save the line number for one level of function nesting. Anything more than that requires some sort of stack, and I'm not sure it's worth the implementation cost. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/