Re: Parallelization of shell scripts for 'configure' etc.

2022-07-08 Thread Alex Ameen
I've been telling folks about the config site file every time this thread comes up. Good on you for actually trying it haha. It can make a huge difference. You can short circuit a lot of checks this way. Now, the disclaimer: you still shouldn't share a cache file between projects, and if you use

Re: Parallelization of shell scripts for 'configure' etc.

2022-07-08 Thread Simon Josefsson via Bug reports for the GNU Bourne Again SHell
Tim Rühsen writes: > a) The maintainer/contributor/hacker setup > This is when you re-run configure relatively often for the same project(s). > I do this normally and and came up with > https://gitlab.com/gnuwget/wget2/-/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain. > > It may be a

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-21 Thread Alexandre Oliva
On Jun 13, 2022, Paul Eggert wrote: > In many Gnu projects, the 'configure' script is the biggest barrier to > building because it takes s long to run. Is there some way that we > could improve its performance without completely reengineering it, by > improving Bash so that it can parallelize

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-20 Thread Chet Ramey
On 6/18/22 3:05 PM, Tim Rühsen wrote: [Configure for wget2 runs 'rm' and 'cat' each roughly 2000x - so I came up with enabling plugins for those two commands (had to write a plugin for 'rm', not sure if it never has been accepted by bash upstream).] The loadable `rm' has been in bash since ba

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-18 Thread Tim Rühsen
Hi all, On 14.06.22 00:39, Paul Eggert wrote: In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes s long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parall

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-18 Thread L A Walsh
On 2022/06/13 15:39, Paul Eggert wrote: In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes s long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parallelize '

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-15 Thread Bruno Haible
Paul Eggert wrote: > It appears that PaSh itself isn't suitable for prime-time use. That's my current conclusion too: I tried running PaSh on - the configure script of a coreutils release, - gnulib-tool (a large shell script), and got a parse error in both cases [1]. Bruno [1] https://github

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-14 Thread Jeffrey Walton
On Tue, Jun 14, 2022 at 11:15 PM Ángel wrote: > > On 2022-06-13 at 18:32 -0700, Paul Eggert wrote: > > Yes, all that could be done in theory, but it'd take a lot of > > hacking and it's been decades and it hasn't happened. > > > > I'd rather have shell scripts "just work" in parallel with a minimu

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-14 Thread Paul Eggert
On 6/14/22 18:55, Ángel wrote: Do you have any handy example of configure that takes too long to run? Sure. Coreutils, emacs. Pretty much any nontrivial configure script takes longer than it should. I understand that parallelization of shell scripts is nontrivial.

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-14 Thread Paul Eggert
On 6/14/22 10:11, Nick Bowler wrote: The resulting config.h is correct but pa.sh took almost 1 minute to run the configure script, about ten times longer than dash takes to run the same script. More than half of that time appears to be spent just loading the program into pa.sh, before a single

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-14 Thread Ángel
On 2022-06-13 at 18:32 -0700, Paul Eggert wrote: > Yes, all that could be done in theory, but it'd take a lot of > hacking and it's been decades and it hasn't happened. > > I'd rather have shell scripts "just work" in parallel with a minimum > of fuss. If this hasn't happened, that might be becau

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-14 Thread Steffen Nurpmeso
Chet Ramey wrote in <211b74c0-caed-227f-ffae-b85868ef7...@case.edu>: |On 6/13/22 6:39 PM, Paul Eggert wrote: |> In many Gnu projects, the 'configure' script is the biggest barrier to |> building because it takes s long to run. Is there some way that we |> could improve its performance with

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-14 Thread Chet Ramey
On 6/13/22 6:39 PM, Paul Eggert wrote: > In many Gnu projects, the 'configure' script is the biggest barrier to > building because it takes s long to run. Is there some way that we > could improve its performance without completely reengineering it, by > improving Bash so that it can paralleliz

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-13 Thread Alex Ameen
You can try to use the `requires` toposort routine to identify "Strongly Connected Sub-Components", which is where I imagine you'll get the best results. What you'll need to watch out for is undeclared ordering requirements that parallelism would break. The `m4sh` and `m4sugar` source code is docu

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-13 Thread Alex Ameen
Yeah honestly splitting most of the `configure` checks into multiple threads is definitely possible. Caching between projects is even a straightforward extension with systems like `Nix`. The "gotcha" here in both cases is that existing scripts that are living in source tarballs are not feasible t

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-13 Thread Paul Eggert
On 6/13/22 18:25, Dale R. Worley wrote: It seems to me that bash provides the needed tools -- "( ... ) &" lets you run things in parallel. Similarly, if you've got a lot of small tasks with a complex web of dependencies, you can encode that in a "makefile". It seems to me that the heavy work is

Re: Parallelization of shell scripts for 'configure' etc.

2022-06-13 Thread Dale R. Worley
Paul Eggert writes: > In many Gnu projects, the 'configure' script is the biggest barrier to > building because it takes s long to run. Is there some way that we > could improve its performance without completely reengineering it, by > improving Bash so that it can parallelize 'configure' s

Parallelization of shell scripts for 'configure' etc.

2022-06-13 Thread Paul Eggert
In many Gnu projects, the 'configure' script is the biggest barrier to building because it takes s long to run. Is there some way that we could improve its performance without completely reengineering it, by improving Bash so that it can parallelize 'configure' scripts? For ideas about thi