Hi!
On Wed, 2024-12-04 at 14:37:45 +0000, Stefano Rivera wrote:
> Hi Guillem (2024.12.04_13:03:29_+0000)
> > > Are there other layers that could reasonably be used to implement a more
> > > general form of parallelism limiting based on system RAM? Ideally, we'd
> > > consolidate these implementations into fewer places.
> >
> > I think adding this in dpkg-buildpackage itself would make most sense
> > to me, where it is already deciding what amount of parallelism to use
> > when specifying «auto» for example.
> >
> > Given that this would be and outside-in interface, I think this would
> > imply declaring these parameters say as debian/control fields for example,
> > or some other file to be parsed from the source tree.
>
> I don't think this can be entirely outside-in, the package needs to say
> how much ram it needs per-core, to be able to calculate the appropriate
> degree of parallelism. So, we have to declare a value that then gets
> calculated against the proposed parallelism.
I _think_ we are saying the same, and there might just be a mismatch
in nomenclature (most probably stemming from me being non-native and
using/reusing terms incorrectly)? So let me clarify what I meant,
otherwise I might be misunderstanding your comment, and I'd appreciate
a clarification. :)
When dealing with dpkg packaging build interfaces, in my mind there are
two main models:
* outside-in: where the build driver (dpkg-buildpackage in this case)
can reach for all needed information and then do stuff based on that,
or pass that information down into debian/rules process hierarchy,
or to tools it invokes itself (say dpkg-genchanges); another such
interface could be R³ where trying to change the default from
debian/rules is already too late, as that's managed by the
build driver.
* inside-out: where debian/rules, files sourced from it, or tools
invoked from it, fully control the outcome of the operation, and
then dpkg-buildpackage might not be able to tell beforehand
exactly what will happen and will need to pick up the results after
the fact, for example that would include dpkg-deb or dpkg-distaddfile
being currently fully delegated to debian/rules, and then
dpkg-buildpackage, et al. picking that up through debian/files;
debhelper would be a similar paradigm.
(With some exceptions, I consider that the bulk of our build interfaces
are unfortunately mostly inside-out.)
For this particular case, I'd envision the options could look something
like:
* outside-in:
- We add a new field, say (with this not very good name that would
need more thought) Build-Parallel-Mem-Limit-Per-Core for the
debian/control source stanza, then dpkg-buildpackage would be able
to check the current system memory, and clamp the number of
computed parallel jobs based on the number of system cores, the
number of specified parallel jobs and the limit from the above
field. This then would be passed down as the usual parallel=
DEB_BUILD_OPTIONS.
- If we needed the package to provide a dynamic value depending on
other external factors outside its control, although there's no
current precedent that I'm aware, and it seems a bit ugly, I guess
we could envision some kind of new entry point and a way to
let the build drivers know it needs to call it, for example a
debian/rules target that gets called and generates some file or
a program to call under debian/ that prints some value, which
dpkg-buildpackage could use in a similar way as the above point.
* inside-out:
For this, there could be multiple variants, where a build driver
like dpkg-buildpackage is completely out of the picture, and were
we might end up with parallel settings that are out-of-sync
between DEB_BUILD_OPTIONS parallel= and the inner one, for example:
- One could be the initially proposed buildopts.mk extension,
- Add a new dpkg-something helper or a new command to an existing
tool, that could compute the value that debian/rules would use
or pass further down,
- debhelper/debputy/etc does it, but that leaves out non-helper
using packages, which was one of the initial concerns from
Helmut.
Hope this clarifies.
Thanks,
Guillem
