Re: Building with many cores without OOM

Hi, Let me plop in right into this discussion with no general solution and more things to think about. For the context I'm packaging Java things, and Java has historically been notoriously bad at guessing how much memory it could actually use on a given system. I'm not sure things are much be

Re: Building with many cores without OOM

Hi! On Thu, 2024-12-05 at 09:23:24 +0100, Helmut Grohne wrote: > On Wed, Dec 04, 2024 at 02:03:29PM +0100, Guillem Jover wrote: > > On Thu, 2024-11-28 at 10:54:37 +0100, Helmut Grohne wrote: > > > For one thing, I propose extending debhelper to provide > > > --min-ram-per-parallel-core as that see

Re: Building with many cores without OOM

Hi Guillem and others, Thanks for your extensive reply and the followup clarifying the inside-out and outside-in distinction. On Wed, Dec 04, 2024 at 02:03:29PM +0100, Guillem Jover wrote: > On Thu, 2024-11-28 at 10:54:37 +0100, Helmut Grohne wrote: > > I think this demonstrates that we probably

Re: Building with many cores without OOM

Hi, On 12/4/24 23:37, Stefano Rivera wrote: I don't think this can be entirely outside-in, the package needs to say how much ram it needs per-core, to be able to calculate the appropriate degree of parallelism. So, we have to declare a value that then gets calculated against the proposed parall

Re: Building with many cores without OOM

Hi! On Wed, 2024-12-04 at 14:37:45 +, Stefano Rivera wrote: > Hi Guillem (2024.12.04_13:03:29_+) > > > Are there other layers that could reasonably be used to implement a more > > > general form of parallelism limiting based on system RAM? Ideally, we'd > > > consolidate these implementati

Re: Building with many cores without OOM

Hi Guillem (2024.12.04_13:03:29_+) > > Are there other layers that could reasonably be used to implement a more > > general form of parallelism limiting based on system RAM? Ideally, we'd > > consolidate these implementations into fewer places. > > I think adding this in dpkg-buildpackage itse

Re: Building with many cores without OOM

Hi! On Wed, 2024-12-04 at 14:03:30 +0100, Guillem Jover wrote: > On Thu, 2024-11-28 at 10:54:37 +0100, Helmut Grohne wrote: > > Are there other layers that could reasonably be used to implement a more > > general form of parallelism limiting based on system RAM? Ideally, we'd > > consolidate these

Re: Building with many cores without OOM

Hi! On Thu, 2024-11-28 at 10:54:37 +0100, Helmut Grohne wrote: > I am one of those who builds a lot of different packages with different > requirements and found that picking a good parallel=... value in > DEB_BUILD_OPTIONS is hard. Go too low and your build takes very long. Go > too high and you

Re: Building with many cores without OOM

Hi Helmut, On 11/29/24 07:59, Helmut Grohne wrote: On Thu, Nov 28, 2024 at 02:39:36PM +0100, Paul Gevers wrote: And doing it in a way that can be reused by how autopkgtests are run would maybe be good too. Can you clarify what you mean here? There is autopkgtest --build-parallel and my unders

Re: Building with many cores without OOM

Helmut Grohne: Hi Guillem and other developers, I am one of those who builds a lot of different packages with different requirements and found that picking a good parallel=... value in DEB_BUILD_OPTIONS is hard. Go too low and your build takes very long. Go too high and you swap until the OOM ki

Re: Building with many cores without OOM

Hi Paul, On Thu, Nov 28, 2024 at 02:39:36PM +0100, Paul Gevers wrote: > On 11/28/24 13:01, Chris Hofstaedtler wrote: > > IMO it would be good to support dealing with this earlier than > > later. > > And doing it in a way that can be reused by how autopkgtests are run would > maybe be good too. C

Re: Building with many cores without OOM

Hi Helmut, On 11/28/24 13:01, Chris Hofstaedtler wrote: IMO it would be good to support dealing with this earlier than later. And doing it in a way that can be reused by how autopkgtests are run would maybe be good too. Paul

Re: Building with many cores without OOM

On Thu, Nov 28, 2024 at 10:54:37AM +0100, Helmut Grohne wrote: > I think this demonstrates that we probably have something between 10 and > 50 packages in unstable that would benefit from a generic parallelism > limit based on available RAM. Do others agree that this is a problem > worth solving in

Re: Building with many cores without OOM

* Helmut Grohne [241128 10:59]: > I think this demonstrates that we probably have something between 10 and > 50 packages in unstable that would benefit from a generic parallelism > limit based on available RAM. Do others agree that this is a problem > worth solving in a more general way? Yes. Loo