Re: Proposed MBF: packages that FTBFS with make --shuffle
On 05/05/25 at 22:14 +0200, Santiago Vila wrote: > In some cases, the bug is already known, because debian/rules > has --max-parallel=1. Example: The alpine package. > > (I wonder how much feasible would be to skip those packages) The alpine package is indeed a good example of a package that makes extensive use of the sequentiality of 'make', and that is going to be hard to adjust to switch to parallel building or arbitrary orders. However I still think that there's value in filing bugs for such packages, because --shuffle=reverse makes it much easier to debug such issues: instead of trying a parallel build and getting a subtlely different race conditions at each run, you get a reproducible ordering that exhibits one issue that you can debug, and then move on to the next issue. Also it's not trivial to distinguish between packages that do not build in parallel on purpose, vs those that just happen not to build in parallel (yet). Lucas
Re: discussion extension (was: Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
Hi Mo, Am Mon, May 05, 2025 at 06:24:32PM -0400 schrieb M. Zhou: > Hi Andreas, > > According to constitution A.1.6, would you mind helping us extend > the discussion period by a week? > https://www.debian.org/devel/constitution I hereby extend the discussion period by a week. I admit that I consider this one of the most complex problems I've encountered during my time in Debian, and I hope the additional week will be sufficient. In any case, thanks to Mo for initiating the discussion and to everyone who contributed constructively. Kind regards Andreas. > >From the feedbacks I've heard, there are a couple of problems we are > facing currently. > > * Myself being confident in proposal A is one thing. But the audience > of this GR needs more information in order to make a well-thought > vote, while I'm suffering from low bandwidth in recent weeks. > > * People holding different opinions have really short time to prepare > a formal polished proposal. > > Maybe those factors indicate this is not the best time to vote. > I'm considering whether I should withdraw the proposal, collect more > information to fill up the overlooked aspects, further polish the > proposal A, and come back when it is ready. Doing so also allows > enough time for proposal B,C,D... since people know that I'm really > pushing this forward now. > > ... > > I don't believe we have enough information to do the GR now (or one > > week from today, the longest we can delay). I am unclear on whether > > existing packages in Debian are affected. Your proposal does not > > indicate whether the GR would be effective immediately. > > > > My suggestion is for you to ask the DPL to extend the discussion > > period by a week (for constitutional reasons) followed by an immediate > > withdrawal of the GR. Withdrawing the GR allows you to resubmit later > > and wait 2-3 weeks from that point. > > > > Thank you, > > Jeremy Bícha > > -- https://fam-tille.de signature.asc Description: PGP signature
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
On Mon, May 05, 2025 at 11:15:01AM -0400, Mo Zhou wrote: More information can be found at: https://www.debian.org/vote/2025/vote_002 I guess this is still in "discussion period"? When does that period end and the vote begin? That's described in https://www.debian.org/devel/constitution.en.html#item-A (mostly in A.1). There is no single numeric answer, but the minimum period is 2 weeks. "The minimum discussion period is 2 weeks. The maximum discussion period is 3 weeks." This is surprising to me. That's unfortunate. Does that mean we must start to vote when we reach the maximum discussion period? A.3.1: "After the discussion period has ended, the Project Secretary will publish the ballot and call for a vote. The Project Secretary may do this immediately following the end of the discussion period and must do so within seven days of the end of the discussion period." It is too rush to start to vote for this within 3 weeks Does this maybe sound like the GR call was premature? The project consensus, especially after https://www.debian.org/vote/2021/vote_003, seems to say that we don't want multi-month GR discussions. -- WBR, wRAR signature.asc Description: PGP signature
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
Hi, On Mon, 2025-05-05 at 11:15 -0400, Mo Zhou wrote: > It is too rush to start to vote for this within 3 weeks as I'm > completely not available for involving into discussions. It is two weeks unless something specific happens, so discussion period might already have ended by now... Ansgar
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
Hi Mo (2025.05.05_15:15:01_+) "The minimum discussion period is 2 weeks. The maximum discussion period is 3 weeks." This is surprising to me. Does that mean we must start to vote when we reach the maximum discussion period? I just thought I can go back and reply to the detailed issues in the threads after getting through the time trap of paper submission deadline. Read section A.2 of the constitution: you can withdraw your ballot option, and the GR won't happen. Others may pick it up and carry it through to a GR, though. This doesn't stop you from calling a GR later. Stefano -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
Hi, On Mon, 2025-05-05 at 16:13 +, Stefano Rivera wrote: > Read section A.2 of the constitution: you can withdraw your ballot > option, and the GR won't happen. Others may pick it up and carry it > through to a GR, though. There is that part though: +--- | No new ballot options may be proposed, no ballot options may | be amended, and no proposers or sponsors may withdraw if less than | 24 hours remain in the discussion period, unless this action | successfully extends the discussion period under §A.1.4 by at | least 24 additional hours. +--- One could add an additional ballot option or the project leader might extend the discussion period (unless it is already over I guess). Ansgar
discussion extension (was: Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
Hi Andreas, According to constitution A.1.6, would you mind helping us extend the discussion period by a week? https://www.debian.org/devel/constitution >From the feedbacks I've heard, there are a couple of problems we are facing currently. * Myself being confident in proposal A is one thing. But the audience of this GR needs more information in order to make a well-thought vote, while I'm suffering from low bandwidth in recent weeks. * People holding different opinions have really short time to prepare a formal polished proposal. Maybe those factors indicate this is not the best time to vote. I'm considering whether I should withdraw the proposal, collect more information to fill up the overlooked aspects, further polish the proposal A, and come back when it is ready. Doing so also allows enough time for proposal B,C,D... since people know that I'm really pushing this forward now. On Mon, 2025-05-05 at 16:09 -0400, Jeremy Bícha wrote: > On Mon, May 5, 2025 at 2:49 PM Mo Zhou wrote: > > On 5/5/25 11:44, Andrey Rakhmatullin wrote: > > > > It is too rush to start to vote for this within 3 weeks > > > > > > Does this maybe sound like the GR call was premature? > > > The project consensus, especially after > > > https://www.debian.org/vote/2021/vote_003, seems to say that we don't > > > want multi-month GR discussions. > > > > > > > Not quite. Proposal A is mature and I'm confident in it. > > Potential proposal B,C,D,... are premature. > > I have no intention to let people holding different opinions > > to have a very short time to prepare a formal proposal B,C,D. > > > > But, the whole series of discussions started 7 years ago. > > And I have already mailed everywhere about my intention > > to submit the GR. If there is no proposal B, that could mean > > it is really difficult to formally write a proposal B. > > > > I lean towards going ahead. > > I don't believe we have enough information to do the GR now (or one > week from today, the longest we can delay). I am unclear on whether > existing packages in Debian are affected. Your proposal does not > indicate whether the GR would be effective immediately. > > My suggestion is for you to ask the DPL to extend the discussion > period by a week (for constitutional reasons) followed by an immediate > withdrawal of the GR. Withdrawing the GR allows you to resubmit later > and wait 2-3 weeks from that point. > > Thank you, > Jeremy Bícha
Re: Proposed MBF: packages that FTBFS with make --shuffle
El 5/5/25 a las 21:26, Lucas Nussbaum escribió:
[...]
Thanks a lot for this. I was never brave enough to go ahead
and announce a MBF.
May I know what kind of machines did you use to found those bugs?
Machines with 8 CPUs only? (I ask because I found more than 800
packages with makefile issues triggered by make --shuffle,
using machines with 1 and 2 CPUs).
I'd rather do the bug submitting now to avoid refreshing build results
later, but I agree that this is not release-critical material of course,
so I can also wait until after the trixie release.
Debian Policy does not say "Makefiles should be correct", but I think
that has always been implicit. The GPL speaks about Makefiles as an integral
part of source code ("plus the scripts used to control compilation
and installation of the executable" in GPL-2).
So, I would say this is a normal bug.
Thanks.
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
On Mon, May 5, 2025 at 2:49 PM Mo Zhou wrote: > On 5/5/25 11:44, Andrey Rakhmatullin wrote: > >> It is too rush to start to vote for this within 3 weeks > > > > Does this maybe sound like the GR call was premature? > > The project consensus, especially after > > https://www.debian.org/vote/2021/vote_003, seems to say that we don't > > want multi-month GR discussions. > > > > Not quite. Proposal A is mature and I'm confident in it. > Potential proposal B,C,D,... are premature. > I have no intention to let people holding different opinions > to have a very short time to prepare a formal proposal B,C,D. > > But, the whole series of discussions started 7 years ago. > And I have already mailed everywhere about my intention > to submit the GR. If there is no proposal B, that could mean > it is really difficult to formally write a proposal B. > > I lean towards going ahead. I don't believe we have enough information to do the GR now (or one week from today, the longest we can delay). I am unclear on whether existing packages in Debian are affected. Your proposal does not indicate whether the GR would be effective immediately. My suggestion is for you to ask the DPL to extend the discussion period by a week (for constitutional reasons) followed by an immediate withdrawal of the GR. Withdrawing the GR allows you to resubmit later and wait 2-3 weeks from that point. Thank you, Jeremy Bícha
Re: Proposed MBF: packages that FTBFS with make --shuffle
Hi,
On 05/05/25 at 21:53 +0200, Santiago Vila wrote:
> El 5/5/25 a las 21:26, Lucas Nussbaum escribió:
> > [...]
>
> Thanks a lot for this. I was never brave enough to go ahead
> and announce a MBF.
>
> May I know what kind of machines did you use to found those bugs?
> Machines with 8 CPUs only? (I ask because I found more than 800
> packages with makefile issues triggered by make --shuffle,
> using machines with 1 and 2 CPUs).
Yes, 8 cores. I looked at the differences between your results and
mine, and while I did not look at all packages that failed for you but
are not included in my list of 511 packages, I identified the following
causes for differences, order by decreasing impact in the small sample I
checked:
- I did not check packages that are not currently in testing.
- I excluded packages that FTBFS even without --shuffle.
- some packages failed for you for other reasons, but now build fine
even with --shuffle.
- I excluded packages that FTBFS even with --shuffle=none (but build
fine without --shuffle). Examples are packages that use bmake
(BSD make), such as csh. What happens is that GNU Make sets MAKEFLAGS
to --shuffle, which bmake does not understand.
> > I'd rather do the bug submitting now to avoid refreshing build results
> > later, but I agree that this is not release-critical material of course,
> > so I can also wait until after the trixie release.
>
> Debian Policy does not say "Makefiles should be correct", but I think
> that has always been implicit. The GPL speaks about Makefiles as an integral
> part of source code ("plus the scripts used to control compilation
> and installation of the executable" in GPL-2).
>
> So, I would say this is a normal bug.
I'm not going to argue about 'normal' vs 'minor'.
Lucas
Re: Proposed MBF: packages that FTBFS with make --shuffle
In some cases, the bug is already known, because debian/rules has --max-parallel=1. Example: The alpine package. (I wonder how much feasible would be to skip those packages) Thanks.
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
On 5/5/25 01:58, Andrey Rakhmatullin wrote: On Sun, May 04, 2025 at 09:24:45PM -0500, Steven Robbins wrote: More information can be found at: https://www.debian.org/vote/2025/vote_002 I guess this is still in "discussion period"? When does that period end and the vote begin? That's described in https://www.debian.org/devel/constitution.en.html#item-A (mostly in A.1). There is no single numeric answer, but the minimum period is 2 weeks. "The minimum discussion period is 2 weeks. The maximum discussion period is 3 weeks." This is surprising to me. Does that mean we must start to vote when we reach the maximum discussion period? I just thought I can go back and reply to the detailed issues in the threads after getting through the time trap of paper submission deadline. It is too rush to start to vote for this within 3 weeks as I'm completely not available for involving into discussions. At least I'd like to see potential voters to be well informed, as the current status is the number of questions is much greater than answers. Missing a formal "Proposal B" is also an indicator for such situation. I have belief in my proposal. But now it seems the time needed to justify the design details of proposal A is longer than the GR process. For example, proposal A has explicitly defined the scope to one single (and most problematic) case to stay at maximum simplicity. By proposing "XXX is not DFSG-compliant" can directly avoid the trap of "defining what is free software AI within a single GR". That problem is far beyond the capacity that a GR can handle. OSI has spent years on this issue (while I started mentioning this much earlier), and yet they converged into a controvertial definition. I have been arguing on the training data issue at the very beginning of OSI's working group on OSAID and my argument is continuously being ignored. Eventually I no longer bother to argue. I know what I'm doing as a person who simultaneously speaks from @debian.org and trains neural networks (both small ones and large ones) every single day.And now what I mentioned at the very begining of OSAID is exactly the most controvertial point of OSAID after its release. Given the trajectory of OSI's attempt about this, I would like emphasize the proposal A is in my opinion the most problematic and the most vote-able point. Trying to explicitly expand the definition of "DFSG-compliant AI" will result in the expansion of countless troubles and mess up the scope of the GR. Thus, to make the GR proposal practical, it only says "what is NOT" instead of "what IS". As I mentioned in Appendix D "Independence Utopia". It makes me uncomfortable to define "DFSG-compliant AI" before I can really create such a thing with my limited resource for FOSS actitvity. https://salsa.debian.org/lumin/gr-ai-dfsg/-/blob/main/AppendixD.txt I think people in the FOSS community understands what simplicity means. Even if we are forced to vote after extension, I'm still very confident in proposal A. It's just a pity that I cannot involve in discussion for my poor bandwidth.
Re: General Resolution: Interpretation of DFSG on Artificial Intelligence (AI) Models
On 5/5/25 11:44, Andrey Rakhmatullin wrote: It is too rush to start to vote for this within 3 weeks Does this maybe sound like the GR call was premature? The project consensus, especially after https://www.debian.org/vote/2021/vote_003, seems to say that we don't want multi-month GR discussions. Not quite. Proposal A is mature and I'm confident in it. Potential proposal B,C,D,... are premature. I have no intention to let people holding different opinions to have a very short time to prepare a formal proposal B,C,D. But, the whole series of discussions started 7 years ago. And I have already mailed everywhere about my intention to submit the GR. If there is no proposal B, that could mean it is really difficult to formally write a proposal B. I lean towards going ahead.
Proposed MBF: packages that FTBFS with make --shuffle
Hi,
GNU Make now has a --shuffle option that simulates non-deterministic
ordering of target prerequisites. See
https://trofi.github.io/posts/238-new-make-shuffle-mode.html and also
previous work in Debian by Santiago Vila:
https://people.debian.org/~sanvila/make-shuffle/
While make always processes prerequisites left-to-right when running
sequentially, prerequisites can be evaluated in an arbitrary order when
building in parallel (make -j). This option is thus useful to trigger
and debug issues that occur when building in parallel.
I identified 511 packages in unstable[0] (dd-list[1]) that fail to build
with 'make --shuffle=reverse' or 'make --shuffle=random', but do not fail to
build with 'make --shuffle=none'.
[0] http://qa-logs.debian.net/2025/05/05/shuffle/sources.txt
[1] http://qa-logs.debian.net/2025/05/05/shuffle/dd-list.txt
Builds logs are available in http://qa-logs.debian.net/2025/05/05/shuffle/
I would like to submit bugs (severity:minor) against those packages.
I'd rather do the bug submitting now to avoid refreshing build results
later, but I agree that this is not release-critical material of course,
so I can also wait until after the trixie release.
One could argue that those issues are not bugs in Debian since they
cannot be triggered in packages that do not build in parallel. However
building in parallel is now the default, so I think that even for those
packages that do not build in parallel yet, it is good to know that
there is something broken that could bite with a
difficult-to-reproduce-and-explain race condition if parallel building
was enabled.
I prepared a wiki page [2] and debugged a few failures listed on that
page (#1104421 #1104428 #1104429 #1104430 #1104751).
[2] https://wiki.debian.org/qa.debian.org/FTBFS/Shuffle
Note to self or anyone interested: I did not check if packages built with
or without --shuffle were identical. That could be fun as a future work.
Proposed bug template:
--->8
From: {{ fullname }} <{{ email }}>
To: sub...@bugs.debian.org
Subject: {{ package }}: FTBFS with make --shuffle: {{ summary }}
Source: {{ package }}
Version: {{ version }}
Severity: minor
Tags: trixie sid ftbfs
User: lu...@debian.org
Usertags: ftbfs-shuffle
Hi,
GNU Make now has a --shuffle option that simulates non-deterministic ordering
of target prerequisites. See
https://trofi.github.io/posts/238-new-make-shuffle-mode.html and also previous
work in Debian by Santiago Vila:
https://people.debian.org/~sanvila/make-shuffle/
This package fails to build with make --shuffle=reverse or make
--shuffle=random. This is likely to be caused by a missing dependency in
debian/rules or an upstream Makefile.
More information about this mass bug filing is available at
https://wiki.debian.org/qa.debian.org/FTBFS/Shuffle
[...]
--->8
- Lucas
Re: Proposed MBF: packages that FTBFS with make --shuffle
On Monday, May 5, 2025 12:26:00 PM Mountain Standard Time Lucas Nussbaum wrote: > Hi, > > GNU Make now has a --shuffle option that simulates non-deterministic > ordering of target prerequisites. See > https://trofi.github.io/posts/238-new-make-shuffle-mode.html and also > previous work in Debian by Santiago Vila: > https://people.debian.org/~sanvila/make-shuffle/ > > While make always processes prerequisites left-to-right when running > sequentially, prerequisites can be evaluated in an arbitrary order when > building in parallel (make -j). This option is thus useful to trigger > and debug issues that occur when building in parallel. > > I identified 511 packages in unstable[0] (dd-list[1]) that fail to build > with 'make --shuffle=reverse' or 'make --shuffle=random', but do not fail to > build with 'make --shuffle=none'. > > [0] http://qa-logs.debian.net/2025/05/05/shuffle/sources.txt > [1] http://qa-logs.debian.net/2025/05/05/shuffle/dd-list.txt > > Builds logs are available in http://qa-logs.debian.net/2025/05/05/shuffle/ > > I would like to submit bugs (severity:minor) against those packages. > > I'd rather do the bug submitting now to avoid refreshing build results > later, but I agree that this is not release-critical material of course, > so I can also wait until after the trixie release. > > One could argue that those issues are not bugs in Debian since they > cannot be triggered in packages that do not build in parallel. However > building in parallel is now the default, so I think that even for those > packages that do not build in parallel yet, it is good to know that > there is something broken that could bite with a > difficult-to-reproduce-and-explain race condition if parallel building > was enabled. Filing these bug reports sounds like a good idea to me. I don’t see any reason to wait as these will be severity:minor, so they won’t interfere with the trixie release. -- Soren Stoutner so...@debian.org signature.asc Description: This is a digitally signed message part.
Re: Bug#1104643: Don't consider tests during build that can use internet if available as rc buggy
[adding -devel] On 05/05/2025 2:49 pm, Bill Allombert wrote: On Sat, May 03, 2025 at 09:11:21PM +0530, Pirate Praveen wrote: Package: debian-policy Version: 4.7.2.0 Dear Pirate, Control: block 1104509 by -1 As a general policy, such block is inappropriate. Package are supposed to comply with policy at the time they are uploaded. They cannot depend on future potential policy update. It was more like a reference as the other option to declare relationship is affects which only allows mentioning a package and not a specific bug. Current policy text says: Except for packages in the non-free archive with the Autobuild control field unset or set to no, required targets must not attempt network access, except, via the loopback interface, to services on the build host that have been started by the build. I think it should be changed to, Except for packages in the non-free archive with the Autobuild control field unset or set to no, required targets must not require network access, except, via the loopback interface, to services on the build host that have been started by the build. I think enforcing there is no internet access is a better way to achieve the goal of actually ensuring there is no internet during build rather than considering packages that can use internet when available for tests as rc buggy. I disagree. This was not the consensus at the time I made this change to policy, and I do not think it is the consensus now. We want more reproducible builds, not depending on external resources that are bound to change, and not being tracked via server logs. In your case building the package with internet access - fails if timestamp.digicert.com is down - leaks the system IP to DIGICERT I think we have to consider test target in rules differently from build targets as the effect on these on the final binaries we ship is different. I agree the current policy fit well when applied to the build target. As we don't want to ship anything not present / built from the source package. But anything runs in the test target does not affect the final binaries. The only difference that makes is more functionalities are tested. Because of the current policy we are unable to test any functionality that needs an internet connection. So I think the current policy is hiding potential problems that could be discovered early if these tests are actually run during build. Completly disabling access to internet during a build is harder than it sound. I think sbuild unshare backend does this by default (correct me if I am wrong). I think trying to work around this limitation is actually hurting the quality of our packages since are skipping many tests entirely due to this policy restriction. I don't think we should be skipping these tests in salsa ci or debci. In my opinion, only buildd need to enforce no internet during build restriction. Cheers, OpenPGP_0x8F53E0193B294B75.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Bug#1104643: Don't consider tests during build that can use internet if available as rc buggy
On Tue, May 06, 2025 at 01:12:43AM +0530, Pirate Praveen wrote: > I think we have to consider test target in rules differently from build > targets as the effect on these on the final binaries we ship is different. > > I agree the current policy fit well when applied to the build target. As we > don't want to ship anything not present / built from the source package. > > But anything runs in the test target does not affect the final binaries. The > only difference that makes is more functionalities are tested. Because of > the current policy we are unable to test any functionality that needs an > internet connection. So I think the current policy is hiding potential > problems that could be discovered early if these tests are actually run > during build. That assumes that the buildd operators are willing to let your tests run with network access. autopkgtest has an option to allow network access, so you can use that. If you need the test to be performed as part of the build, propose a new DEB_BUILD_OPTION 'allownetworktest' and make your tests conditional on that. Cheers, -- Bill. Imagine a large red swirl here.
Re: Bug#1104643: Don't consider tests during build that can use internet if available as rc buggy
On 06/05/2025 1:33 am, Bill Allombert wrote: On Tue, May 06, 2025 at 01:12:43AM +0530, Pirate Praveen wrote: I think we have to consider test target in rules differently from build targets as the effect on these on the final binaries we ship is different. I agree the current policy fit well when applied to the build target. As we don't want to ship anything not present / built from the source package. But anything runs in the test target does not affect the final binaries. The only difference that makes is more functionalities are tested. Because of the current policy we are unable to test any functionality that needs an internet connection. So I think the current policy is hiding potential problems that could be discovered early if these tests are actually run during build. That assumes that the buildd operators are willing to let your tests run with network access. buildds don't need to allow it, only salsa ci and debci need to allow it. autopkgtest has an option to allow network access, so you can use that. If you need the test to be performed as part of the build, propose a new DEB_BUILD_OPTION 'allownetworktest' and make your tests conditional on that. that is a good idea so this could be enabled on salsa ci for example. Cheers, OpenPGP_0x8F53E0193B294B75.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
