Produce extra build artifacts during package builds

[Mattia Rizzolo]

Hello people,

just now at DebConf we had a small BOF to discuss this problem, that has
several times come up.

Lastly I think https://lists.debian.org/debian-devel/2022/02/msg00216.html
to which Simon had a quite nice answer.


This is what we came up with:
https://wiki.debian.org/BuildArtifacts#Design_proposal

I'm copying it here for reference:

 * decide on a directory name (`./debian/export_artifacts/`?)
 * the build process will dump any files that could be interesting to
   export in there (no decision if/how to define the structure within
   this directory)
   * potentially, all `dh_*` could copy there whatever is relevant to
 them (i.e., `dh_auto_configure` could export configure logs,
 `dh_auto_test` the various test logs, ..)
 * the containering tool (sbuild, pbuilder, ...) at the end of the
   build, regardless of the result of the build:
   * check if the directory exists, and it's non-empty
   * if true, tar it up
   * place the tar'ed-up artifacts in the usual results directory, with
 a predictive name (`$srcpkg_$ver_$arch.artifacts.tar.xz`?)
   * tbd if this should happen always or opt-in by the build caller
 * `buildd`/`pybuildd` will check if such file has been exported, if it
is, then it will be uploaded to some service
   * something as simple as an authenticate HTTP POST somewhere will
 do - no strong auth needed (right?)
 * this new service will simply just accept whatever upload and store it
 * possibly define an expiry rule for these artifacts to not hog all
   space needlessly


Matthias Klose would like to find a way to do something that at the end
of a build will look for and collect all the relevant files that are
produced during an ICE (for example all the pre-processor inputs, etc).
I'm not sure how to best solve it, but he was imagining a hook somewhere
(in dpkg?) that would look for them and copy them into the directory.




Looking for inputs!


-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Andrea Pappacoda]

Hi Otto, and all the others participating in this thread :)

Il giorno sab 27 lug 2024 alle 15:38:40 -07:00:00, Otto Kekäläinen 
 ha scritto:
I have drafted a new DEP at 
https://salsa.debian.org/dep-team/deps/-/merge_requests/8 titled 
"DEP-18: Enable true open collaboration on all Debian packages".


Thanks for your work on this. Trying to unify aspects of Debian 
development is one of the things I think we need to do to not only 
"enable true open collaboration", but also to attract more new 
contributors.


While the proposal has good intentions and goals I agree with, it tries 
to achieve it with tools which, to my eyes, don't really enable "true" 
open collaboration, which Jonas has expressed really well already.


The issue to me is when you add the word "Salsa", or more precisely 
"GitLab" to the mix. Don't get me wrong: these days development *has* 
to be done with Git and with CI workflows, but having to use GitLab 
just to have these two things is overkill.


Before a certain way of doing things can be mandated or "warmly 
recommended", the technology has to be as flawless as possible - and 
today I wouldn't call Salsa "flawless", would you? Issues with 
Salsa/GitLab:


1. It is so slow that it makes me want to close by browser and do 
something else instead

2. It doesn't support email-based workflows
3. It has a lot of features we don't need. Or rather, it has more 
features we *don't* need than we do.
4. Its user interface is confusing, it's often hard to find what I'm 
looking for

5. Did I mention how slow it is?
6. It is developed by a company who's philosophy and interests are 
misaligned with Debian's
7. The product it tries to mimic, GitHub, is also misaligned with 
Debian's philosophy and interests


While performance is something we could reasonably do something about, 
all the other points are part of GitLab by design, and we're stuck with 
them.


It has also been mentioned that email-based workflows are for 
"dinosaurs". Well, I'm 21, so I'm a living example that that's not 
necessarily true :)


But the point isn't that much about being able to use emails 
specifically, it is about having the possibility of having a software 
forge which is interoperable with a wide range of tools and can stay 
out of the way if wanted. With GitLab, you either use the full package 
(which includes browsing, reviewing, and commenting) or you get a 
sub-optimal experience. But it doesn't have to be this way.


I'll now talk a bit about SourceHut. Not to suggest that we should use 
that instead, but just to point out how things *can* be done.


1. It is really lightweight
2. It places emphasis on email-based workflows, while also providing a 
web interface which can be used to view and interact with email 
submissions.
3. It is modular. If you don't want the mailing list manager or the 
issue tracker you can simply not install them.

4. The web interface exposes few things, maybe even too little
5. Yes, it's fast
6. Its developers truly value software freedom, just like Debian does
7. It tries to offer an independent product, without following GitHub's 
lead


Since I'm lucky enough to still go to university, I conducted some 
small scale experiment with a handful of students which were new to 
software development. For their first Git group project, I let them use 
SourceHut instead of GitHub, to see how accessible SourceHut was to new 
users. They've been able to use it without issues, and I was happy. 
Some time after this, they also had the opportunity of using GitHub, 
which is not a surprise given its dominant position. The interesting 
part though is that these students, when starting a new personal 
project, chose to use SourceHut even if they also knew GitHub. They 
found it more usable!


In conclusion, I think we should aim at providing a set of tools which 
provides a "normalized view/workflow" which anyone can use, while also 
giving the possibility to individuals to use their own preferred 
personal workflow for their own work. GitLab provides this normalized 
workflow, but by making it the only viable option. Just like how dgit 
provides a canonical "dgit view" while letting maintainers user their 
own packaging workflow, an ideal Debian code forge would provide, for 
example, a canonical web UI which can also be interacted with by 
email/command line, just like the rest of Debian's infrastructure.


I would've liked to provide concrete solutions, but for now I'll limit 
myself at exposing problems :)


Bye!

Bug#1077794: ITP: python3-panflute -- pythonic pandoc filters

[Kyle Robbertze]

Package: wnpp
Severity: wishlist
Owner: Kyle Robbertze 
X-Debbugs-Cc: debian-devel@lists.debian.org

* Package name: python3-panflute
  Version : 2.3.0
  Upstream Contact: Sergio Correia 
* URL : https://scorreia.com/software/panflute/
* License : BDS3
  Programming Lang: python
  Description : pythonic pandoc filters

A Python implementation of pandoc filters making filters as simple and
clear as possible. It comes with some batteries included for converting
between markdown and Python objects, calling external programs and
running panflute as a filter.

It will be maintained under the Python team

Re: Produce extra build artifacts during package builds

[Simon McVittie]

On Fri, 02 Aug 2024 at 16:40:15 +0900, Mattia Rizzolo wrote:
> Lastly I think https://lists.debian.org/debian-devel/2022/02/msg00216.html
> to which Simon had a quite nice answer.

I also did some more concrete design and wrote a prototype -
 as revised
in , and
 -
although I never got as far as doing enough testing and polishing to get
it to a production-ready status. I'm sorry that this did not stay as my
top priority, but I can only have one top priority at a time.

(There is also some good input from Paul Wise in the 2022 thread)

>  * decide on a directory name (`./debian/export_artifacts/`?)
>  * the build process will dump any files that could be interesting to
>export in there (no decision if/how to define the structure within
>this directory)
>* potentially, all `dh_*` could copy there whatever is relevant to
>  them (i.e., `dh_auto_configure` could export configure logs,
>  `dh_auto_test` the various test logs, ..)

My concern about this, and the reason my prototype doesn't work this way,
is what I said in
.

The artifacts that I'm primarily interested in are the results of failed
testing, because at the moment librsvg and gtk4 have to exfiltrate
their failed test results (some of which are in the form of images,
because these are GUI libraries) by uuencoding PNG files and writing
the result to the log, and that's both inconvenient and inefficient.

But, when a test has failed, writing arbitrary imperative code to gather
up the package-specific results (for example the images that were generated
by the gtk4 test suite) is extra complexity for the package maintainer,
which can have bugs, and is rarely tested because hopefully the tests
usually pass. I dislike that as a pattern, and I'd strongly prefer a
declarative syntax of some sort.

(Take a look at all the ad-hoc scripting that src:gtk4 has around what
to do after test failures and I think you'll see what I mean!)

In the design that I prototyped, it's declarative, loosely inspired by
the equivalent Gitlab-CI feature:

- the maintainer can write patterns into debian/build-artifacts for
  package-specific quirks like GTK's reftests
  (#-prefixed comments are allowed)
- tools like debhelper can append patterns to debian/extra-build-artifacts,
  for example dh_auto_* in Meson mode should append "obj-*/meson-logs/*"
- the buildd operator can configure $artifact_patterns in .sbuildrc if
  they want to (I'm unsure how useful this genuinely is, but maintainers
  running sbuild locally might find it more convenient than editing the
  source package during ad-hoc debugging)
- the patterns reuse machine-readable debian/copyright syntax
- as a result the -artifacts.tar.gz is always a subset of the build tree,
  and inherits its layout, same as Gitlab's artifacts.zip
- if a sufficiently desperate maintainer really wants to, they can tar up
  the entire build tree by specifying a broad enough pattern like '*',
  although this should be discouraged when building larger packages :-)

and sbuild (or pbuilder if you prefer) is responsible for enumerating files
matching the given patterns and packing them into a tarball.

>* place the tar'ed-up artifacts in the usual results directory, with
>  a predictive name (`$srcpkg_$ver_$arch.artifacts.tar.xz`?)

I prototyped this as the same name as the log file, except replacing the
.build extension with -artifacts.tar.gz (so the filename has a timestamp
in it). My rationale for this choice is that when tests fail, it's common
to retry the build to see whether the tests are reproducibly failing or
just flaky, so we want to collect one blob of artifacts per attempt, the
same way we collect one log per attempt.

I don't know how the layers "above" sbuild such as buildd and wanna-build
operate, and I don't have a way to test them, so my prototype stops at
sbuild.

> Matthias Klose would like to find a way to do something that at the end
> of a build will look for and collect all the relevant files that are
> produced during an ICE (for example all the pre-processor inputs, etc).
> I'm not sure how to best solve it, but he was imagining a hook somewhere
> (in dpkg?) that would look for them and copy them into the directory.

If we need an imperative hooks mechanism for something like this, then so
be it, but I'd really prefer executing arbitrary code to be the uncommon
case rather than something we always have to do.

Perhaps dpkg or dh_auto_build could append appropriate patterns to my
debian/extra-build-artifacts on build failure, so that this hook would
just be acting as an extension to the declarative mechanism?

smcv
(not at Debconf)

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Gioele Barabucci]

On 02/08/24 11:20, Andrea Pappacoda wrote:

Issues with Salsa/GitLab:

1. It is so slow that it makes me want to close by browser and do 
something else instead

[...]
5. Did I mention how slow it is?


Just as a side note: yes, salsa.d.o/GitLab is not the snappiest Web 
application ever and sometimes it takes seconds for a page to appear, 
but we are talking about Debian here, where bug reports are acknowledged 
by debbugs minutes after being filed, uploaded packages are dinstalled 
once every 6 hours, official CI builds take days to complete, patches 
are merged months after being submitted.


BTW, the `glab` command-line program is a nice(r) and faster alternative 
to the Web interface for interacting with salsa.d.o.


Regards,

--
Gioele Barabucci

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Blair Noctis]

On 02/08/2024 06:38, Salvo Tomaselli wrote:
> Also, there's IRC/matrix for more real time communication, but I challenge 
> you 
> to follow those long threads on d-devel on something like teams or slack.
Discourse. Or some other "forum" software. IMO online forums and mailing lists
are pretty similar in core functions (threaded conversations), they just differ
in UI.

-- 
Sdrager,
Blair Noctis


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Fabio Fantoni]

Il 02/08/2024 11:20, Andrea Pappacoda ha scritto:

Hi Otto, and all the others participating in this thread :)

Il giorno sab 27 lug 2024 alle 15:38:40 -07:00:00, Otto Kekäläinen 
 ha scritto:
I have drafted a new DEP at 
https://salsa.debian.org/dep-team/deps/-/merge_requests/8 titled 
"DEP-18: Enable true open collaboration on all Debian packages".


Thanks for your work on this. Trying to unify aspects of Debian 
development is one of the things I think we need to do to not only 
"enable true open collaboration", but also to attract more new 
contributors.


While the proposal has good intentions and goals I agree with, it 
tries to achieve it with tools which, to my eyes, don't really enable 
"true" open collaboration, which Jonas has expressed really well already.


The issue to me is when you add the word "Salsa", or more precisely 
"GitLab" to the mix. Don't get me wrong: these days development *has* 
to be done with Git and with CI workflows, but having to use GitLab 
just to have these two things is overkill.



Having the packaging on git i think is very useful, whether it is on 
salsa or another system it would bring benefits to some packages not 
currently on git. Not to force to use it but at least recommend it (and 
then later reconsider whether to force it)


One particular thing noticed in some cases (and I hope they are not 
many) is the lack of use or especially updates of the Vcs-* fields in 
d/control. I think is important point to packaging repository from the 
tracker if present and to update it if moved, this can help 
collaboration and reduce possible waste of time doing things that are 
perhaps already done by others (or WIP) but which have not yet been 
uploaded.





Before a certain way of doing things can be mandated or "warmly 
recommended", the technology has to be as flawless as possible - and 
today I wouldn't call Salsa "flawless", would you? Issues with 
Salsa/GitLab:


1. It is so slow that it makes me want to close by browser and do 
something else instead

2. It doesn't support email-based workflows
3. It has a lot of features we don't need. Or rather, it has more 
features we *don't* need than we do.
4. Its user interface is confusing, it's often hard to find what I'm 
looking for

5. Did I mention how slow it is?
6. It is developed by a company who's philosophy and interests are 
misaligned with Debian's
7. The product it tries to mimic, GitHub, is also misaligned with 
Debian's philosophy and interests


While performance is something we could reasonably do something about, 
all the other points are part of GitLab by design, and we're stuck 
with them.


It has also been mentioned that email-based workflows are for 
"dinosaurs". Well, I'm 21, so I'm a living example that that's not 
necessarily true :)


I think that both email and systems like salsa/github/gitlab etc. are 
useful, both with pros and cons. Forcing people to use only one or the 
other could be counterproductive at the moment. One thing that I think 
would be useful is to have certain, fast and simple information for each 
package about which actual collaboration methods it uses and prefers.


For example seeing a package it would be very useful to know immediately 
if it wants a collaboration only through bugtracker and patch, only 
through vcs or both are accepted. If managed in a team point more easily 
to information about the team and any pages with details (for example on 
wiki).


I mainly use salsa, it has many advantages and could bring improvements 
in many cases but as mentioned there are also disadvantages (partly 
subjective) and therefore forcing it I don't think is a good thing, 
rather recommending it instead.


But first of all I think it is essential to improve salsa performance 
and reduce downtime and problems. Over the last few years I have found 
myself too many times in cases where I could work on packages but salsa 
was not working or very slow, and many cases where I needed salsa-ci for 
quick checks but it had problems. This is discouraging for contributors 
and counterproductive (especially for those with little time available). 
Recently I think there has been some improvement compared to past 
periods when it was worse both with salsa speed/reachability and 
salsa-ci problems (which lasted longer), big thanks to all those who 
work on it to improve it and I hope it will improve further in the future.


Another thing that seems underestimated and I think it is good to 
emphasize is the excessive slowness of the wiki.debian.org, it seems 
much worse than salsa to me, and I think it's important to solve. There 
is a lot of useful (or in some cases essential) information on the wiki 
for developers (and also for users), and finding yourself too often when 
you need to read something there (or contribute to some page) that 
doesn't load or takes a very long time is counterproductive.


Trying to help new contributors by pointing them in many cases to 
information on the wiki a

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Jonas Smedegaard]

Hi Fabio,

Quoting Fabio Fantoni (2024-08-02 14:31:04)
> One particular thing noticed in some cases (and I hope they are not 
> many) is the lack of use or especially updates of the Vcs-* fields in 
> d/control. I think is important point to packaging repository from the 
> tracker if present and to update it if moved, this can help 
> collaboration and reduce possible waste of time doing things that are 
> perhaps already done by others (or WIP) but which have not yet been 
> uploaded.

That's tracked as OLD key at https://qa.debian.org/cgi-bin/vcswatch -
also included in the Maintainer Dashboard for each package at
https://udd.debian.org/dmd.cgi

Whenever you stumble upon a package misaligned with its declared Vcs
metadata, please file a bugreport, and while doing so consider
encouraging the maintainer to use the Maintainer Dashboard.


> I think that both email and systems like salsa/github/gitlab etc. are 
> useful, both with pros and cons. Forcing people to use only one or the 
> other could be counterproductive at the moment. One thing that I think 
> would be useful is to have certain, fast and simple information for each 
> package about which actual collaboration methods it uses and prefers.
> 
> For example seeing a package it would be very useful to know immediately 
> if it wants a collaboration only through bugtracker and patch, only 
> through vcs or both are accepted. If managed in a team point more easily 
> to information about the team and any pages with details (for example on 
> wiki).

I guess that you mean something like this:
```patch
--- a/debian/control
+++ b/debian/control
@@ -60,6 +60,7 @@ Standards-Version: 4.7.0
 Homepage: https://github.com/oxigraph/oxigraph
 Vcs-Git: https://salsa.debian.org/debian/oxigraph.git
 Vcs-Browser: https://salsa.debian.org/debian/oxigraph
+Contact: https://bugs.debian.org/oxigraph
 Rules-Requires-Root: no

 Package: oxigraph
```

In principle I find that an interesting suggestion, as I can imagine
such information being helpful in understanding if debbugs is used only
by "dinosaurs".

I see two problems, however:

a) Maintainers will be bothered to add that new field to every package
(or at least a substantial subset) for it to be of practical use.

b) Which other answers exist for that field?  I mean, is it ok in Debian
for a maintainer to not use Debbugs?  Is it ok in Debian for a
maintainer to also use another issue tracker and not replicate whatever
is collected elsewhere in Debbugs?

Or perhaps I am missing the point of your suggestion, and you do not
mean where *bug* chatter goes, but instead where *non-bug* conversations
go.  If that is the case, then I believe we have a field already for
that, called "Maintainer:".  If you mean neither issue tracking nor the
main contact information for a package, then please elaborate what you
had in mind, because that's pretty essential to get clarified before
discussing any further...



> Another thing that seems underestimated and I think it is good to 
> emphasize is the excessive slowness of the wiki.debian.org, it seems 
> much worse than salsa to me, and I think it's important to solve.

I don't think that the speed of wiki affects the use of Salsa.

Please stick to the topic - i.e. if you want to shift to another topic
then do so explicitly by changing the Subject: line in your email post.


> If someone thinks that these speed/reachabilityare not important because 
> they are used to even longer times for many operations, for example 
> regarding the bugtracker, tracker, upload etc., unfortunately we live in 
> an increasingly frenetic and continuously worsening world (this world 
> causes more and more stress and less and less time available).

If you mean psychological stress, then I find your reasoning flawed, as
that is about a *perceived* lack of time (or other pressure), not
necessarily actual lack of it.  Possibly but not necessarily.

If you mean stress on the Earth, then I find it highly unlikely that we
can solve that crisis by speeding things up.  Instead we need to find
ways to *reduce* the *amount* of computing we do, and find ways to time
that computing in ways that fit more optimally with the availability of
needed resources (e.g. consume less electricity when there is less wind
and sunny, to reduce stress on green parts of related electricity grid).

Please elaborate what kind of stress you are really talking about, and
again please consider the topic of the conversation (see above about
changing Subject: line).

Thanks for your inspiring input,

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Otto Kekäläinen]

Hi!

On Fri, 2 Aug 2024 at 02:27, Andrea Pappacoda  wrote:
..
> Before a certain way of doing things can be mandated or "warmly
> recommended", the technology has to be as flawless as possible - and
> today I wouldn't call Salsa "flawless", would you? Issues with
> Salsa/GitLab:
>
> 1. It is so slow that it makes me want to close by browser and do
> something else instead
> 2. It doesn't support email-based workflows
> 3. It has a lot of features we don't need. Or rather, it has more
> features we *don't* need than we do.
> 4. Its user interface is confusing, it's often hard to find what I'm
> looking for
> 5. Did I mention how slow it is?
> 6. It is developed by a company who's philosophy and interests are
> misaligned with Debian's
> 7. The product it tries to mimic, GitHub, is also misaligned with
> Debian's philosophy and interests
>
> While performance is something we could reasonably do something about,
> all the other points are part of GitLab by design, and we're stuck with
> them.

GitLab of course has more features than what we need; it does not mean
it has to be used. The basic feature set in GitLab to browse
repositories, show and compare history, show CI status, list MRs,
separate drafts and ready ones, show approvals, show assignees etc
work well. GitHub has done many things well and trying to compete with
them is a good thing. Both GitLab and GitHub also support email
workflows to some degree (but of course most people use them via the
UI as the UI offers many things email does not and never will).

I agree that Salsa is sometimes a bit sluggish
(https://salsa.debian.org/salsa/support/-/issues/395), but I hope we
can do something to improve it and it is now a permanent reason to not
use Salsa.

...
> But the point isn't that much about being able to use emails
> specifically, it is about having the possibility of having a software
> forge which is interoperable with a wide range of tools and can stay
> out of the way if wanted. With GitLab, you either use the full package
> (which includes browsing, reviewing, and commenting) or you get a
> sub-optimal experience. But it doesn't have to be this way.
>
> I'll now talk a bit about SourceHut. Not to suggest that we should use
> that instead, but just to point out how things *can* be done.

I have a paid SourceHut account and have been testing it for some
time. I can see the appeal of simplicity, but for actual team work it
is *too* simple. It will probably take a couple of years for it to
mature.

Note that a DEP is not a vehicle to drive out new version control
systems or source forges. It is a vehicle to formalize something that
is already popular as an official best practice. The salsa.debian.org
is what we have today, and Debian has been using for a 5+ years. If
some day in the future something vastly superior to git or GitLab
surfaces and a lot of people start using it, the DEP could be updated.
But I would not today recommend new Debian maintainers to host on
SourceHut nor GitHub nor self-host. They can mirror there if they
want, but for collaboration prior to uploading to work well the source
code should be on salsa.debian.org for now.

When I today reviewed recent submissions on mentors.debian.net, most
of them were hosted on Salsa, but 4 on GitHub and 1 in nowhere in git
at all. As it currently stands, there is no official document I could
lean on to ask the submitters to please use salsa.debian.org instead.
This is one of the motivations for this DEP.

Re: Produce extra build artifacts during package builds

[Niels Thykier]

Simon McVittie:

On Fri, 02 Aug 2024 at 16:40:15 +0900, Mattia Rizzolo wrote:

[...]



  * decide on a directory name (`./debian/export_artifacts/`?)
  * the build process will dump any files that could be interesting to
export in there (no decision if/how to define the structure within
this directory)
* potentially, all `dh_*` could copy there whatever is relevant to
  them (i.e., `dh_auto_configure` could export configure logs,
  `dh_auto_test` the various test logs, ..)


My concern about this, and the reason my prototype doesn't work this way,
is what I said in
.

The artifacts that I'm primarily interested in are the results of failed
testing, because at the moment librsvg and gtk4 have to exfiltrate
their failed test results (some of which are in the form of images,
because these are GUI libraries) by uuencoding PNG files and writing
the result to the log, and that's both inconvenient and inefficient.

But, when a test has failed, writing arbitrary imperative code to gather
up the package-specific results (for example the images that were generated
by the gtk4 test suite) is extra complexity for the package maintainer,
which can have bugs, and is rarely tested because hopefully the tests
usually pass. I dislike that as a pattern, and I'd strongly prefer a
declarative syntax of some sort.

(Take a look at all the ad-hoc scripting that src:gtk4 has around what
to do after test failures and I think you'll see what I mean!)



I agree with Simon on this part. This logic will rarely be tested by the 
maintainer, so we should minimize the amount of code a maintainer should 
do for this.


Thanks to previous design work, I think this problem is now largely down 
to various tool writers (pbuilder/sbuilder + the build helper tooling) 
doing a prototype of it and letting it loose.



In the design that I prototyped, it's declarative, loosely inspired by
the equivalent Gitlab-CI feature:

- the maintainer can write patterns into debian/build-artifacts for
   package-specific quirks like GTK's reftests
   (#-prefixed comments are allowed)


I am a bit tempted to keep this one out of the spec and leave it to the 
build-helper to define it. I find that files in `debian/` are not very 
discoverable and they are harder to support properly in tooling like 
`debputy lsp server`. Each file format requires extra setup and there is 
not a good way to announce they exist unlike fields in `debian/control`. 
 For `debputy` based packages, I would prefer to have this in the 
`debian/debputy.manifest`, since that will enable me to guide the 
packager to its existence and provide on-line documentation for it.



- tools like debhelper can append patterns to debian/extra-build-artifacts,
   for example dh_auto_* in Meson mode should append "obj-*/meson-logs/*"


As a minor detail, I would make this a .d directory where tools can 
write files in. This avoids issues with concurrency control. It is 
possible to run `dh_auto_configure` in parallel. That is not the only 
way concurrency issues could happen and I am already not excited about 
solving concurrency issues even assuming debhelper is the only source of 
this problem (hint: it won't be). Therefore, a directory and tools using 
random generated filenames. Be that standard `mktemp -p $DIR` or even 
just uuidv4'ing the filename. Either way the concurrency issue would be 
reduced to "an already solved problem".


Secondly, I would want clarity on how this directory is created and 
cleaned up. Personally, I do non-chroot host builds all the time for 
rapid development and I would mind the clutter if every build leaves 
this directory behind - which realistically will end up in the 
`.debian.tar.gz` at some point if not cleaned by the build process 
itself. At the same time, I quite understand always wanting the 
directory for CI/buildd builds.
  An alternative here is that the build tool exports an ENV variable 
for the directory to put these "extra-build-artifacts" into and is then 
responsible for cleaning it up. Since the ENV var would only be exported 
when you wrap your build in sbuild/pbuilder, the clutter would not occur.


However, I think further detailing on this part is between me and 
whoever is going to do it on the pbuilder/sbuild side.



- the buildd operator can configure $artifact_patterns in .sbuildrc if
   they want to (I'm unsure how useful this genuinely is, but maintainers
   running sbuild locally might find it more convenient than editing the
   source package during ad-hoc debugging)
- the patterns reuse machine-readable debian/copyright syntax
- as a result the -artifacts.tar.gz is always a subset of the build tree,
   and inherits its layout, same as Gitlab's artifacts.zip
- if a sufficiently desperate maintainer really wants to, they can tar up
   the entire build tree by specifying a broad enough pattern like '*',
   although this should be discouraged when building 

Re: Produce extra build artifacts during package builds

[Simon McVittie]

On Fri, 02 Aug 2024 at 18:17:52 +0200, Niels Thykier wrote:
> Simon McVittie:
> > In the design that I prototyped, it's declarative, loosely inspired by
> > the equivalent Gitlab-CI feature:
> > 
> > - the maintainer can write patterns into debian/build-artifacts for
> >package-specific quirks like GTK's reftests
> >(#-prefixed comments are allowed)
> 
> I am a bit tempted to keep this one out of the spec and leave it to the
> build-helper to define it. I find that files in `debian/` are not very
> discoverable and they are harder to support properly in tooling like
> `debputy lsp server`. Each file format requires extra setup and there is not
> a good way to announce they exist unlike fields in `debian/control`.  For
> `debputy` based packages, I would prefer to have this in the
> `debian/debputy.manifest`, since that will enable me to guide the packager
> to its existence and provide on-line documentation for it.

That's fine, debputy could read a list of patterns out of its own input
file and "register" them via whatever mechanism exists
(debian/extra-build-artifacts in my prototype, or writing them into an
"artifacts patterns" directory in your suggestion).

I wanted there to be a straightforward way for a maintainer to opt-in
to this, even if they are not yet living in the debputy future, because
I'd prefer to be able to adopt useful features one at a time, rather
than having to rewrite key packages' build systems in debputy syntax
as a blocker for finding out why their tests fail. But in an imperative
(dh-style) build system, that could be something more like:

ifneq ($(PUT_ARTIFACT_PATTERNS_HERE),)
execute_before_dh_auto_configure:
cp debian/build-artifacts $(PUT_ARTIFACT_PATTERNS_HERE)/
endif

and that's not so bad.

Or, is it possible to have a debputy manifest exist, and get the
information in it about artifacts copied out appropriately, without
needing to make debputy drive the entire build?

> Mattia's wiki page does list "A reproducible
> compiler error (ICE) produces a /tmp/cc*.out containing the preprocessed
> source.", which might need some tweaking compared to the "the
> -artifacts.tar.gz is always a subset of the build tree".

Hmm, yes, that's unfortunate. This wouldn't be compatible with the
Gitlab-CI feature that I was inspired by, either.

smcv

Re: Produce extra build artifacts during package builds

[Niels Thykier]

Simon McVittie:

On Fri, 02 Aug 2024 at 18:17:52 +0200, Niels Thykier wrote:

Simon McVittie:

In the design that I prototyped, it's declarative, loosely inspired by
the equivalent Gitlab-CI feature:

- the maintainer can write patterns into debian/build-artifacts for
package-specific quirks like GTK's reftests
(#-prefixed comments are allowed)


I am a bit tempted to keep this one out of the spec and leave it to the
build-helper to define it. I find that files in `debian/` are not very
discoverable and they are harder to support properly in tooling like
`debputy lsp server`. Each file format requires extra setup and there is not
a good way to announce they exist unlike fields in `debian/control`.  For
`debputy` based packages, I would prefer to have this in the
`debian/debputy.manifest`, since that will enable me to guide the packager
to its existence and provide on-line documentation for it.


That's fine, debputy could read a list of patterns out of its own input
file and "register" them via whatever mechanism exists
(debian/extra-build-artifacts in my prototype, or writing them into an
"artifacts patterns" directory in your suggestion).

I wanted there to be a straightforward way for a maintainer to opt-in
to this, even if they are not yet living in the debputy future, because
I'd prefer to be able to adopt useful features one at a time, rather
than having to rewrite key packages' build systems in debputy syntax
as a blocker for finding out why their tests fail.  [...]



My argument was that `debhelper` and `debputy` would each define how 
they provide this functionality to their consumers in a way that fits 
how their style. Not that `debputy` was going to be mandatory for this 
feature.




Or, is it possible to have a debputy manifest exist, and get the
information in it about artifacts copied out appropriately, without
needing to make debputy drive the entire build?



The `debputy` framework already has two "integration modes" where `dh` 
drives the build. As an example, the `dh-sequence-zz-debputy-rrr` 
build-dependency is useful for getting rid of implicit `fakeroot` / 
`Rules-Requires-Root: binary-targets` for static ownership while keeping 
*most* things as you know them.


While adding a third mode would be possible at this junction, I would 
not go down this route for the `debhelper` support as it would imply an 
extra dependency for `debhelper` (a circular one at that) plus it would 
not fit the usual `debhelper` pattern.



Mattia's wiki page does list "A reproducible
compiler error (ICE) produces a /tmp/cc*.out containing the preprocessed
source.", which might need some tweaking compared to the "the
-artifacts.tar.gz is always a subset of the build tree".


Hmm, yes, that's unfortunate. This wouldn't be compatible with the
Gitlab-CI feature that I was inspired by, either.

 smcv



Indeed. But as said, I am pretty sure we can figure out a solution to 
this problem. I think it is more of a question whether it will be 
supported initially (might require a per-source TMPDIR too for buildd 
support, so artifacts does not get tainted because the buildd was 
running two builds at the same time, where once had ICE errors and the 
other had not). It will be fun to solve, I am sure. :D


Best regards,
Niels

PS: If anyone is maintaining a `Rules-Requires-Root: binary-targets` 
package and want to migrate away from the implicit `fakeroot` 
build-dependency, please let me know. I am happy to look at providing 
patches (`debputy` is only required for static ownership though. Often 
it is just a matter of masking `chown/chgrp` or `-o` + `-g` for `install`).

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Fabio Fantoni]

Il 02/08/2024 15:49, Jonas Smedegaard ha scritto:

Hi Fabio,

Quoting Fabio Fantoni (2024-08-02 14:31:04)

One particular thing noticed in some cases (and I hope they are not
many) is the lack of use or especially updates of the Vcs-* fields in
d/control. I think is important point to packaging repository from the
tracker if present and to update it if moved, this can help
collaboration and reduce possible waste of time doing things that are
perhaps already done by others (or WIP) but which have not yet been
uploaded.

That's tracked as OLD key at https://qa.debian.org/cgi-bin/vcswatch -
also included in the Maintainer Dashboard for each package at
https://udd.debian.org/dmd.cgi

Whenever you stumble upon a package misaligned with its declared Vcs
metadata, please file a bugreport, and while doing so consider
encouraging the maintainer to use the Maintainer Dashboard.


Thanks for your reply, I know about there and is useful for found vcs 
not working but vcs working/reachable but no longer used, in favor of 
something else they are not identifiable.


The system is only able to notify partially for those not updated but it 
is not possible to identify if you are working on another repository it 
is not identifiable but it will be only if the vcs field will be 
manually changed, so it is up to the maintainer to change it, in 
addition there is a slight problem that it changes only with a new 
upload, if a lot of time were to pass and a lot of work was done during 
it would not be possible to notice immediately. In some cases you could 
move the repository or notify the move inside but unfortunately in the 
case of repositories on which you do not have permissions (for example 
in abandoned packages and someone is about to adopt or recover them). 
They seem quite rare but unfortunately they happen. Even if it was 
recommended to keep them updated and if it changed update them 
immediately doing a new upload just for that doesn't seem like the best 
to me, could it be useful to have another way, or is there already one? 
anyway this is just a minor thing and even if it were to improve it 
would have little influence compared to what I suppose is what has the 
most influence






I think that both email and systems like salsa/github/gitlab etc. are
useful, both with pros and cons. Forcing people to use only one or the
other could be counterproductive at the moment. One thing that I think
would be useful is to have certain, fast and simple information for each
package about which actual collaboration methods it uses and prefers.

For example seeing a package it would be very useful to know immediately
if it wants a collaboration only through bugtracker and patch, only
through vcs or both are accepted. If managed in a team point more easily
to information about the team and any pages with details (for example on
wiki).

I guess that you mean something like this:
```patch
--- a/debian/control
+++ b/debian/control
@@ -60,6 +60,7 @@ Standards-Version: 4.7.0
  Homepage: https://github.com/oxigraph/oxigraph
  Vcs-Git: https://salsa.debian.org/debian/oxigraph.git
  Vcs-Browser: https://salsa.debian.org/debian/oxigraph
+Contact: https://bugs.debian.org/oxigraph
  Rules-Requires-Root: no

  Package: oxigraph
```

In principle I find that an interesting suggestion, as I can imagine
such information being helpful in understanding if debbugs is used only
by "dinosaurs".

I see two problems, however:

a) Maintainers will be bothered to add that new field to every package
(or at least a substantial subset) for it to be of practical use.

b) Which other answers exist for that field?  I mean, is it ok in Debian
for a maintainer to not use Debbugs?  Is it ok in Debian for a
maintainer to also use another issue tracker and not replicate whatever
is collected elsewhere in Debbugs?

Or perhaps I am missing the point of your suggestion, and you do not
mean where *bug* chatter goes, but instead where *non-bug* conversations
go.  If that is the case, then I believe we have a field already for
that, called "Maintainer:".  If you mean neither issue tracking nor the
main contact information for a package, then please elaborate what you
had in mind, because that's pretty essential to get clarified before
discussing any further...



contact field don't exist now right? even if the contact field maybe 
doesn't give you an idea, maybe something like "contributing" that links 
to the bugtracker, to the repository MR/PR or a wiki page or site with 
any details (especially in case of a group)


how to do it on a technical level, inside debian/control or in another 
way if it was better I don't think it would be a problem, to not disturb 
the maintainers too much if you add something you can only put it 
recommended and not mandatory.


maybe I'm not good at explaining myself, I think the problem is that if 
a contributor, maybe even new or who wants to make even occasional 
contributions on specific packages is not able to find in a simple

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Otto Kekäläinen (2024-08-02 17:23:51)
> I agree that Salsa is sometimes a bit sluggish
> (https://salsa.debian.org/salsa/support/-/issues/395),

what kind of hardware do you have? For people like me who are on slower
hardware, the web experience is absolutely not funny and "a bit sluggish"
doesn't begin to describe it. It is hard to find any other website I'm visiting
that is slower than gitlab. For example, when I run this on my Firefox I get a
score of 1.53: https://browserbench.org/Speedometer3.0/#summary

On an intel core i7 1280P you will get around 7. If the result on your machine
is towards the latter instead of the former, then I understand how you would
describe the gitlab experience only as "a bit sluggish" instead of "atrocious".

> but I hope we can do something to improve it and it is now a permanent reason
> to not use Salsa.

You now said "I hope we can do something to improve it and it is now a
permanent reason to not use Salsa." twice:

https://lists.debian.org/caou6tadwmnc__rvqpob7a1+zyysngotfiq4i+69hhpatfxe...@mail.gmail.com

Did you typo it twice or do you actually mean that it's now a permanent reason
to not use salsa?

I am not suggesting salsa use because I think it's the best thing since the
invention of sliced bread. But personally, I rather use something suboptimal if
it means that we can more or less agree on a "default" and I think that the
current situation (submission of patches by mail and the debian archive is the
bts) is deterring to newcomers. I know that most people probably have faster
machines than I.  As it was pointed out elsewhere in this thread, Debian work
already implies a lot more waiting than "a few seconds" for salsa to finish
loading a page or finish yet another animation, so meh. With the glab tool I
think I can be happy enough.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Jonas Smedegaard]

Quoting Fabio Fantoni (2024-08-02 23:51:26)
> Il 02/08/2024 15:49, Jonas Smedegaard ha scritto:
> >> I think that both email and systems like salsa/github/gitlab etc. are
> >> useful, both with pros and cons. Forcing people to use only one or the
> >> other could be counterproductive at the moment. One thing that I think
> >> would be useful is to have certain, fast and simple information for each
> >> package about which actual collaboration methods it uses and prefers.
> >>
> >> For example seeing a package it would be very useful to know immediately
> >> if it wants a collaboration only through bugtracker and patch, only
> >> through vcs or both are accepted. If managed in a team point more easily
> >> to information about the team and any pages with details (for example on
> >> wiki).
> > I guess that you mean something like this:
> > ```patch
> > --- a/debian/control
> > +++ b/debian/control
> > @@ -60,6 +60,7 @@ Standards-Version: 4.7.0
> >   Homepage: https://github.com/oxigraph/oxigraph
> >   Vcs-Git: https://salsa.debian.org/debian/oxigraph.git
> >   Vcs-Browser: https://salsa.debian.org/debian/oxigraph
> > +Contact: https://bugs.debian.org/oxigraph
> >   Rules-Requires-Root: no
> >
> >   Package: oxigraph
> > ```
> >
> > In principle I find that an interesting suggestion, as I can imagine
> > such information being helpful in understanding if debbugs is used only
> > by "dinosaurs".
> >
> > I see two problems, however:
> >
> > a) Maintainers will be bothered to add that new field to every package
> > (or at least a substantial subset) for it to be of practical use.
> >
> > b) Which other answers exist for that field?  I mean, is it ok in Debian
> > for a maintainer to not use Debbugs?  Is it ok in Debian for a
> > maintainer to also use another issue tracker and not replicate whatever
> > is collected elsewhere in Debbugs?
> >
> > Or perhaps I am missing the point of your suggestion, and you do not
> > mean where *bug* chatter goes, but instead where *non-bug* conversations
> > go.  If that is the case, then I believe we have a field already for
> > that, called "Maintainer:".  If you mean neither issue tracking nor the
> > main contact information for a package, then please elaborate what you
> > had in mind, because that's pretty essential to get clarified before
> > discussing any further...
> 
> 
> contact field don't exist now right? even if the contact field maybe 
> doesn't give you an idea, maybe something like "contributing" that links 
> to the bugtracker, to the repository MR/PR or a wiki page or site with 
> any details (especially in case of a group)

Yes, the contact field exists:
https://www.debian.org/doc/debian-policy/ch-controlfields.html#maintainer


> maybe I'm not good at explaining myself, I think the problem is that if 
> a contributor, maybe even new or who wants to make even occasional 
> contributions on specific packages is not able to find in a simple and 
> fast way about the package on which he wants to contribute as he should 
> (or is better to do), in some cases you can understand in short time by 
> looking a bit at the bugtracker and the vcs while, looking for any wiki 
> pages if he is part of a team but in many cases it is not simple/fast to 
> understand how he should contribute for that package in order to get the 
> best result. using patches on bugtracker or vcs (like salsa) is just one 
> part, then there could be more
> 
> you could say to force everyone to use the same system, in theory it 
> would solve the problem, but in practice I find it difficult and perhaps 
> more harmful. I try to summarize: the contributors are people and not 
> machines, moreover many do it as a passion in a little free time and not 
> as if it were a job.

We all do it as a passion in little free time and not if it were a job.
Also Maintainers.  What is the point of introducing additional ways to
interact with maintainers than the ones that already exist and (as I
understand it) is mandatory: A general contact *email* address, and tied
to that a connection to the *Debbugs* issue tracker.

If a new contributor is unable to contact a package maintainer through
that main contact address, then I am concerned about their ability to
help out.  Don't get me wrong, I do understand how some people can be
excellent coders or testers or translators without ever using email, but
how can those who are fluent in Gitlab but unable to send and receive
emails avoid being a burden in their offers for help to a software
distribution which at its very core is email-based?

As I see it, those maintainers in Debian - single persons or persons
part of a larger team, who chooses to not only handle the mandatory
communication channel of Debian, which is email, are free to act as
proxies for communication at Gitlab, Matrix and whatever else, as long
as they ensure that the communication is proxied (e.g. summarized) to
the mandatory communication channels of Debian.

It feels backwards to me to start 

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Otto Kekäläinen]

Hi,

On Fri, 2 Aug 2024 at 16:27, Johannes Schauer Marin Rodrigues
 wrote:
> Quoting Otto Kekäläinen (2024-08-02 17:23:51)
> > I agree that Salsa is sometimes a bit sluggish
> > (https://salsa.debian.org/salsa/support/-/issues/395),
>
> what kind of hardware do you have? For people like me who are on slower

I have a 4-year old Dell XPS 13 (that came with Ubuntu preinstalled
and has perfect Linux support, great laptop by the way).

> hardware, the web experience is absolutely not funny and "a bit sluggish"
> doesn't begin to describe it. It is hard to find any other website I'm 
> visiting
> that is slower than gitlab. For example, when I run this on my Firefox I get a
> score of 1.53: https://browserbench.org/Speedometer3.0/#summary

I got 9.25.

Some of the GitLab slowness is due to the server side, and some due to
JavaScript. I prefer to use mild terms "a bit sluggish" out of empathy
to the people who maintain Salsa, as it is probably not fun for them
to read too vocal complaints after the incredible amount of work they
have put in to get us this far. I am very grateful for their work and
with the language in the bug report I try to be constructive on what
things to prioritize next.

...
> You now said "I hope we can do something to improve it and it is now a
> permanent reason to not use Salsa." twice:
>
> Did you typo it twice or do you actually mean that it's now a permanent reason
> to not use salsa?

Thanks for pointing out the typo. I meant 'not a permanent reason'. I
typo now/not frequently for some odd reason and since both words are
in the dictionary, my spell checker does not flag it.

> I am not suggesting salsa use because I think it's the best thing since the
> invention of sliced bread. But personally, I rather use something suboptimal 
> if
> it means that we can more or less agree on a "default" and I think that the
> current situation (submission of patches by mail and the debian archive is the
> bts) is deterring to newcomers. I know that most people probably have faster
> machines than I.  As it was pointed out elsewhere in this thread, Debian work
> already implies a lot more waiting than "a few seconds" for salsa to finish
> loading a page or finish yet another animation, so meh. With the glab tool I
> think I can be happy enough.

This I think summarizes well the opinion of most people I have spoken
with. GitLab is not perfect, but it was chosen and we have been using
it for 5+ years mostly successfully. Most packages are there and we
should state that it is the recommended option. Currently the second
most popular option is to use GitHub, or not use any vcs at all.

A lot of this thread has gone into people expressing their dislike for
Salsa. Most people still use Salsa - I guess the silent mass isn't
chiming in in this thread that much now. Some people probably have
very optimized email workflows, and nobody can argue against the
statement that a pure email workflow for sure is simple, and has its
elegance. But it also has shortcomings such as no lack of
metadata/status, no built-in way to run CI and share the code+logs+CI
results etc.

Following the principles 1 (use git) and 2 (use Salsa) allows for the
next principles 3, 4 and 5 to take place. I would be curious to hear
more views about them.

DEP-18 summary (https://salsa.debian.org/dep-team/deps/-/merge_requests/8):
1. Have package source code in version control, using git
2. Host package source on Debian's code forge Salsa
3. Run Salsa CI at least once before every upload to ensure minimal
level of quality
4. Allow Merge Requests to be submitted
5. Allow changes to be reviewed before uploads to Debian

My plan is to summarize the discussion and update the proposal in the
coming days, incorporating as many views as possible. I will also in
the next update include additional relevant context info such as
tag2upload, glab and examples of how to easily work with both Debian
bug tracker and MRs in parallel.

Please share your views, and also +1 or -1 the proposal on Salsa so we
can incorporate that too in measuring the support of this.

Remember that DEPs are soft rules - unlike General Resolutions (GRs) -
and maintainers who have specific reasons to not  want to use git or
Salsa will not be forced to do so.

Re: Produce extra build artifacts during package builds

[Mattia Rizzolo]

On Fri, Aug 02, 2024 at 11:01:24AM +0100, Simon McVittie wrote:
> I also did some more concrete design and wrote a prototype -
>  as revised
> in , and
>  -
> although I never got as far as doing enough testing and polishing to get
> it to a production-ready status. I'm sorry that this did not stay as my
> top priority, but I can only have one top priority at a time.

Right, I read most of the emails in that thread and I did look at the
MR.
Unfortunately, I did so _after_ the BOF :\

> The artifacts that I'm primarily interested in are the results of failed
> testing

Right, that's also what we thought.

> But, when a test has failed, writing arbitrary imperative code to gather
> up the package-specific results (for example the images that were generated
> by the gtk4 test suite) is extra complexity for the package maintainer,
> which can have bugs, and is rarely tested because hopefully the tests
> usually pass. I dislike that as a pattern, and I'd strongly prefer a
> declarative syntax of some sort.

Consider that in my mind, most packages wouldn't have needed to write
any imperative code.
I was considering it for example within dh_auto_configure and
dh_auto_test, etc, those tools would be responsible for copying the
relevant test files into my proposed directory.

> In the design that I prototyped, it's declarative, loosely inspired by
> the equivalent Gitlab-CI feature:
> 
> - the maintainer can write patterns into debian/build-artifacts for
>   package-specific quirks like GTK's reftests
>   (#-prefixed comments are allowed)
> - tools like debhelper can append patterns to debian/extra-build-artifacts,
>   for example dh_auto_* in Meson mode should append "obj-*/meson-logs/*"

Well, I don't use debputy nor really understand it, so I don't
understand Niels' concerns.
But I could accept a file debian/extra-artifacts (for
maintainer-specified globs) and directory debian/extra-artifacts.d/
(for tools-provided globs).  dh_auto_clean should (eventually) learn to
clean this directory.

And I suspect that packages that want to be dynamic in the content of
this archive can simply `echo path/to/stuff >>
debian/extra-artifacts.d/package` during the build instead of my
proposed `cp`.


Just, if we do we need to keep this as simple as possible as multiple
tools will try to parse it.  I'd be ok with a plan list, at most accept
# at the start of a line for comments and empty lines.

> - the patterns reuse machine-readable debian/copyright syntax
> - as a result the -artifacts.tar.gz is always a subset of the build tree,
>   and inherits its layout, same as Gitlab's artifacts.zip

This might not work as we want to be able to collect, for example, stuff
from /tmp/.
OTOH, this can be handled by doing proper subfolders in the tarball.

> and sbuild (or pbuilder if you prefer) is responsible for enumerating files
> matching the given patterns and packing them into a tarball.

I was trying to have something else do the enumeration and have
sbuild/pbuilder only do the packing, but I can be convinced in doing the
enumeration sure.

> >* place the tar'ed-up artifacts in the usual results directory, with
> >  a predictive name (`$srcpkg_$ver_$arch.artifacts.tar.xz`?)
> 
> I prototyped this as the same name as the log file, except replacing the
> .build extension with -artifacts.tar.gz (so the filename has a timestamp
> in it). My rationale for this choice is that when tests fail, it's common
> to retry the build to see whether the tests are reproducibly failing or
> just flaky, so we want to collect one blob of artifacts per attempt, the
> same way we collect one log per attempt.
> 
> I don't know how the layers "above" sbuild such as buildd and wanna-build
> operate, and I don't have a way to test them, so my prototype stops at
> sbuild.

This can be left as an implementation detail of the build runner
(most likely, both sbuild and pbuilder will obtain a configuration knob
for this filename).

> > Matthias Klose would like to find a way to do something that at the end
> > of a build will look for and collect all the relevant files that are
> > produced during an ICE (for example all the pre-processor inputs, etc).
> > I'm not sure how to best solve it, but he was imagining a hook somewhere
> > (in dpkg?) that would look for them and copy them into the directory.
> 
> Perhaps dpkg or dh_auto_build could append appropriate patterns to my
> debian/extra-build-artifacts on build failure, so that this hook would
> just be acting as an extension to the declarative mechanism?

Possibly.  I suppose there could be a tool developed later that can be
used to search for any relevant file.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org 

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Jonas Smedegaard]

Quoting Otto Kekäläinen (2024-08-03 06:38:38)
> > I am not suggesting salsa use because I think it's the best thing since the
> > invention of sliced bread. But personally, I rather use something 
> > suboptimal if
> > it means that we can more or less agree on a "default" and I think that the
> > current situation (submission of patches by mail and the debian archive is 
> > the
> > bts) is deterring to newcomers. I know that most people probably have faster
> > machines than I.  As it was pointed out elsewhere in this thread, Debian 
> > work
> > already implies a lot more waiting than "a few seconds" for salsa to finish
> > loading a page or finish yet another animation, so meh. With the glab tool I
> > think I can be happy enough.
> 
> This I think summarizes well the opinion of most people I have spoken
> with. GitLab is not perfect, but it was chosen and we have been using
> it for 5+ years mostly successfully. Most packages are there and we
> should state that it is the recommended option. Currently the second
> most popular option is to use GitHub, or not use any vcs at all.
> 
> A lot of this thread has gone into people expressing their dislike for
> Salsa. Most people still use Salsa - I guess the silent mass isn't
> chiming in in this thread that much now. Some people probably have
> very optimized email workflows, and nobody can argue against the
> statement that a pure email workflow for sure is simple, and has its
> elegance. But it also has shortcomings such as no lack of
> metadata/status, no built-in way to run CI and share the code+logs+CI
> results etc.
> 
> Following the principles 1 (use git) and 2 (use Salsa) allows for the
> next principles 3, 4 and 5 to take place. I would be curious to hear
> more views about them.
> 
> DEP-18 summary (https://salsa.debian.org/dep-team/deps/-/merge_requests/8):
> 1. Have package source code in version control, using git
> 2. Host package source on Debian's code forge Salsa
> 3. Run Salsa CI at least once before every upload to ensure minimal
> level of quality
> 4. Allow Merge Requests to be submitted
> 5. Allow changes to be reviewed before uploads to Debian
> 
> My plan is to summarize the discussion and update the proposal in the
> coming days, incorporating as many views as possible. I will also in
> the next update include additional relevant context info such as
> tag2upload, glab and examples of how to easily work with both Debian
> bug tracker and MRs in parallel.
> 
> Please share your views, and also +1 or -1 the proposal on Salsa so we
> can incorporate that too in measuring the support of this.
> 
> Remember that DEPs are soft rules - unlike General Resolutions (GRs) -
> and maintainers who have specific reasons to not  want to use git or
> Salsa will not be forced to do so.

My problem with DEP-18 is that people who have zero problem with using
git and are also not fundamentally against using salsa, but have
reservations surrounding *which parts* of salsa to use and the
consequences for related already used tooling.

I am also not against being welcoming to newcomers, but I am concerned
if the focus on tose unreasonably reshapes the tooling for all of us.

Essentially, my concern is that DEP-18 reduces a spectrum of options to
"either you are for or against true collaboration".

Leaning more on Gitlab is not the "true" choice, but the pragmatic one,
because yes, we have invested in it, and some parts of it might be made
to better use for use.

I imagine that some in the silent crowd hesitate to chime in due to that
lumping together the use of git and the use of Gitlab into an
all-or-nothing choice.  I think you intended that reduction, for the
purpose of simplifying the conversation. I don't think that
simplification is helpfull, however.


 - Jonas


-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Shengjing Zhu]

On Sat, Aug 3, 2024 at 2:26 PM Jonas Smedegaard  wrote:
>
> My problem with DEP-18 is that people who have zero problem with using
> git and are also not fundamentally against using salsa, but have
> reservations surrounding *which parts* of salsa to use and the
> consequences for related already used tooling.
>
> I am also not against being welcoming to newcomers, but I am concerned
> if the focus on tose unreasonably reshapes the tooling for all of us.
>
> Essentially, my concern is that DEP-18 reduces a spectrum of options to
> "either you are for or against true collaboration".
>
> Leaning more on Gitlab is not the "true" choice, but the pragmatic one,
> because yes, we have invested in it, and some parts of it might be made
> to better use for use.
>
> I imagine that some in the silent crowd hesitate to chime in due to that
> lumping together the use of git and the use of Gitlab into an
> all-or-nothing choice.  I think you intended that reduction, for the
> purpose of simplifying the conversation. I don't think that
> simplification is helpfull, however.

+1

I also feel uncomfortable with this proposal that pushes the use of
Gitlab in the name of true collaboration.

-- 
Shengjing Zhu