Re: What to do when DD considers policy to be optional? [kubernetes]

[Holger Levsen]

On Mon, Mar 23, 2020 at 05:32:54PM +1100, Dmitry Smirnov wrote:
> Something interesting just happened. An inexperienced DD adopted a very 
> complicated package (kubernetes) and uploaded it with changes that would have 
> never been accepted by ftp-masters.

file bugs then. maybe that new maintainer will be very happy about them being
pointed out?!

> This is not a technical disagreement so I think that involving technical 
> committee may not be the right way to handle the problem... Or is it?

involving the TC without disagreements documented in the BTS seems premature.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

"... the premise [is] that privacy is about hiding a wrong. It's not.
 Privacy is an inherent human right, and a requirement for maintaining
 the human condition with dignity and respect." (Bruce Schneier)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Paul Wise]

On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:

> Kubernetes is already using Go modules. They happen to have decided to
> keep shipping a `vendor/` directory but this is not uncommon. It is
> often considered as a protection against disappearing modules. So, there
> is nothing to be done upstream. And BTW, there are currently 616
> dependencies, pinned to a specific version.

I wonder if the existence of Software Heritage could convince them
disappearing modules aren't a problem, or if another service is
needed.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

On Mon, 23 Mar 2020 18:47:18 -0700
Sean Whitton  wrote:

> Hello Dmitry, Janos, others,
> 
> On Mon 23 Mar 2020 at 05:32PM +11, Dmitry Smirnov wrote:
> 
> > What would be best to do in such situation?  
>
> [...]
> 
> I think that I would start by filing an RC bug.

+1

If you run into issues, then you'll want to contact the ftp-masters team.

It would be helpful if the bug mentioned the two solutions I'm aware of:
- Revert the offending changes
- Migrate from main to non-free

The latter would be much easier, but would destroy all the work you put in. :(

> > [...]
> > As a person who originally introduced Kubernetes to Debian I can say that it
> > is indeed a lot of work to maintain this package and to reuse packaged
> > libraries. But I've demonstrated that it is possible at least to some 
> > extent.

As a person who temporarily introduced gitea into Debian, I fully understand.
Unfortunately, I've found that such problems are often a result of poorly
written code where the approach tends to be, "I don't know how to do this
thing, so I'll copy a module that does this and 200 other things just as
poorly." 

The lesson I learned was-
Just because something /can/ be packaged, does not mean it /should/ be packaged.
(pabs warned me about this hundreds of hours prior to me giving up)

> > I don't recall a situation when policing of how a package is maintained 
> > would
> > be necessary long after package is accepted...

It's rare, but it happens. My most recent experience was with bitlbee.
Unfortunately, our current auto-reject system is quite limited. Catching things
like this automatically is (currently) not possible and Debian relies of folks
like you. (btw- thanks for this report)

> > What do we do to ensure quality work in situation of technological hijack
> > when maintainer is unwilling to follow the practice or comply with policy?
> >
> > This is not a technical disagreement so I think that involving technical
> > committee may not be the right way to handle the problem... Or is it?  

TC is not needed. This is a clear policy violation that the new maintainer
appears to have known about, even before the upload.

It concerns me that they thought this package warranted an exception...
-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

On Tue, 24 Mar 2020 10:14:08 +
Paul Wise  wrote:

> On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:
> 
> > Kubernetes is already using Go modules. They happen to have decided to
> > keep shipping a `vendor/` directory but this is not uncommon. It is
> > often considered as a protection against disappearing modules. So, there
> > is nothing to be done upstream. And BTW, there are currently 616
> > dependencies, pinned to a specific version.  
> 
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

I think this is a symptom of the tools being used. Using 'go vendor' is a
documented step in nearly all golang-based "release tutorials." Most never even
get as far as considering that maybe their source should have a version,
because the toolset mentality is "download latest at build time."

The 'go vendor' approach is especially bad within the Debian context because it
will download any/all modules that are referenced. In some cases, 'go get [..]'
can go from downloading a single repository to downloading 200+ because one (1)
extra dependency was added for one (1) extra feature that almost nobody will
ever use.

It's nearly guaranteed that at least a large handful of those will have no
license at all and at least one is going to have large embedded non-dfsg blobs.

Or, to summarize my rant...

These lazy young whipper snappers don't know what good source looks like!

.. back in my day, we coded on paper, had real bugs, and that's just the way we
liked it.

-- 
Michael Lustfield

Bug#954841: ITP: wev -- tool for debugging events on a Wayland window

[Birger Schacht]

Package: wnpp
Severity: wishlist
Owner: Birger Schacht 

* Package name: wev
  Version : 1.0.0
  Upstream Author : Drew DeVault
* URL : https://git.sr.ht/~sircmpwn/wev
* License : MIT
  Programming Lang: C
  Description : tool for debugging events on a Wayland window

This is a tool for debugging events on a Wayland window, analagous to
the X11 tool xev.
I plan to maintain this package in the swaywm team.

Bug#954844: ITP: mangohud -- An overlay for monitoring FPS, temperatures, and more

[Stephan Lachnit]

Package: wnpp
Severity: wishlist
Owner: Stephan Lachnit 

  Package name: mangohud
  Version : >0.3.1
  Upstream Author : flightlessmango 
  URL : https://github.com/flightlessmango/MangoHud
  License : MIT
  Programming Lang: C, C++
  Description : An overlay for monitoring FPS, temperatures, and more

A modification of the Mesa Vulkan overlay. Including GUI improvements,
temperature reporting, and logging capabilities. Helpful to analyze performance
in games (works under OpenGL as well).

Building under Debian should works, current version has no release tarball with
the git submodules included, so I'll wait for the next release before starting
a repo on salsa. More on this topic here:
https://github.com/flightlessmango/MangoHud/issues/34

It would be possible to release this under the Debian Games Team, if anyone
there is interested, otherwise I'll maintain it myself.

Best Regards
Stephan

Bug#954847: ITP: setzer -- Simple yet full-featured LaTeX editor

[Stephan Lachnit]

Package: wnpp
Severity: wishlist
Owner: Stephan Lachnit 

  Package name: setzer
  Version : >0.2.1
  Upstream Author : cvfosa 
  URL : https://github.com/cvfosa/Setzer
  License : GPL3+
  Programming Lang: Python with Gtk
  Description : Simple yet full-featured LaTeX editor

Setzer is a LaTeX with very high content-per-window-size ratio, and has a very
clean Gtk interface. It also features side bars with a lot of symbols commonly
used in mathmode, so one doesn't have to look up the name in the internet, and
features spell checking.

The latest release doesn't have an install system yet (local only), but I wired
up some basic meson support, which was merged into master. Still some things to
do, but once that's done it can be easily build in Debian.

Could be maintained with the Debian TeX team if someone is interested there,
otherwise I'll maintain it myself.

Best Regards
Stephan

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 10:14 +00, Paul Wise:

>> Kubernetes is already using Go modules. They happen to have decided to
>> keep shipping a `vendor/` directory but this is not uncommon. It is
>> often considered as a protection against disappearing modules. So, there
>> is nothing to be done upstream. And BTW, there are currently 616
>> dependencies, pinned to a specific version.
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

There are other reasons, notably that you speed up builds by having all
the source code ready. If the vendor/ directory wasn't there, the
presence of `go.sum` would ensure you get something reproducible by
downloading all the deps, but I think it implies a full checkout of all
deps, which can be lengthy. There is also a caching mechanism (local and
remote).
-- 
Make sure every module hides something.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 05:37 -05, Michael Lustfield:

>> > Kubernetes is already using Go modules. They happen to have decided to
>> > keep shipping a `vendor/` directory but this is not uncommon. It is
>> > often considered as a protection against disappearing modules. So, there
>> > is nothing to be done upstream. And BTW, there are currently 616
>> > dependencies, pinned to a specific version.  
>> 
>> I wonder if the existence of Software Heritage could convince them
>> disappearing modules aren't a problem, or if another service is
>> needed.
>
> I think this is a symptom of the tools being used. Using 'go vendor' is a
> documented step in nearly all golang-based "release tutorials." Most never 
> even
> get as far as considering that maybe their source should have a version,
> because the toolset mentality is "download latest at build time."

With Go modules, that's not true anymore. It will use the minimal
version satisfying the minimal versions specified in go.mod.
-- 
Follow each decision as closely as possible with its associated action.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Julien Puydt]

Le mardi 24 mars 2020 à 14:03 +0100, Vincent Bernat a écrit :
>  
> There are other reasons, notably that you speed up builds by having
> all
> the source code ready.

Sorry, I don't know much about how go works, but : can't the developer
just have the deps ready -- and just not commit them to the repo and
not ship them?

>From what I have read in this thread, go developers seem to be about as
sloppy as javascript ones : put junk together and throw it to the
world...

J.Puydt

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 24 mars 2020 14:18 +01, Julien Puydt:

>> There are other reasons, notably that you speed up builds by having
>> all the source code ready.
>
> Sorry, I don't know much about how go works, but : can't the developer
> just have the deps ready -- and just not commit them to the repo and
> not ship them?

These projects heavily rely on automated builds. Depending on platform,
it may or may not be easy to have shared cache for these builds. If they
have, you have to debug them as well. From their point of view, it's
simpler to deliver the artifacts with the rest of the source code.
Fast, simple and more reliable.

Moreover, the vendor directory is absolutely not a problem for Debian
(except for licensing where you would have to do a repack for stuff not
distributable). Debian just has to ignore the vendor directory and have
all the dependencies ready. The tooling around Go in Debian already
handles that. The problem is the number of dependencies.

> From what I have read in this thread, go developers seem to be about as
> sloppy as javascript ones : put junk together and throw it to the
> world...

That's not comparable. Go is not using micro-dependencies. However, they
don't have optional dependencies, so anything cloud-based has a lot of
dependencies to manage.
-- 
Make it clear before you make it faster.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Sean Whitton]

Hello,

On Tue 24 Mar 2020 at 05:37AM -05, Michael Lustfield wrote:

> The 'go vendor' approach is especially bad within the Debian context because 
> it
> will download any/all modules that are referenced. In some cases, 'go get 
> [..]'
> can go from downloading a single repository to downloading 200+ because one 
> (1)
> extra dependency was added for one (1) extra feature that almost nobody will
> ever use.
>
> It's nearly guaranteed that at least a large handful of those will have no
> license at all and at least one is going to have large embedded non-dfsg 
> blobs.
>
> Or, to summarize my rant...
>
> These lazy young whipper snappers don't know what good source looks like!
>
> .. back in my day, we coded on paper, had real bugs, and that's just the way 
> we
> liked it.

If there are multiple DFSG issues then I would say that this is more
than a rant.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: email backend for fedmsg

[Peter Silva]

hi, totally different take on this...

We could talk about the merits of various protocols (I see fedmsg uses
ZeroMQ) but that is a
deep rabbit hole... to me, fedmsg looks like it is making a ZeroMQ version
of a broker (which is a bit ironic given the original point of that
protocol) trying to build a broker ecosystem is hard. Using an existing one
is much easier.  so to me it makes sense that fedmsg is not really working
out.

However,



I work in telecom for meteorology, and we ended up with a general method
for file copying (catchphrase: rsync on steroids*.) ( *every catchphrase is
a distortion, no dis to rsync, but in certain cases we do work much faster,
it just communicates the idea.) Sarracenia (
https://github.com/MetPX/Sarracenia) is a GPL2 app (Python and C
implementations) that use mozilla public license rabbitmq broker, as well
as openssh and/or any web server to do fastish file synching, and/or
processing/orchestration. The app is just json messages with file metadata
sent through the broker. Then you daisy chain brokers through clients.  No
centralization (every entity installs their own broker), No federated
identity required (authentication is to each broker, but they can pass
files/messages to each other.)

A firstish thing to do with it would be to sync the debian mirrors in
real-time rather than periodically.  Each mirror has a broker, they get
advertisements (AMQP messages containing JSON file metadata) download the
corresponding file, and re-advertise (publish on the local broker with the
local file URL) for downstream clients. You can then make a mesh of
mirrors, where, if each mirror is subscribed to at least two others, then
it can withstand the failure of any node.  If you add more connections, you
increase redundancy.

Once you have that sort of anchor tenant for an AMQP message bus, people
might want to use it to provide other forms of automation, but way quicker
and in some ways much simpler than SMTP.  but yeah... SMTP is a lot more
well-known/common. RabbitMQ is the industry dominant open solution for AMQP
brokers. sounds like marketing bs, but if you look around it is what the
vast majority are using, and there are thousands upon thousands of
deployments. It's a much more viable starting point, for stability, and a
lot less assembly required to get something going. Sarracenia makes it a
bit easier again, but messages are kind of alien and different, so it takes
a while to get used to them.




On Sun, Mar 22, 2020 at 8:24 AM clime  wrote:

> Hello!
>
> Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html -
> fedmsg usage in Debian.
>
> There is a note: "it seems that people actually like parsing emails"
>
> What about adding email backend to fedmsg then. Wouldn't it be an
> interesting idea? It could basically rely on postfix for sending
> messages, hence providing decentralization as well as high
> reliability. I think that amount of events that happen in distribution
> (like package update, package build) is never so huge that email
> infrastructure wouldn't handle it and also the machine mailing
> infrastructure could be optionally be separated from the human one if
> needed.
>
> So fedmsg would become a tiny wrapper over email that would just
> serialize and parse json data to and from email messages and check
> signatures.
>
> I am asking because I like the idea of distribution-independent
> infrastructure message bus that this project had.
>
> Btw. instead of json, yaml could be used so it is nicer to human eyes.
>
> clime
>
>

Re: email backend for fedmsg

[Jeremy Stanley]

On 2020-03-24 13:09:35 -0400 (-0400), Peter Silva wrote:
[...]
> We could talk about the merits of various protocols (I see fedmsg
> uses ZeroMQ) but that is a deep rabbit hole... to me, fedmsg looks
> like it is making a ZeroMQ version of a broker (which is a bit
> ironic given the original point of that protocol) trying to build
> a broker ecosystem is hard. Using an existing one is much easier.
> so to me it makes sense that fedmsg is not really working out.
[...]

In the OpenDev collaboratory we added an event stream for our
services some years ago using the MQTT protocol (a long-established
ISO/OASIS standard). I gather there was some work done to make
fedmsg support MQTT as a result of that, so it might be an
alternative to relying on ZeroMQ at least.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: email backend for fedmsg

[Peter Silva]

MQTT is the best thing going for interop purposes.

On Tue, Mar 24, 2020 at 1:20 PM Jeremy Stanley  wrote:

> On 2020-03-24 13:09:35 -0400 (-0400), Peter Silva wrote:
> [...]
> > We could talk about the merits of various protocols (I see fedmsg
> > uses ZeroMQ) but that is a deep rabbit hole... to me, fedmsg looks
> > like it is making a ZeroMQ version of a broker (which is a bit
> > ironic given the original point of that protocol) trying to build
> > a broker ecosystem is hard. Using an existing one is much easier.
> > so to me it makes sense that fedmsg is not really working out.
> [...]
>
> In the OpenDev collaboratory we added an event stream for our
> services some years ago using the MQTT protocol (a long-established
> ISO/OASIS standard). I gather there was some work done to make
> fedmsg support MQTT as a result of that, so it might be an
> alternative to relying on ZeroMQ at least.
> --
> Jeremy Stanley
>

Re: What to do when DD considers policy to be optional? [kubernetes]

[Florian Weimer]

* Paul Wise:

> On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:
>
>> Kubernetes is already using Go modules. They happen to have decided to
>> keep shipping a `vendor/` directory but this is not uncommon. It is
>> often considered as a protection against disappearing modules. So, there
>> is nothing to be done upstream. And BTW, there are currently 616
>> dependencies, pinned to a specific version.
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

The required versions of the modules could disappear from Debian, though.
Services like Software Heritage will not help with that.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Janos LENART]

Hi Dimitry, FTP masters and others,

I know Dimitry was fighting an uphill battle with kubernetes between 2016
and 2018 and he experienced first hand the problems posed by vendored code.

We see more and more software making excessive use of vendored code. Pretty
much everything that is written in Go. Some of these are crucially
important, like Docker or Kubernetes. So I understand the concern everyone
has about how this fits with the Debian Policy.

Debian Policy, paragraph 4.13 states:
(for your convenience I include it below :) )
https://www.debian.org/doc/debian-policy/ch-source.html#convenience-copies-of-code

=
4.13 Convenience copies of code

Some software packages include in their distribution convenience copies of
code from other software packages, generally so that users compiling from
source don’t have to download multiple packages. Debian packages should not
make use of these convenience copies unless the included package is
explicitly intended to be used in this way. [17] If the included code is
already in the Debian archive in the form of a library, the Debian
packaging should ensure that binary packages reference the libraries
already in Debian and the convenience copy is not used. If the included
code is not already in Debian, it should be packaged separately as a
prerequisite if possible. [18]

[18] Having multiple copies of the same code in Debian is inefficient,
often creates either static linking or shared library conflicts, and, most
importantly, increases the difficulty of handling security vulnerabilities
in the duplicated code.
=

I think this is the part that has the most bearing on the vendored code
problem, especially the footnote. I agree with this principle. But we
should apply it to the state of affairs in 2020, and to this specific
situation.

Keeping all that in mind, here are the reasons why I think it is acceptable
for now to package Kubernetes with the vendored code, and even the best
solution that is available currently:

1. OTHER EXAMPLES. If we take this paragraph completely literally and to
the extreme then other packages are also in violation of it. True, the
current packaging of kubernetes does this to a greater extent than its
predecessor for example, but perhaps this shows that this section was
always open for interpretation. Examples of some prominent packages in
Debian that bundle and use the vendored code (in parentheses is the number
of go packages bundled, estimate):
- docker.io (58, including some that are vendored more than once within the
same source package, but not including the fact that docker.io itself is
made up of 7 tarballs)
- kubernetes (20 for the previous version, 200 now)
- prometheus (4)
- golang (4)
None of these were REJECTed, and please don't sabotage these packages now
:-D The idea was only to show that, at least for now, vendoring is a fact
in Debian. There is an effort to improve the situation but in the meantime
we just go on. Not great, not terrible..

2. MAINTAINABILITY. Having every single vendored repo available as a
separate package in Debian is not feasible. It is true that some of them
are already packaged. But the expectation that all of them are (with the
exact version that is needed for Kubernetes), is not going to happen. Also,
the golang-* packages have a number of different maintainers. Hundreds of
such packages would be required to build Kubernetes. So one can be rest
assured that every future release in Debian will be blocked on waiting for
dozens of these packages to be updated. Dimitry and a few others worked
hard on trying to pull this off but even they could not do it. Since 2016 a
total of 3 Kubernetes releases made it into Debian/unstable, but there have
been 17 major and countless minor upstream releases of Kubernetes.
Thousands of issues were fixed upstream, including serious security flaws,
these never made it into Debian. Exactly because the packaging was too
difficult to maintain. So, how maintainable was that solution then, despite
the huge amount of effort put in? In my opinion this shows that the
reasoning on maintainability in DP does not apply here.

3. NO FORKS. Debian developers hacking Kubernetes source code, so it
compiles with a lucky enough version of a dependency that made it into
Debian, makes the Debian version of Kubernetes different from the standard
one that everyone expects. This is totally unwelcome by almost every user.
No sane cluster admin would dare to use this "fork", ever. There were some
attempts to get the Kubernetes contributors to update dependencies to a
specific version: https://github.com/kubernetes/kubernetes/issues/27543 .
Reading the whole thread helps to put some perspective on this. The
Kubernetes contributors were actually quite helpful throughout but they
have made it clear that they will not update dependencies for update's
sake. Maybe with some projects Debian would have the upper hand, but not
with Kubernetes.

4. TESTING. The Kubernetes 

Re: email backend for fedmsg

[Nicolas Dandrimont]

Hi!

On Sun, Mar 22, 2020, at 13:06, clime wrote:
> Hello!
> 
> Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html -
> fedmsg usage in Debian.
> 
> There is a note: "it seems that people actually like parsing emails"

This was just a way to say that fedmsg never got much of a user base in the 
services that run on Debian infra, and that even the new services introduced at 
the time kept parsing emails.

> [...]
>
> So fedmsg would become a tiny wrapper over email that would just
> serialize and parse json data to and from email messages and check
> signatures.

The only native fedmsg producer in Debian was mentors.debian.net. Other events 
were generated by various email parsers connected to mailing lists 
(debian-devel-announce, debian-bugs-announce).

> I am asking because I like the idea of distribution-independent
> infrastructure message bus that this project had.

Yes, it was a nice idea.

> [...]

Cheers,
-- 
Nicolas Dandrimont

Re: What to do when DD considers policy to be optional? [kubernetes]

[Jeremy Stanley]

On 2020-03-24 19:08:23 + (+), Janos LENART wrote:
> I know Dimitry was fighting an uphill battle with kubernetes
> between 2016 and 2018 and he experienced first hand the problems
> posed by vendored code.
> 
> We see more and more software making excessive use of vendored
> code. Pretty much everything that is written in Go. Some of these
> are crucially important, like Docker or Kubernetes. So I
> understand the concern everyone has about how this fits with the
> Debian Policy.
[snip rationale for packaging Kubernetes while giving up on policy]

If this represents the actual state of building Kubernetes, it's
unclear to me why Debian would package it at all. I don't see the
value to users in consuming Kubernetes from a Debian package if the
result is compromising on Debian's vision and values so that they
can get the exact same thing they'd have if they just used the
Kubernetes community's recommended tooling to install it instead.
I'm all for using the best tool for the job, and while I've been a
die-hard Debian user for more than two decades I also don't install
every last bit of software from its packages. Some software
ecosystems have chosen to focus on tools and workflows which are
incompatible with Debian's, but that doesn't mean that either one is
inherently bad nor that they need to be integrated at all costs.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: email backend for fedmsg

[clime]

On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont  wrote:
>
> Hi!
>
> On Sun, Mar 22, 2020, at 13:06, clime wrote:
> > Hello!
> >
> > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html -
> > fedmsg usage in Debian.
> >
> > There is a note: "it seems that people actually like parsing emails"
>
> This was just a way to say that fedmsg never got much of a user base in the 
> services that run on Debian infra, and that even the new services introduced 
> at the time kept parsing emails.

Hello Nicolas!

Do you remember some such service and how it used email parsing specifically?

I am still a bit unclear how email parsing is used in Debian
infrastructure, don't get me wrong, I find it elegant but from what I
have found (e.g. reportbug), in the beginning there is an
email being sent by some human which will then trigger some automatic
action (e.g. putting the bug into db). So it's like you could do all
your work simply by sending emails (some of them machine-parsable).

So do you have the opposite? I do some clicking action somewhere and
it will send an email to a certain mailing list to inform human
beings? Or let's not just clicking but e.g. `git push` (something that
you can still do from command line).

Do you have: I do some clicking action somewhere and it will send an
email to a certain mailing list where the email is afterward parsed by
another service which will do an action (e.g. launch a build) based on
it?

Thanks
clime

>
> > [...]
> >
> > So fedmsg would become a tiny wrapper over email that would just
> > serialize and parse json data to and from email messages and check
> > signatures.
>
> The only native fedmsg producer in Debian was mentors.debian.net. Other 
> events were generated by various email parsers connected to mailing lists 
> (debian-devel-announce, debian-bugs-announce).
>
> > I am asking because I like the idea of distribution-independent
> > infrastructure message bus that this project had.
>
> Yes, it was a nice idea.
>
> > [...]
>
> Cheers,
> --
> Nicolas Dandrimont

Re: email backend for fedmsg

[Paul Gevers]

Hi Clime,

On 24-03-2020 21:51, clime wrote:
> So do you have the opposite? I do some clicking action somewhere and
> it will send an email to a certain mailing list to inform human
> beings? Or let's not just clicking but e.g. `git push` (something that
> you can still do from command line).
> 
> Do you have: I do some clicking action somewhere and it will send an
> email to a certain mailing list where the email is afterward parsed by
> another service which will do an action (e.g. launch a build) based on
> it?

git push to most salsa based repos with "Closes: #" in the commit
message will trigger a message to the bts which update the bug meta data
to mark the bug as pending and informs the submitter of that bug that
the bug is about to be fixed.

Paul



signature.asc
Description: OpenPGP digital signature

Re: email backend for fedmsg

[Nicolas Dandrimont]

On Tue, Mar 24, 2020, at 21:51, clime wrote:
> On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont  wrote:
> >
> > Hi!
> >
> > On Sun, Mar 22, 2020, at 13:06, clime wrote:
> > > Hello!
> > >
> > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html -
> > > fedmsg usage in Debian.
> > >
> > > There is a note: "it seems that people actually like parsing emails"
> >
> > This was just a way to say that fedmsg never got much of a user base in the 
> > services that run on Debian infra, and that even the new services 
> > introduced at the time kept parsing emails.
> 
> Hello Nicolas!
> 
> Do you remember some such service and how it used email parsing specifically?

I believe that tracker.debian.org was introduced around that time.

At the point it was created, tracker.d.o was mostly consuming emails from 
packages.debian.org to update its data. These days tracker.d.o has replaced 
packages.d.o as "email router", in that it receives all the mails from services 
(e.g. the BTS, the archive maintenance software, buildds, salsa webhooks, ...) 
and forwards them to the public.

> I am still a bit unclear how email parsing is used in Debian
> infrastructure, don't get me wrong, I find it elegant

Ha. I find that it's a big mess.

Here's the set of headers of a message I received today from tracker.d.o, which 
are supposed to make parsing these emails better:

X-PTS-Approved: yes
X-Distro-Tracker-Package: facter
X-Distro-Tracker-Keyword: derivatives
X-Remote-Delivered-To: dispa...@tracker.debian.org
X-Loop: dispa...@tracker.debian.org
X-Distro-Tracker-Keyword: derivatives
X-Distro-Tracker-Package: facter
List-Id: 
X-Debian: tracker.debian.org
X-Debian-Package: facter
X-PTS-Package: facter
X-PTS-Keyword: derivatives
Precedence: list
List-Unsubscribe: 

I'll leave you to judge whether this makes sense or not.

(and it turns out that the actual useful payload was just plaintext with no 
real chance of automated parsing)

> but from what I have found (e.g. reportbug), in the beginning there is an
> email being sent by some human which will then trigger some automatic
> action (e.g. putting the bug into db). So it's like you could do all
> your work simply by sending emails (some of them machine-parsable).
> 
> So do you have the opposite? I do some clicking action somewhere and
> it will send an email to a certain mailing list to inform human
> beings? Or let's not just clicking but e.g. `git push` (something that
> you can still do from command line).
> 
> Do you have: I do some clicking action somewhere and it will send an
> email to a certain mailing list where the email is afterward parsed by
> another service which will do an action (e.g. launch a build) based on
> it?

Both of these are somewhat true.

Some examples of email-based behaviors:
 - Our bug tracking system is fully controlled by email.
 - Closing a bug in reaction to an upload is done by an email from the archive 
maintenance system (dak) to the bug tracking system.
 - Salsa has a webhook service that react to UI clicks (e.g. "clicking the 
merge button") by sending an email to the BTS (e.g. to tag bugs as pending), or 
to tracker.d.o (for new commit notifications).
 - Some of our IRC bots are triggered by procmail rules.
 - At some point mentors.debian.net depended on a NNTP gateway to the 
debian-devel-changes mailing list to trigger removal of superseded packages 
(...)
 - etc. etc.

I'm still not sure where your trail of questions is going? fedmsg in Debian has 
been dead for years at this point, and there still doesn't seem to be much 
interest to implement anything beyond email parsing in some of our core 
systems. 

Bye,
-- 
Nicolas Dandrimont

Re: What to do when DD considers policy to be optional? [kubernetes]

[Russ Allbery]

Jeremy Stanley  writes:

> If this represents the actual state of building Kubernetes, it's
> unclear to me why Debian would package it at all. I don't see the
> value to users in consuming Kubernetes from a Debian package if the
> result is compromising on Debian's vision and values so that they
> can get the exact same thing they'd have if they just used the
> Kubernetes community's recommended tooling to install it instead.

I have been very grateful over the past few months to have Kubernetes
available in Debian (and have been quite annoyed at the irritating things
I have to do get and update Helm, which at least has a snap, and Argo CD,
which doesn't have anything useful).

I have no opinion about the best solution to the huge vendoring problem.
(The Rust team is trying the package everything approach with some success
but is uncovering other limitations in our processes and tools.)  But
having these tools in Debian is hugely valuable for Debian users who need
them, which is sort of the point of Debian at the end of the day.

> I'm all for using the best tool for the job, and while I've been a
> die-hard Debian user for more than two decades I also don't install
> every last bit of software from its packages.

I do when I can because I otherwise have to remember to wire together N
different update mechanisms, which is remarkably not-fun and takes time
away from the actual work I'm trying to do.

-- 
Russ Allbery (r...@debian.org)  

Re: new kubernetes packaging

[Sean Whitton]

Hello Janos,

Thank you for your e-mail.  I agree with you that security support is
the most pressing reason to avoid piles of vendored code, and you make
an interesting argument regarding how it can be difficult to provide
security fixes if our refusal to use vendored code means we lag too far
behind upstream.

I am not sure, however, that your argument applies to security updates
to our stable releases.  These updates are almost always a matter of
backporting small fixes, rather than updating to new upstream releases.
And for backported fixes, vendoring makes things much harder.

You also write:

On Tue 24 Mar 2020 at 07:08PM +00, Janos LENART wrote:

> 1. OTHER EXAMPLES. If we take this paragraph completely literally and to
> the extreme then other packages are also in violation of it. True, the
> current packaging of kubernetes does this to a greater extent than its
> predecessor for example, but perhaps this shows that this section was
> always open for interpretation. Examples of some prominent packages in
> Debian that bundle and use the vendored code (in parentheses is the number
> of go packages bundled, estimate):
> - docker.io (58, including some that are vendored more than once within the
> same source package, but not including the fact that docker.io itself is
> made up of 7 tarballs)
> - kubernetes (20 for the previous version, 200 now)
> - prometheus (4)
> - golang (4)

but I am not sure this is relevant because the number of vendored copies
in the new kubernetes package is an order of magnitude larger than any
of these examples.

Finally, I would like to hear why you think it is valuable for us to
have a package like this in Debian as opposed to expecting people to
install it from upstream:

On Tue 24 Mar 2020 at 08:37PM +00, Jeremy Stanley wrote:

> If this represents the actual state of building Kubernetes, it's
> unclear to me why Debian would package it at all. I don't see the
> value to users in consuming Kubernetes from a Debian package if the
> result is compromising on Debian's vision and values so that they
> can get the exact same thing they'd have if they just used the
> Kubernetes community's recommended tooling to install it instead.
> I'm all for using the best tool for the job, and while I've been a
> die-hard Debian user for more than two decades I also don't install
> every last bit of software from its packages. Some software
> ecosystems have chosen to focus on tools and workflows which are
> incompatible with Debian's, but that doesn't mean that either one is
> inherently bad nor that they need to be integrated at all costs.

I find this persuasive.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: new kubernetes packaging

[Russ Allbery]

Sean Whitton  writes:

> Thank you for your e-mail.  I agree with you that security support is
> the most pressing reason to avoid piles of vendored code, and you make
> an interesting argument regarding how it can be difficult to provide
> security fixes if our refusal to use vendored code means we lag too far
> behind upstream.

> I am not sure, however, that your argument applies to security updates
> to our stable releases.  These updates are almost always a matter of
> backporting small fixes, rather than updating to new upstream releases.
> And for backported fixes, vendoring makes things much harder.

I think this calculus is not entirely obvious.

What you say is true if the library is used by multiple applications in
Debian (although it's still not as good of a story with Go as it is for
C).  We can backport a patch to that one library, and then rebuild the
applications that incorporate it.

However, if a library exists in Debian solely because it is a dependency
of some sprawling application and isn't used by other things, it may be
easier to do a security update if it's vendored.  There are, at the least,
fewer packages to rebuild, and the testing story is slightly more
straightforward.

Possibly more significantly is how the flow of security advisories work.
If the advisory is likely to come from Kubernetes and their security fix
release is a point release update to their package including the vendored
modules, we can potentially adopt the "sane upstream stable point release"
policy and just update stable to their point release.  (Kubernetes does
maintain long-lived stable branches, although I don't know how stringent
they are about what changes they're willing to take in the stable
branches.)  In this case, we create a bit more security work by separately
packaging the dependencies, since we now have to trace down the package
that corresponds to a Kubernetes security advisory and update it.

This of course doesn't apply if the individual libraries are releasing
their own security advisories.

-- 
Russ Allbery (r...@debian.org)  

Re: new kubernetes packaging

[Sean Whitton]

Hello,

On Tue 24 Mar 2020 at 03:14PM -07, Russ Allbery wrote:

> What you say is true if the library is used by multiple applications in
> Debian (although it's still not as good of a story with Go as it is for
> C).  We can backport a patch to that one library, and then rebuild the
> applications that incorporate it.
>
> However, if a library exists in Debian solely because it is a dependency
> of some sprawling application and isn't used by other things, it may be
> easier to do a security update if it's vendored.  There are, at the least,
> fewer packages to rebuild, and the testing story is slightly more
> straightforward.

Right, if something is technically an independent language module but is
used only by kubernetes, and we don't expect that to change, there's no
need for it to be in its own source package just because it might seem
tidier to us.

I doubt that this is true of all the hundreds of dependencies presently
bundled with kubernetes, however.

> Possibly more significantly is how the flow of security advisories work.
> If the advisory is likely to come from Kubernetes and their security fix
> release is a point release update to their package including the vendored
> modules, we can potentially adopt the "sane upstream stable point release"
> policy and just update stable to their point release.  (Kubernetes does
> maintain long-lived stable branches, although I don't know how stringent
> they are about what changes they're willing to take in the stable
> branches.)  In this case, we create a bit more security work by separately
> packaging the dependencies, since we now have to trace down the package
> that corresponds to a Kubernetes security advisory and update it.

This is certainly a reasonable approach, if such releases aren't going
to violate the expectations of users of Debian stable.

I guess we'd have to see whether the Security Team are up for another
package which gets updated in this way.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: new kubernetes packaging

[Simon McVittie]

On Tue, 24 Mar 2020 at 15:14:02 -0700, Russ Allbery wrote:
> I think this calculus is not entirely obvious.

Thank you for applying some much-needed nuance to this issue. I suspect
the ideal policy is neither "never use vendored dependencies" nor
"always use vendored dependencies".

Many of our packaging policies were designed for medium-sized C/C++
libraries - not much smaller than, say, zlib, but not much bigger than
something like GTK either - with a sufficiently stable API and ABI that
versions are somewhat interchangeable, and I think the further away
from that we get, the less well those policies will fit. We have a lot
of trouble with micropackages (as exemplified by the nodejs ecosystem
in which many packages provide a single function), but at the other end
of the scale, monoliths like Libreoffice, TeXLive and Firefox don't fit
a lot of our usual policies and practices so well either.

> Possibly more significantly is how the flow of security advisories work.
> If the advisory is likely to come from Kubernetes and their security fix
> release is a point release update to their package including the vendored
> modules, we can potentially adopt the "sane upstream stable point release"
> policy and just update stable to their point release.
...
> This of course doesn't apply if the individual libraries are releasing
> their own security advisories.

I think the API stability of the libraries is also relevant (and ABI
would be relevant too, if we had dynamically-linked Go libraries), both
in terms of intended API/ABI breaks and unintended behaviour changes
and regressions. The more stable they are, the more appealing it is to
have them in a shared library; the more unstable they are, the more
appealing it is to vendor them into a larger project.

This isn't necessarily such a new thing - the scale is new, but the
practice isn't. There are several C/C++ libraries in Debian that are
specifically designed to be vendored into dependent projects (either
because they are not API-stable or to simplify dependency management),
like gnulib (which exists as a package, but I think it's only there to
facilitate vendoring bits of it?), libstb (which does exist as a separate
package with a shared library, but I don't have a good picture of how API-
and ABI-stable it is), and libglnx.

smcv

Re: new kubernetes packaging

[Russ Allbery]

Simon McVittie  writes:

> I think the API stability of the libraries is also relevant (and ABI
> would be relevant too, if we had dynamically-linked Go libraries), both
> in terms of intended API/ABI breaks and unintended behaviour changes and
> regressions. The more stable they are, the more appealing it is to have
> them in a shared library; the more unstable they are, the more appealing
> it is to vendor them into a larger project.

I think this is also where upstream intentions are important.  For
example, the Rust community does care (intensely) about API stability and
even ABI stability, and is at least thinking about a future of Rust shared
libraries, although that's not currently the normal mechanism of
development of pure Rust packages.  They're sympathetic.  This is part of
what makes our packaging approach viable, I think.

On the other hand (and I don't follow this community closely, so apologies
if I have the details wrong here), my impression is that the Go community
is not planning to support shared libraries, loves its staticly-linked
binaries, and makes extensive use of the fact that different packages can
pin to different versions and this doesn't matter for their intended
output format (a static binary).

Trying to shoehorn the latter into a shared library update model is almost
certain to fail because it's working at intense cross-purposes to
upstream.

> This isn't necessarily such a new thing - the scale is new, but the
> practice isn't. There are several C/C++ libraries in Debian that are
> specifically designed to be vendored into dependent projects (either
> because they are not API-stable or to simplify dependency management),
> like gnulib (which exists as a package, but I think it's only there to
> facilitate vendoring bits of it?), libstb (which does exist as a
> separate package with a shared library, but I don't have a good picture
> of how API- and ABI-stable it is), and libglnx.

Indeed, I have a package, rra-c-util, which is vendored into every C
package that I personally maintain and package, because it's my version of
gnulib plus some other utility functions.  I recognize the potential
concern should a security vulnerability be found in any of its functions,
and accept the cost of providing security updates for every one of my
packages that use it.  This still is, in my opinion, a better maintenance
choice, not so much for Debian but for many non-Debian users of those C
packages who do not want to (and often get confused by trying to) install
a shared library as a prerequisite to installing the thing they actually
care about.  (Also because, like gnulib, rra-c-util consists of a lot of
different pieces, most of which are not needed for any given package, and
includes pieces like Autoconf machinery that are tricky to maintain
separately.)

-- 
Russ Allbery (r...@debian.org)  

Re: email backend for fedmsg

[clime]

On Tue, 24 Mar 2020 at 22:45, Nicolas Dandrimont  wrote:
>
> On Tue, Mar 24, 2020, at 21:51, clime wrote:
> > On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont  wrote:
> > >
> > > Hi!
> > >
> > > On Sun, Mar 22, 2020, at 13:06, clime wrote:
> > > > Hello!
> > > >
> > > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html -
> > > > fedmsg usage in Debian.
> > > >
> > > > There is a note: "it seems that people actually like parsing emails"
> > >
> > > This was just a way to say that fedmsg never got much of a user base in 
> > > the services that run on Debian infra, and that even the new services 
> > > introduced at the time kept parsing emails.
> >
> > Hello Nicolas!
> >
> > Do you remember some such service and how it used email parsing 
> > specifically?
>
> I believe that tracker.debian.org was introduced around that time.
>
> At the point it was created, tracker.d.o was mostly consuming emails from 
> packages.debian.org to update its data. These days tracker.d.o has replaced 
> packages.d.o as "email router", in that it receives all the mails from 
> services (e.g. the BTS, the archive maintenance software, buildds, salsa 
> webhooks, ...) and forwards them to the public.
>
> > I am still a bit unclear how email parsing is used in Debian
> > infrastructure, don't get me wrong, I find it elegant
>
> Ha. I find that it's a big mess.
>
> Here's the set of headers of a message I received today from tracker.d.o, 
> which are supposed to make parsing these emails better:
>
> X-PTS-Approved: yes
> X-Distro-Tracker-Package: facter
> X-Distro-Tracker-Keyword: derivatives
> X-Remote-Delivered-To: dispa...@tracker.debian.org
> X-Loop: dispa...@tracker.debian.org
> X-Distro-Tracker-Keyword: derivatives
> X-Distro-Tracker-Package: facter
> List-Id: 
> X-Debian: tracker.debian.org
> X-Debian-Package: facter
> X-PTS-Package: facter
> X-PTS-Keyword: derivatives
> Precedence: list
> List-Unsubscribe: 
> 
>
> I'll leave you to judge whether this makes sense or not.
>
> (and it turns out that the actual useful payload was just plaintext with no 
> real chance of automated parsing)
>
> > but from what I have found (e.g. reportbug), in the beginning there is an
> > email being sent by some human which will then trigger some automatic
> > action (e.g. putting the bug into db). So it's like you could do all
> > your work simply by sending emails (some of them machine-parsable).
> >
> > So do you have the opposite? I do some clicking action somewhere and
> > it will send an email to a certain mailing list to inform human
> > beings? Or let's not just clicking but e.g. `git push` (something that
> > you can still do from command line).
> >
> > Do you have: I do some clicking action somewhere and it will send an
> > email to a certain mailing list where the email is afterward parsed by
> > another service which will do an action (e.g. launch a build) based on
> > it?
>
> Both of these are somewhat true.
>
> Some examples of email-based behaviors:
>  - Our bug tracking system is fully controlled by email.
>  - Closing a bug in reaction to an upload is done by an email from the 
> archive maintenance system (dak) to the bug tracking system.
>  - Salsa has a webhook service that react to UI clicks (e.g. "clicking the 
> merge button") by sending an email to the BTS (e.g. to tag bugs as pending), 
> or to tracker.d.o (for new commit notifications).
>  - Some of our IRC bots are triggered by procmail rules.
>  - At some point mentors.debian.net depended on a NNTP gateway to the 
> debian-devel-changes mailing list to trigger removal of superseded packages 
> (...)
>  - etc. etc.
>
> I'm still not sure where your trail of questions is going? fedmsg in Debian 
> has been dead for years at this point, and there still doesn't seem to be 
> much interest to implement anything beyond email parsing in some of our core 
> systems.

Cool, so basically what I am thinking about is to create a free
software from what you are describing. I.e. create reusable tooling
out of the Debian messaging system. Something that a new linux
distribution can easily start using to connect their services.

I didn't know Debian infra works like this but I find it very
elegant/efficient and I would like the solution you have to be
reusable by others.

So basically the tooling should contain:
- unified email message format
- library that is able to translate a message to a language data
structure (e.g. dictionary in python)
- email receiver that would be listening for emails coming from the
bus and emitting events based on that (this could be part of the
library so you would be able to attach a callback for an incoming
message or just do blocking waits)
- email publisher - something that can send a new message into the
bus, i.e. to a preconfigured mail server (a "broker" or "hub")
- mail server that would have an http API to manage topic
subscriptions  (i.e. add/delete me from a given topic

Re: email backend for fedmsg

[clime]

On Wed, 25 Mar 2020 at 01:00, clime  wrote:
>
> On Tue, 24 Mar 2020 at 22:45, Nicolas Dandrimont  wrote:
> >
> > On Tue, Mar 24, 2020, at 21:51, clime wrote:
> > > On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont  wrote:
> > > >
> > > > Hi!
> > > >
> > > > On Sun, Mar 22, 2020, at 13:06, clime wrote:
> > > > > Hello!
> > > > >
> > > > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html -
> > > > > fedmsg usage in Debian.
> > > > >
> > > > > There is a note: "it seems that people actually like parsing emails"
> > > >
> > > > This was just a way to say that fedmsg never got much of a user base in 
> > > > the services that run on Debian infra, and that even the new services 
> > > > introduced at the time kept parsing emails.
> > >
> > > Hello Nicolas!
> > >
> > > Do you remember some such service and how it used email parsing 
> > > specifically?
> >
> > I believe that tracker.debian.org was introduced around that time.
> >
> > At the point it was created, tracker.d.o was mostly consuming emails from 
> > packages.debian.org to update its data. These days tracker.d.o has replaced 
> > packages.d.o as "email router", in that it receives all the mails from 
> > services (e.g. the BTS, the archive maintenance software, buildds, salsa 
> > webhooks, ...) and forwards them to the public.
> >
> > > I am still a bit unclear how email parsing is used in Debian
> > > infrastructure, don't get me wrong, I find it elegant
> >
> > Ha. I find that it's a big mess.
> >
> > Here's the set of headers of a message I received today from tracker.d.o, 
> > which are supposed to make parsing these emails better:
> >
> > X-PTS-Approved: yes
> > X-Distro-Tracker-Package: facter
> > X-Distro-Tracker-Keyword: derivatives
> > X-Remote-Delivered-To: dispa...@tracker.debian.org
> > X-Loop: dispa...@tracker.debian.org
> > X-Distro-Tracker-Keyword: derivatives
> > X-Distro-Tracker-Package: facter
> > List-Id: 
> > X-Debian: tracker.debian.org
> > X-Debian-Package: facter
> > X-PTS-Package: facter
> > X-PTS-Keyword: derivatives
> > Precedence: list
> > List-Unsubscribe: 
> > 
> >
> > I'll leave you to judge whether this makes sense or not.
> >
> > (and it turns out that the actual useful payload was just plaintext with no 
> > real chance of automated parsing)
> >
> > > but from what I have found (e.g. reportbug), in the beginning there is an
> > > email being sent by some human which will then trigger some automatic
> > > action (e.g. putting the bug into db). So it's like you could do all
> > > your work simply by sending emails (some of them machine-parsable).
> > >
> > > So do you have the opposite? I do some clicking action somewhere and
> > > it will send an email to a certain mailing list to inform human
> > > beings? Or let's not just clicking but e.g. `git push` (something that
> > > you can still do from command line).
> > >
> > > Do you have: I do some clicking action somewhere and it will send an
> > > email to a certain mailing list where the email is afterward parsed by
> > > another service which will do an action (e.g. launch a build) based on
> > > it?
> >
> > Both of these are somewhat true.
> >
> > Some examples of email-based behaviors:
> >  - Our bug tracking system is fully controlled by email.
> >  - Closing a bug in reaction to an upload is done by an email from the 
> > archive maintenance system (dak) to the bug tracking system.
> >  - Salsa has a webhook service that react to UI clicks (e.g. "clicking the 
> > merge button") by sending an email to the BTS (e.g. to tag bugs as 
> > pending), or to tracker.d.o (for new commit notifications).
> >  - Some of our IRC bots are triggered by procmail rules.
> >  - At some point mentors.debian.net depended on a NNTP gateway to the 
> > debian-devel-changes mailing list to trigger removal of superseded packages 
> > (...)
> >  - etc. etc.
> >
> > I'm still not sure where your trail of questions is going? fedmsg in Debian 
> > has been dead for years at this point, and there still doesn't seem to be 
> > much interest to implement anything beyond email parsing in some of our 
> > core systems.
>
> Cool, so basically what I am thinking about is to create a free
> software from what you are describing. I.e. create reusable tooling
> out of the Debian messaging system. Something that a new linux
> distribution can easily start using to connect their services.
>
> I didn't know Debian infra works like this but I find it very
> elegant/efficient and I would like the solution you have to be
> reusable by others.
>
> So basically the tooling should contain:
> - unified email message format
> - library that is able to translate a message to a language data
> structure (e.g. dictionary in python)
> - email receiver that would be listening for emails coming from the
> bus and emitting events based on that (this could be part of the
> library so you would be able to attach a callback for an incoming
> message

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marco d'Itri]

On Mar 24, Russ Allbery  wrote:

> (The Rust team is trying the package everything approach with some success
> but is uncovering other limitations in our processes and tools.)  But
"Some" success indeed. My personal experience with trying to package 
routinator has been awful, and there is still no actualy package in the 
archive after many months because it depends on a version of a library 
which is different from the version that we have in the archive, and 
there is nothing wrong with this in the Rust world.

The main reason for mostly forbidding vendored libraries has been that 
the security team rightly argues that in the event of a security issue 
it would be too much work to 1) hunt each package using a vendored 
library and 2) patch and rebuild all of them.
This does not really matter for Go and Rust software because 1) the list 
of (vendored) dependencies can be extracted automatically at build time 
and 2) all this software would have to be rebuilt anyway since these 
languages do not support or do not use dynamic linking.

Also, shared libraries save memory when multiple programs using them are 
run concurrently, but nowadays this kind of saving is rarely meaningful.

Because of these reasons maybe we should consider supporting vendored 
libraries in some cases.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

On Wed, 25 Mar 2020 02:07:13 +0100
Marco d'Itri  wrote:
> On Mar 24, Russ Allbery  wrote:
> [...]
> The main reason for mostly forbidding vendored libraries has been that 
> the security team rightly argues that in the event of a security issue 
> it would be too much work to 1) hunt each package using a vendored 
> library and 2) patch and rebuild all of them.
> This does not really matter for Go and Rust software because 1) the list 
> of (vendored) dependencies can be extracted automatically at build time 
> and 2) all this software would have to be rebuilt anyway since these 
> languages do not support or do not use dynamic linking.
> 
> Also, shared libraries save memory when multiple programs using them are 
> run concurrently, but nowadays this kind of saving is rarely meaningful.

My point earlier was that it's not just a security problem. We have established
that Debian does, and will continue to, care about fulfilling license
obligations, including distributing license and copyright information with the
package.

Bluntly put, I have yet to see a project that doesn't treat 'vendor/' as a black
box of collected libraries. These directories are rarely reviewed and it is
trivial (default) to wind up including extra libs between builds. Even if it's
possible to restrict that list, I doubt it would be done.

I was (pleasantly) surprised to see d/copyright updated in this package.
However, this is not maintainable-
   https://sources.debian.org/src/kubernetes/1.17.4-1/debian/copyright/


> Because of these reasons maybe we should consider supporting vendored 
> libraries in some cases.

My opinion is still a very hard "No."

This sounds (to me) rather akin to- "since app devs tend to be lazy, should we
be the same?"


One last thing to consider... NEW reviews are already an intense process. If
this package hit NEW /and/ we allowed vendored libs, you could safely expect me
to never complete that particular review. I doubt I'm the only one; that's
essentially ~200 package reviews wrapped into 1.

-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Russ Allbery]

Michael Lustfield  writes:

> One last thing to consider... NEW reviews are already an intense
> process. If this package hit NEW /and/ we allowed vendored libs, you
> could safely expect me to never complete that particular review. I doubt
> I'm the only one; that's essentially ~200 package reviews wrapped into
> 1.

I'll repeat a point that I made earlier but put a bit of a sharper point
on it: We should thoughtfully question whether the current approach to
license review that we as a project ask ftpmasters to do is a correct
investment of project resources.

The current approach to NEW license review is not a law of nature.  It's a
choice.  Other choices are possible, such as trusting license declarations
by upstream and then handling upstream mistakes that are later discovered
as RC bugs.  The project is much better at sharing the load of handling RC
bugs than it is at sharing the load of NEW license reviews.

Most of the ecosystems that make extensive use of vendored packages also
make extensive use of license metadata.  Sometimes that license metadata
is wrong, and we will have to have a process for dealing with those
errors.  But the purpose of Debian is not to be a license checking service
for the entire free software world.  It's to build a distribution adhering
to a set of principles but with the understanding that we will sometimes
make errors and fix them later as bugs, not be perfect and error-free at
any of our principles, including our licensing principles.  For everything
other than licenses, we accept the risk that we will incorporate upstream
errors in our distribution until someone points them out via a bug report.

We do not *have* to do a detailed file-by-file review of the correctness
of upstream's license metadata when packaging.  This is a choice.  By
choosing to do this, we absolutely catch bugs... just like we would catch
bugs if we did a detailed file-by-file review of any other property of
upstream code.  For many other types of bugs (including ones that arguably
have a more severe impact on our users than licensing bugs, such as
security bugs), we often make trade-off decisions in favor of accepting a
risk of upstream mistakes and having to fix them later.

-- 
Russ Allbery (r...@debian.org)  

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Wednesday, 25 March 2020 6:08:23 AM AEDT Janos LENART wrote:
> I know Dimitry was fighting an uphill battle with kubernetes between 2016
> and 2018 and he experienced first hand the problems posed by vendored code.

No. This is a incorrect. Largest chunk of work that I did on Kubernetes was 
in 2015..2016 when I was packaging and introducing many new libraries and 
stabilizing their relationships. A lot of this work has been done with 
various upstreams and a lot of it was in Kubernetes' dependency packages such 
as docker.io. Licensing/copyright documentation and addressing numerous DFSG 
problems took a lot of time and effort to meet requirements for new packages. 
This effort took more than a year or work. Between 2016...2018 I barely 
touched Kubernetes as it was already orphaned.

Challenge is not with "vendored code" as such, or at least not the way you 
think.


> We see more and more software making excessive use of vendored code. Pretty
> much everything that is written in Go. Some of these are crucially
> important, like Docker or Kubernetes. So I understand the concern everyone
> has about how this fits with the Debian Policy.

No you don't understand.

Vendored code is nuisance but not a problem if/when you can throw it away in 
favor of packaged, tested, reusable libraries.

You've made no effort to reuse any existing packaged libraries. Most of them 
could be used without much effort. And there are many advantages of doing so 
beyond obvious benefits for security and de-duplication.

There is integrity of build to care about, and automated QA/CI checks for 
libraries to appreciate. Most of the libraries are tested on more 
architectures that upstreams ever test. We know when there is a problem and 
can do something about it. Concept of good quality reusable components is too 
powerful to throw it away just because you think it is inconvenient.

Vendored Golang libraries run no tests on build. IMHO Kubernetes bundle (with 
vendor directory) is unmaintainable and you can find how much upstream 
struggles with it to extent when they would not upgrade a component to fix a 
problem due to fears that everything might fall apart.

Using properly packages libraries is an advantage to maintainer and a benefit 
to a whole ecosystem.


> Debian Policy, paragraph 4.13 states:

Before we discuss the policy let me focus on your attitude first.

Kubernetes -- one of the most sophisticated packages -- was you _second_ ever 
upload to Debian. You are woefully inexperienced maintainer, knowing little 
if anything about team work, packaging practices, their meanings and 
implications. IMHO your interpretation of policy is mostly irrelevant as you 
are merely trying to use the policy to justify what you did.

There are several problems with how you did it too. You did not use anyone's 
advise, ignored Salsa repository, threw away _everything_ and made no effort 
to understand how and why things were implemented, let alone appreciated 
prior work or tried to improve it. What you did is technological hijack of 
the package, a gross violation of practices.

Imagine I'll upload a package to NEW, get in reviewed and accepted for what 
it is then re-upload as something entirely different bundled with 500+ 
dependencies that were not reviewed then claim that policy allows it?


> =
> I think this is the part that has the most bearing on the vendored code
> problem, especially the footnote. I agree with this principle. But we
> should apply it to the state of affairs in 2020, and to this specific
> situation.

Nonsense. In 2020 using packaged libraries is much easier than before.
You simply have no excuse not to do so. Again, you are not understanding what 
is the problem.


> Keeping all that in mind, here are the reasons why I think it is acceptable
> for now to package Kubernetes with the vendored code, and even the best
> solution that is available currently:

I keep telling you that it is not a best solution but the sloppiest and the 
most inferior one.


> 1. OTHER EXAMPLES.
> [...]
> - docker.io (58, including some that are vendored more than once within the
> same source package, but not including the fact that docker.io itself is
> made up of 7 tarballs)

Docker.io is a special, exceptionally difficult case which should be an 
example to you that even with Docker it is possible to leverage packaged 
libraries. Docker upstream is one of the worst with their abuse of versioning 
practices. On top of that Docker code base is shipped in several name spaces 
that make it impossible to package some components separately due to mutual 
circular dependencies. In Docker we use strategically bundled components only 
when necessary.


> - kubernetes (20 for the previous version, 200 now)
> - prometheus (4)
> - golang (4)
> None of these were REJECTed, and please don't sabotage these packages now

Perhaps Nomad or recently accepted Vault would serve as a better example. 
Vault is one of packages with gre

Re: What to do when DD considers policy to be optional? [kubernetes]

[Wookey]

On 2020-03-24 19:08 +, Janos LENART wrote:
> Hi Dimitry, FTP masters and others,
> 4. TESTING. The Kubernetes releases are meticulously tested, with far greater
> technical resources that Debian can collectively muster.

On how many architectures? Debian's support of multiple architectures
often differentiates it from upstream, which sometimes knows little or
nothing of !x86. I don't know if kubernetes is doing a good job of
testing on other arches, but it's important to consider those when
thinking about the tradeoffs here.

I do know this is important software on arm64 too, and agree with Russ
that it's nice to have things packaged even where they are largely
what you would get from upstream, just because one only has to learn
one package manager, and can assume that policy has been followed at
least in terms of layout.

I am not at all happy with the tendency of some developer
groups/projects to bundle like crazy, but it's also true that there
are constraints on the degree to which we can and should (try to) fix
that for them. You make some good points in defense of your approach,
and thank you for responding cogently at length, but I'm not (yet)
convinced by the argument that we shouldn't try at all in this case.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Regarding vendor/ libraries... [Was: Re: What to do when DD considers policy to be optional? [kubernetes]]

[Michael Lustfield]

On Tue, 24 Mar 2020 19:25:49 -0700
Russ Allbery  wrote:

> Michael Lustfield  writes:
> 
> > One last thing to consider... NEW reviews are already an intense
> > process. If this package hit NEW /and/ we allowed vendored libs, you
> > could safely expect me to never complete that particular review. I doubt
> > I'm the only one; that's essentially ~200 package reviews wrapped into
> > 1.  
> 
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.  The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.
> 
> Most of the ecosystems that make extensive use of vendored packages also
> make extensive use of license metadata.  Sometimes that license metadata
> is wrong, and we will have to have a process for dealing with those
> errors.  But the purpose of Debian is not to be a license checking service
> for the entire free software world.  It's to build a distribution adhering
> to a set of principles but with the understanding that we will sometimes
> make errors and fix them later as bugs, not be perfect and error-free at
> any of our principles, including our licensing principles.  For everything
> other than licenses, we accept the risk that we will incorporate upstream
> errors in our distribution until someone points them out via a bug report.
> 
> We do not *have* to do a detailed file-by-file review of the correctness
> of upstream's license metadata when packaging.  This is a choice.  By
> choosing to do this, we absolutely catch bugs... just like we would catch
> bugs if we did a detailed file-by-file review of any other property of
> upstream code.  For many other types of bugs (including ones that arguably
> have a more severe impact on our users than licensing bugs, such as
> security bugs), we often make trade-off decisions in favor of accepting a
> risk of upstream mistakes and having to fix them later.

Scott K. reminded me that the policy in question is a "should" and not a
"must." This is a pretty important distinction that I forgot about. In
practice, it's useful for when a single file is taken from a large project,
sometimes including weird build tools.

Assuming d/copyright is actually correct (it's not..), then it's not a policy
violation. However, for all of the reasons mentioned, and especially because
most of that work was already complete, I have to agree with the term sloppy.

We obviously need to continue the vendor/ discussion, but I think switching
threads would be a good idea.

-- 
Michael Lustfield

Re: What to do when DD considers policy to be optional? [kubernetes]

[Michael Lustfield]

Ehm... perhaps we should practice some de-escalation techniques, please. :/

On Wed, 25 Mar 2020 13:55:50 +1100
Dmitry Smirnov  wrote:
> On Wednesday, 25 March 2020 6:08:23 AM AEDT Janos LENART wrote:
> > Debian Policy, paragraph 4.13 states:  
> 
> There are several problems with how you did it too. You did not use anyone's 
> advise, ignored Salsa repository, threw away _everything_ and made no effort 
> to understand how and why things were implemented, let alone appreciated 
> prior work or tried to improve it. What you did is technological hijack of 
> the package, a gross violation of practices.
> 
> Imagine I'll upload a package to NEW, get in reviewed and accepted for what 
> it is then re-upload as something entirely different bundled with 500+ 
> dependencies that were not reviewed then claim that policy allows it?

I missed this earlier...

With regard to the kubernetes package, I don't see anything to indicate it was
abandoned. I'll also assume that the current maintainer (Dmitry) was not
contacted by Janos. Considering how quickly this started after the upload, this
seems like a safe bet.

Janos- If that's correct, then this would indeed be considered a "hostile
takeover" of the package. It would be nice to assume that this was simple
oversight of forgotten communication.

No matter the actual facts- please remember how important communication is to
our community.

So... with Dmitry still active in Debian and still holding interest in the
package, the path forward seems obvious, doesn't it? One person is interested
in maintaining the package per our standards, and another is interested in
getting it updated.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Wednesday, 25 March 2020 2:34:03 PM AEDT Michael Lustfield wrote:
> With regard to the kubernetes package, I don't see anything to indicate it
> was abandoned.

Sorry if I did not make it clear: the package was orphaned as per #886739.
The takeover was only technological. I don't dispute the ownership of the 
package.


> One person is
> interested in maintaining the package per our standards, and another is
> interested in getting it updated.

It is all about balance... I think we can have both: a package updated with 
respect to our standards or at least with best effort to respect standards. 

I'm not interested in maintaining Kubernetes any more. I simply can't afford 
to spend more time on it after investing so much already. I'm somewhat 
disappointed in upstream and their handling of the project. I'm not even 
using Kubernetes and my concerns are about packaging practices and 
methodologies.

-- 
All the best,
 Dmitry Smirnov.

---

Doublethink means the power of holding two contradictory beliefs in one's
mind simultaneously, and accepting both of them.
-- George Orwell, 1984.


signature.asc
Description: This is a digitally signed message part.

Re: new kubernetes packaging

[Shengjing Zhu]

Another question for the current kubernetes maintainer.

What's your plan for the k8s.io/* libraries, eg k8s.io/api k8s.io/client-go.
They are supposed to be built from src:kubernetes, but it currently doesn't.

Some existing packages already embed them, like
https://codesearch.debian.net/search?q=filetype%3Ago+k8s.io%2Fapi&literal=1
Because we thought it's hard to maintain kubernetes package in Debian,
meanwhile follow the common practice in pkg-go team.

Some new packages are blocked by not having kubernetes libraries. I
haven't checked the wnpp list. But recently there's a thread in
debian-go@.

If you provide k8s.io/* libraries, what about the libraries that
k8s.io/* depends?

-- 
Shengjing Zhu

Re: new kubernetes packaging

[Vincent Bernat]

 ❦ 24 mars 2020 16:30 -07, Russ Allbery:

> On the other hand (and I don't follow this community closely, so apologies
> if I have the details wrong here), my impression is that the Go community
> is not planning to support shared libraries, loves its staticly-linked
> binaries, and makes extensive use of the fact that different packages can
> pin to different versions and this doesn't matter for their intended
> output format (a static binary).

Go supports shared libraries but I don't think it's widely used.
Notably, the tooling around it is quite primitive.
>
> Trying to shoehorn the latter into a shared library update model is almost
> certain to fail because it's working at intense cross-purposes to
> upstream.
>
>> This isn't necessarily such a new thing - the scale is new, but the
>> practice isn't. There are several C/C++ libraries in Debian that are
>> specifically designed to be vendored into dependent projects (either
>> because they are not API-stable or to simplify dependency management),
>> like gnulib (which exists as a package, but I think it's only there to
>> facilitate vendoring bits of it?), libstb (which does exist as a
>> separate package with a shared library, but I don't have a good picture
>> of how API- and ABI-stable it is), and libglnx.
>
> Indeed, I have a package, rra-c-util, which is vendored into every C
> package that I personally maintain and package, because it's my version of
> gnulib plus some other utility functions.  I recognize the potential
> concern should a security vulnerability be found in any of its functions,
> and accept the cost of providing security updates for every one of my
> packages that use it.  This still is, in my opinion, a better maintenance
> choice, not so much for Debian but for many non-Debian users of those C
> packages who do not want to (and often get confused by trying to) install
> a shared library as a prerequisite to installing the thing they actually
> care about.  (Also because, like gnulib, rra-c-util consists of a lot of
> different pieces, most of which are not needed for any given package, and
> includes pieces like Autoconf machinery that are tricky to maintain
> separately.)

-- 
This night methinks is but the daylight sick.
-- William Shakespeare, "The Merchant of Venice"


signature.asc
Description: PGP signature

Re: new kubernetes packaging

[Vincent Bernat]

 ❦ 24 mars 2020 16:30 -07, Russ Allbery:

> On the other hand (and I don't follow this community closely, so apologies
> if I have the details wrong here), my impression is that the Go community
> is not planning to support shared libraries, loves its staticly-linked
> binaries, and makes extensive use of the fact that different packages can
> pin to different versions and this doesn't matter for their intended
> output format (a static binary).

Go supports shared libraries since quite some time but I don't think
it's widely used. Notably, the tooling around it is quite primitive.
Even the plugin system (which is mostly like dlopen() and could be
useful in many cases) is seldomly used.
-- 
This night methinks is but the daylight sick.
-- William Shakespeare, "The Merchant of Venice"


signature.asc
Description: PGP signature