Re: new kubernetes packaging

[Florian Weimer]

* Vincent Bernat:

>  ❦ 24 mars 2020 16:30 -07, Russ Allbery:
>
>> On the other hand (and I don't follow this community closely, so apologies
>> if I have the details wrong here), my impression is that the Go community
>> is not planning to support shared libraries, loves its staticly-linked
>> binaries, and makes extensive use of the fact that different packages can
>> pin to different versions and this doesn't matter for their intended
>> output format (a static binary).
>
> Go supports shared libraries since quite some time but I don't think
> it's widely used. Notably, the tooling around it is quite primitive.
> Even the plugin system (which is mostly like dlopen() and could be
> useful in many cases) is seldomly used.

That's true, but also somewhat besides the point because in order to
use dynamic shared objects to avoid recompilation of applications, you
also need practical ABI stability, both between compiler versions and
versions of the library.  Go does not have a low-level ABI that
remains unchanged across compiler versions, and (like C and C++) it
encodes struct offsets and sizes directly in the machine code,
sometimes in unexpected places due to inlining.  So even if the Go
standard library was linked as a shared object, you would still have
to rebuild all applications using it.

I believe GHC is similar in this regard.

Using shared objects under such circumstances only makes updates
harder for end users because live systems end up in inconsistent
states (ideally only for a brief time).

De-vendoring sources might still be an advantage because applications
can be fixed with a bin-NMU, but it's a lot of work.  The resulting
divergence from upstream can result in additional bugs.  On the other
hand, there are projects which bundle sources only for developer
convenience, but expect production binaries to use different library
sources for the dependencies.  I don't know if Kubernetes is one of
those projects.

Re: new kubernetes packaging

[Simon McVittie]

On Wed, 25 Mar 2020 at 08:41:45 +0100, Florian Weimer wrote:
> De-vendoring sources might still be an advantage because applications
> can be fixed with a bin-NMU, but it's a lot of work.  The resulting
> divergence from upstream can result in additional bugs.  On the other
> hand, there are projects which bundle sources only for developer
> convenience, but expect production binaries to use different library
> sources for the dependencies.

I think it's important to distinguish between those two sets of
expectations, yes.  Indeed, the same is true in C/C++. We de-vendor
dependencies that are a stable library or CLI tool in their own right
("convenience copies"), but we don't de-vendor dependencies that are
designed to be vendored or that are tightly coupled to the parent package:

- flatpak's vendored bubblewrap and xdg-dbus-proxy: convenience copy,
  use system copy instead
- flatpak's vendored libglnx: unstable "copylib" designed to be vendored,
  keep
- ioquake3's vendored libjpeg: convenience copy, use system copy instead
  (and indeed it's been excluded from the .orig tarball to make d/copyright
  less onerous to maintain, since unrelated files had to be excluded for
  DFSG reasons anyway)
- mutter's fork of code that used to be cogl/clutter: unstable and tightly
  coupled, keep

The meson build system somewhat formalizes this with its concept of
subprojects, which can either be embedded in the upstream git repository
(by direct copying or git subtree), referenced by URL in the upstream
git repository but copied into in dist tarballs (git submodule),
or an external reference by URL (.wrap file), and can either be used
unconditionally, used only as a fallback if there is no suitable system
copy, or have a configure-time choice.

smcv

Re: new kubernetes packaging

[Alastair McKinstry]

On 24/03/2020 23:05, Simon McVittie wrote:
> On Tue, 24 Mar 2020 at 15:14:02 -0700, Russ Allbery wrote:
>> I think this calculus is not entirely obvious.
> Thank you for applying some much-needed nuance to this issue. I suspect
> the ideal policy is neither "never use vendored dependencies" nor
> "always use vendored dependencies".
>
> Many of our packaging policies were designed for medium-sized C/C++
> libraries - not much smaller than, say, zlib, but not much bigger than
> something like GTK either - with a sufficiently stable API and ABI that
> versions are somewhat interchangeable, and I think the further away
> from that we get, the less well those policies will fit. We have a lot
> of trouble with micropackages (as exemplified by the nodejs ecosystem
> in which many packages provide a single function), but at the other end
> of the scale, monoliths like Libreoffice, TeXLive and Firefox don't fit
> a lot of our usual policies and practices so well either.
>
We should probably look at formalizing a concept of "bundle" packages,
shipping either a number of static libs-like files (go) in a -dev package

or a bundle of microlibs in a libjs-* package.

Dependencies using them would use Built-Using: for package construction
but also ship a description of the exact content versions.

Note that we also have bunches of lib* packages shipping multiple small
C/C++ shared libs, as upstream does it this way;

we need to set policies in place for  handling incompatible upgrades in
these cases: what do we need to do to ship multiple incompatible copies
of a Go static lib (do we ever do that? )

One example: openmpi ships multiple shared libs in a libopenmpi3
package. I'm arguing with upstream that the SOVERSIONS of the shared
libs need to be kept in lockstep

so that if for example package 'libopenmpi3' ships 'libopen-pal.so.40',
then 'libopenmpi4' does not; the soversions gets bumped to '*.so.50*'
for them all, even if no code changes in this particular library.

> smcv
>
-- 
Alastair McKinstry, email: alast...@sceal.ie, matrix: @alastair:sceal.ie, 
phone: 087-6847928
Green Party Councillor, Galway County Council 

we have choice, but no change (Re: What to do when DD considers policy to be optional? [kubernetes])

[Holger Levsen]

On Tue, Mar 24, 2020 at 07:25:49PM -0700, Russ Allbery wrote:
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible
[...]
> The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.
[...] 

ohh, yes!

So what's the way to change the project's choice on this?


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

Russ,

On 25.03.20 03:25, Russ Allbery wrote:
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.
> 
> The current approach to NEW license review is not a law of nature.  It's a
> choice.  Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.  The project is much better at sharing the load of handling RC
> bugs than it is at sharing the load of NEW license reviews.
> 
> Most of the ecosystems that make extensive use of vendored packages also
> make extensive use of license metadata.  Sometimes that license metadata
> is wrong, and we will have to have a process for dealing with those
> errors.  But the purpose of Debian is not to be a license checking service
> for the entire free software world.  It's to build a distribution adhering
> to a set of principles but with the understanding that we will sometimes
> make errors and fix them later as bugs, not be perfect and error-free at
> any of our principles, including our licensing principles.  For everything
> other than licenses, we accept the risk that we will incorporate upstream
> errors in our distribution until someone points them out via a bug report.
> 
> We do not *have* to do a detailed file-by-file review of the correctness
> of upstream's license metadata when packaging.  This is a choice.  By
> choosing to do this, we absolutely catch bugs... just like we would catch
> bugs if we did a detailed file-by-file review of any other property of
> upstream code.  For many other types of bugs (including ones that arguably
> have a more severe impact on our users than licensing bugs, such as
> security bugs), we often make trade-off decisions in favor of accepting a
> risk of upstream mistakes and having to fix them later.

if I could +1000 just one post, ever, it's this one.

The vast majority of people developing Open Source Software nowadays
just publish stuff on GitHub and friends. They fork code, they merge
other people's contributions, etc.

And at no point can I see those people engage in even a fraction of the
bureaucracy that we follow when sharing, forking, publishing, and
contributing code.

This is not to say that licensing is an unimportant issue -- it clearly
is. But our analyze-and-document down-to-the-file approach is on the
other extreme end of the spectrum, and it causes lots of tiresome work
that nobody apart from us seems to care about.

> Other choices are possible, such as trusting license declarations
> by upstream and then handling upstream mistakes that are later discovered
> as RC bugs.

This would be fantastic. Just fantastic.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Tomas Pospisek]

On 25.03.20 14:43, Christian Kastner wrote:

> This is not to say that licensing is an unimportant issue -- it clearly
> is. But our analyze-and-document down-to-the-file approach is on the
> other extreme end of the spectrum, and it causes lots of tiresome work
> that nobody apart from us seems to care about.

I'd contest this. Whenever Open Source standards come up in a
discussion, Debian is always the gold reference. You know it can be done
right and it is: in Debian.

Having the possibility to look up to that standard and being able to
compare to it is a value and valuable in it self.

That said, Debian doesn't have to hold up that standard if it doesn't
deem it useful of course. Me personally I deem it very valuable. I can
be sure that if stuff lands in Debian then I won't get screwed by weird,
dirty, missleading, underhanded licensing rules, which seems to be the
standard outside the Open Source world and even on its fringes.
*t

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> On 25.03.20 14:43, Christian Kastner wrote:
> 
> > This is not to say that licensing is an unimportant issue -- it clearly
> > is. But our analyze-and-document down-to-the-file approach is on the
> > other extreme end of the spectrum, and it causes lots of tiresome work
> > that nobody apart from us seems to care about.
> 
> I'd contest this. Whenever Open Source standards come up in a
> discussion, Debian is always the gold reference. You know it can be done
> right and it is: in Debian.
Or you can look at the Redhat approach as a minimal working one.
You know it can be done much easier and still work: in Redhat.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Tomas Pospisek]

On 25.03.20 15:19, Andrey Rahmatullin wrote:
> On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
>> On 25.03.20 14:43, Christian Kastner wrote:
>>
>>> This is not to say that licensing is an unimportant issue -- it clearly
>>> is. But our analyze-and-document down-to-the-file approach is on the
>>> other extreme end of the spectrum, and it causes lots of tiresome work
>>> that nobody apart from us seems to care about.
>>
>> I'd contest this. Whenever Open Source standards come up in a
>> discussion, Debian is always the gold reference. You know it can be done
>> right and it is: in Debian.
> Or you can look at the Redhat approach as a minimal working one.
> You know it can be done much easier and still work: in Redhat.

(in case it hasn't already been discussed in this thread, but don't
bother rehashing...): What are they doing differently?

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:32:20PM +0100, Tomas Pospisek wrote:
> On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> >> On 25.03.20 14:43, Christian Kastner wrote:
> >>
> >>> This is not to say that licensing is an unimportant issue -- it clearly
> >>> is. But our analyze-and-document down-to-the-file approach is on the
> >>> other extreme end of the spectrum, and it causes lots of tiresome work
> >>> that nobody apart from us seems to care about.
> >>
> >> I'd contest this. Whenever Open Source standards come up in a
> >> discussion, Debian is always the gold reference. You know it can be done
> >> right and it is: in Debian.
> > Or you can look at the Redhat approach as a minimal working one.
> > You know it can be done much easier and still work: in Redhat.
> 
> (in case it hasn't already been discussed in this thread, but don't
> bother rehashing...): What are they doing differently?
rpm packages record the package license information in a one-line License:
field.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Adam Borowski]

On Wed, Mar 25, 2020 at 03:32:20PM +0100, Tomas Pospisek wrote:
> On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> >> On 25.03.20 14:43, Christian Kastner wrote:
> >>
> >>> This is not to say that licensing is an unimportant issue -- it clearly
> >>> is. But our analyze-and-document down-to-the-file approach is on the
> >>> other extreme end of the spectrum, and it causes lots of tiresome work
> >>> that nobody apart from us seems to care about.
> >>
> >> I'd contest this. Whenever Open Source standards come up in a
> >> discussion, Debian is always the gold reference. You know it can be done
> >> right and it is: in Debian.
> > Or you can look at the Redhat approach as a minimal working one.
> > You know it can be done much easier and still work: in Redhat.
> 
> (in case it hasn't already been discussed in this thread, but don't
> bother rehashing...): What are they doing differently?

Bad: much poorer _technical_ standards.
Good: single-line description of copyright.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀   --  on #linux-sunxi
⠈⠳⣄

Re: What to do when DD considers policy to be optional? [kubernetes]

[Scott Kitterman]

On March 25, 2020 1:43:26 PM UTC, Christian Kastner  wrote:
>Russ,
>
>On 25.03.20 03:25, Russ Allbery wrote:
>> I'll repeat a point that I made earlier but put a bit of a sharper
>point
>> on it: We should thoughtfully question whether the current approach
>to
>> license review that we as a project ask ftpmasters to do is a correct
>> investment of project resources.
>> 
>> The current approach to NEW license review is not a law of nature. 
>It's a
>> choice.  Other choices are possible, such as trusting license
>declarations
>> by upstream and then handling upstream mistakes that are later
>discovered
>> as RC bugs.  The project is much better at sharing the load of
>handling RC
>> bugs than it is at sharing the load of NEW license reviews.
>> 
>> Most of the ecosystems that make extensive use of vendored packages
>also
>> make extensive use of license metadata.  Sometimes that license
>metadata
>> is wrong, and we will have to have a process for dealing with those
>> errors.  But the purpose of Debian is not to be a license checking
>service
>> for the entire free software world.  It's to build a distribution
>adhering
>> to a set of principles but with the understanding that we will
>sometimes
>> make errors and fix them later as bugs, not be perfect and error-free
>at
>> any of our principles, including our licensing principles.  For
>everything
>> other than licenses, we accept the risk that we will incorporate
>upstream
>> errors in our distribution until someone points them out via a bug
>report.
>> 
>> We do not *have* to do a detailed file-by-file review of the
>correctness
>> of upstream's license metadata when packaging.  This is a choice.  By
>> choosing to do this, we absolutely catch bugs... just like we would
>catch
>> bugs if we did a detailed file-by-file review of any other property
>of
>> upstream code.  For many other types of bugs (including ones that
>arguably
>> have a more severe impact on our users than licensing bugs, such as
>> security bugs), we often make trade-off decisions in favor of
>accepting a
>> risk of upstream mistakes and having to fix them later.
>
>if I could +1000 just one post, ever, it's this one.
>
>The vast majority of people developing Open Source Software nowadays
>just publish stuff on GitHub and friends. They fork code, they merge
>other people's contributions, etc.
>
>And at no point can I see those people engage in even a fraction of the
>bureaucracy that we follow when sharing, forking, publishing, and
>contributing code.
>
>This is not to say that licensing is an unimportant issue -- it clearly
>is. But our analyze-and-document down-to-the-file approach is on the
>other extreme end of the spectrum, and it causes lots of tiresome work
>that nobody apart from us seems to care about.
>
>> Other choices are possible, such as trusting license declarations
>> by upstream and then handling upstream mistakes that are later
>discovered
>> as RC bugs.
>
>This would be fantastic. Just fantastic.

This is basically what we do now, so don't get too excited.

The FTP Team review of debian/copyright is about DFSG and upstream license 
compliance.  Most licenses require things like copyright statement preservation 
in binary distribution and debian/copyright is how we do that.  Occasionally, 
in the process of evaluating if Debian would be in compliance with the upstream 
license and if the license meets our DFSG requirements, we detect larger 
issues.  

When I'm reviewing something for New I take upstream's copyright/license 
statements at face value unless there's something too obvious to be ignored.  I 
certainly don't go hunting for reasons not to like upstream licensing 
statements, but sometimes they slap you in the face.

By far the most common case for rejections is incomplete debian/copyright and 
that's unrelated to trusting upstream.  Personally, I don't think the problem 
is that the bar is too high.  Almost all debian/copyright issue I find could 
have been located before upload if the maintainer had simply grep -ir copyright 
* | less and checked the results before uploading.

Scott K
P.S. Speaking only for myself, no hats

Re: What to do when DD considers policy to be optional? [kubernetes]

[Jonas Smedegaard]

Quoting Andrey Rahmatullin (2020-03-25 15:46:10)
> On Wed, Mar 25, 2020 at 03:32:20PM +0100, Tomas Pospisek wrote:
> > On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > > On Wed, Mar 25, 2020 at 03:14:41PM +0100, Tomas Pospisek wrote:
> > >> On 25.03.20 14:43, Christian Kastner wrote:
> > >>
> > >>> This is not to say that licensing is an unimportant issue -- it clearly
> > >>> is. But our analyze-and-document down-to-the-file approach is on the
> > >>> other extreme end of the spectrum, and it causes lots of tiresome work
> > >>> that nobody apart from us seems to care about.
> > >>
> > >> I'd contest this. Whenever Open Source standards come up in a
> > >> discussion, Debian is always the gold reference. You know it can be done
> > >> right and it is: in Debian.
> > > Or you can look at the Redhat approach as a minimal working one.
> > > You know it can be done much easier and still work: in Redhat.
> > 
> > (in case it hasn't already been discussed in this thread, but don't
> > bother rehashing...): What are they doing differently?
> rpm packages record the package license information in a one-line License:
> field.

Is your point that 9 lines can be reduced to one, or that 100 lines can 
be reduced to one?

It is legal in Debian to write debian/copyright files looking like this:

 debian/copyright ==
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: Foo
Upstream-Contact: https://github.com/foo/foo/issues
Source: https://github.com/foo/foo

Files: *
Copyright: Foo Bar
License: WTFPL
 Do what the fuck
 debian/copyright ==

Obviously that is legal only when it actually covers the full licensing 
situation for that source code!

If you are talking about omitting some licensing, then I fail to 
recognize how that can be a "gold standard" which I believe is what you 
claimed above.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:57:29PM +0100, Jonas Smedegaard wrote:
> > > >> I'd contest this. Whenever Open Source standards come up in a
> > > >> discussion, Debian is always the gold reference. You know it can be 
> > > >> done
> > > >> right and it is: in Debian.
> > > > Or you can look at the Redhat approach as a minimal working one.
> > > > You know it can be done much easier and still work: in Redhat.
> > > 
> > > (in case it hasn't already been discussed in this thread, but don't
> > > bother rehashing...): What are they doing differently?
> > rpm packages record the package license information in a one-line License:
> > field.
> 
> Is your point that 9 lines can be reduced to one, or that 100 lines can 
> be reduced to one?
I'll repeat my point:

You know it can be done much easier and still work: in Redhat.

> If you are talking about omitting some licensing, then I fail to 
> recognize how that can be a "gold standard" which I believe is what you 
> claimed above.
I'm just saying that there is an approach of providing a gold standard and
there are other approaches.
The project decides what to provide.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Christian Kastner]

On 25.03.20 15:14, Tomas Pospisek wrote:
> I can be sure that if stuff lands in Debian then I won't get screwed
> by weird, dirty, missleading, underhanded licensing rules, which
> seems to be the standard outside the Open Source world and even on
> its fringes.
The only thing you can be sure about is that the declared license is up
to Debian standards.

No such thing can be said about most of the code itself, unless you have
an actual audit trail documenting its origin.

To use an extreme example: me grabbing code from $EMPLOYER and slapping
the GPL onto it, doesn't make it GPL.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Florian Weimer]

* Andrey Rahmatullin:

> Or you can look at the Redhat approach as a minimal working one.
> You know it can be done much easier and still work: in Redhat.

I think you are referring to a Fedora process, not a Red Hat process.
The Red Hat process does not seem much simpler than what ftpmaster are
doing, as far as I can tell.

Re: length of Debian copyright files

[Simon McVittie]

On Wed, 25 Mar 2020 at 15:32:20 +0100, Tomas Pospisek wrote:
> On 25.03.20 15:19, Andrey Rahmatullin wrote:
> > Or you can look at the Redhat approach as a minimal working one.
> > You know it can be done much easier and still work: in Redhat.
> 
> (in case it hasn't already been discussed in this thread, but don't
> bother rehashing...): What are they doing differently?

Here is a concrete example of a medium-to-large package with a relatively
typical licensing situation (LGPL plus some more-permissive bits): GTK 4,
for which I redid the copyright file somewhat recently in preparation
for getting it through NEW.

Debian:
https://tracker.debian.org/media/packages/g/gtk%2B4.0/copyright-3.98.0-1
plus we ship the LGPL in base-files' common-licenses.

Fedora:
one line in gtk4.spec "License: LGPLv2+", plus they ship these
files in their binary package:
https://gitlab.gnome.org/GNOME/gtk/-/blob/master/AUTHORS
https://gitlab.gnome.org/GNOME/gtk/-/blob/master/COPYING (it's the LGPL)

I'm pretty sure the long list of maybe-copyright-holders in the Debian
package is still incomplete; merge requests welcome. I'm also fairly sure
we are the only distribution that does this (not counting our derivatives
like Ubuntu), even though some of the others have lawyers.

If people have recommendations for how to make gtk+4.0's copyright file
more minimal, I'd be happy to review merge requests or otherwise receive
suggestions.

I think part of the problem might be this vicious cycle: the NEW queue
is an asynchronous gatekeeper/progress blocker, which gives maintainers
a strong incentive to get things accepted first time (because a NEW
rejection will double the delay), which means maintainers are incentivized
to dot every i and cross every t in the copyright file even if it isn't
strictly necessary, which means the ftp team are given larger and more
verbose copyright files to read, which presumably means the NEW queue
takes longer than it otherwise could.

smcv

Re: What to do when DD considers policy to be optional? [kubernetes]

[Pierre-Elliott Bécue]

Le mardi 24 mars 2020 à 19:08:23+, Janos LENART a écrit :
> [snip]
> I do think there is a good case for Kubernetes to be an exception from 4.13 
> for
> now, just like other Go packages effectively are.

Using other examples to say "people do bas stuff elsewhere so I'm
entitled to do the same" is not a good practice.

And, really, comparing uploads with 4, 6, 10 bundles libraries (bad)
with an upload of a package with TWO HUNDRED libraries??? You shouldn't
have done that hoping that it'd slip through.

> It is a massively popular project topped only by the Linux kernel. We
> cannot afford not to have up to date versions in Debian, or have forks
> that no one can use.

Yes we can afford not to have an up to date version in Debian. We could
also afford not having Kubernetes into Debian.

And the fact that the project is popular is not a good reason either.

I'm sorry, but the way you do things and try to justify yourself
afterwards is not really something I find normal for a DD who commited
themselves to the standard the project defined as its own.

-- 
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.


signature.asc
Description: PGP signature

Re: length of Debian copyright files

[Sean Whitton]

Hello,

On Wed 25 Mar 2020 at 03:43PM +00, Simon McVittie wrote:

> I think part of the problem might be this vicious cycle: the NEW queue
> is an asynchronous gatekeeper/progress blocker, which gives maintainers
> a strong incentive to get things accepted first time (because a NEW
> rejection will double the delay), which means maintainers are incentivized
> to dot every i and cross every t in the copyright file even if it isn't
> strictly necessary, which means the ftp team are given larger and more
> verbose copyright files to read, which presumably means the NEW queue
> takes longer than it otherwise could.

Yes, I think this is exactly what happens.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: length of Debian copyright files

[Andrey Rahmatullin]

On Wed, Mar 25, 2020 at 03:43:17PM +, Simon McVittie wrote:
> I think part of the problem might be this vicious cycle: the NEW queue
> is an asynchronous gatekeeper/progress blocker, which gives maintainers
> a strong incentive to get things accepted first time (because a NEW
> rejection will double the delay), which means maintainers are incentivized
> to dot every i and cross every t in the copyright file even if it isn't
> strictly necessary, which means the ftp team are given larger and more
> verbose copyright files to read, which presumably means the NEW queue
> takes longer than it otherwise could.
Do you mean it's not essential to track and list all licenses and
copyrights? IIRC only one simplification is permitted and I don't even
remember which one is it (maybe combining copyright years and names into
one entry? or just years?).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: length of Debian copyright files

[Sean Whitton]

Hello,

On Wed 25 Mar 2020 at 08:58PM +05, Andrey Rahmatullin wrote:

> On Wed, Mar 25, 2020 at 03:43:17PM +, Simon McVittie wrote:
>> I think part of the problem might be this vicious cycle: the NEW queue
>> is an asynchronous gatekeeper/progress blocker, which gives maintainers
>> a strong incentive to get things accepted first time (because a NEW
>> rejection will double the delay), which means maintainers are incentivized
>> to dot every i and cross every t in the copyright file even if it isn't
>> strictly necessary, which means the ftp team are given larger and more
>> verbose copyright files to read, which presumably means the NEW queue
>> takes longer than it otherwise could.
> Do you mean it's not essential to track and list all licenses and
> copyrights? IIRC only one simplification is permitted and I don't even
> remember which one is it (maybe combining copyright years and names into
> one entry? or just years?).

With the machine-readable format, you can combine copyright years and
names for files under the same license, yes.

Policy currently requires you to include all copyright claims and all
licenses (§§ 2.3, 4.5, 12.5).

If a d/copyright doesn't include all copyright claims but it's not a
license violation -- some licenses require they all be included in
binary distributions but some don't -- the FTP team typically ACCEPT but
file an RC bug citing Policy.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Marc Haber]

On Tue, 24 Mar 2020 20:37:16 +, Jeremy Stanley 
wrote:
>If this represents the actual state of building Kubernetes, it's
>unclear to me why Debian would package it at all. I don't see the
>value to users in consuming Kubernetes from a Debian package if the
>result is compromising on Debian's vision and values so that they
>can get the exact same thing they'd have if they just used the
>Kubernetes community's recommended tooling to install it instead.
>I'm all for using the best tool for the job, and while I've been a
>die-hard Debian user for more than two decades I also don't install
>every last bit of software from its packages. Some software
>ecosystems have chosen to focus on tools and workflows which are
>incompatible with Debian's, but that doesn't mean that either one is
>inherently bad nor that they need to be integrated at all costs.

Software packages like kubernetes, docker, and many of the other "hip
tools of the day" are moving way too fast for our release scheme.
Additionally, many communities and developers stop caring for their
software once it has cleared the door and laugh upon people who are
using anything but their latest release.

I think we're not doing the world a favor packaging those software and
releasing it with our stable release, making our users appear on
community support channels with a version that is many months old and
will only trigger a "first, upgrade to our current version, check
whether your issue still exists and then come back" answer from the
upstream community. In the long run, people who don't know how our
releases work will make their judgement about Debian being "always out
of date", worsening our reputation.

We do provide a rock solid base system where people can install their
software, in the case of those fast moving things in the way upstream
wants them to. This causes a much better result for all those
involved.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: What to do when DD considers policy to be optional? [kubernetes]

[Vincent Bernat]

 ❦ 25 mars 2020 15:57 +01, Jonas Smedegaard:

>> rpm packages record the package license information in a one-line License:
>> field.
>
> Is your point that 9 lines can be reduced to one, or that 100 lines can 
> be reduced to one?
>
> It is legal in Debian to write debian/copyright files looking like
> this:

Redhat also drops the license files in /usr/share/licenses/packagename
without much structure.
-- 
Don't over-comment.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: length of Debian copyright files

[Simon McVittie]

On Wed, 25 Mar 2020 at 09:04:52 -0700, Sean Whitton wrote:
> On Wed 25 Mar 2020 at 08:58PM +05, Andrey Rahmatullin wrote:
> > On Wed, Mar 25, 2020 at 03:43:17PM +, Simon McVittie wrote:
> >> maintainers are incentivized
> >> to dot every i and cross every t in the copyright file even if it isn't
> >> strictly necessary
> >>
> > Do you mean it's not essential to track and list all licenses and
> > copyrights?

Policy just says "a verbatim copy of its copyright information and
distribution license", which is not really enough to answer your question
either way. The ftp team are responsible for interpreting this part
of Policy in order to accept or reject packages, and have made various
announcements about what is or isn't acceptable.

One thing that the ftp team clarified somewhat recently is that in
most cases, we must track all the copyright notices that exist in the
upstream source, and copy them into d/copyright.
https://lists.debian.org/debian-devel-announce/2018/10/msg4.html

#904729 is a related Policy bug that would benefit from ftp team input.

> > IIRC only one simplification is permitted and I don't even
> > remember which one is it (maybe combining copyright years and names into
> > one entry? or just years?).
> 
> With the machine-readable format, you can combine copyright years and
> names for files under the same license, yes.

Strictly speaking, I am not aware of the ftp team having said this is
acceptable - although I've had packages ACCEPTed where I did this, so
presumably it must be (and the copyright file for non-trivial packages
would become even larger, and presumably more frustrating for the ftp team
to review, if this was considered to be unacceptable).

Can you also combine licenses, like this real-life example from gtk+4.0?

Files: *
Copyright:
 2009-2010 A S Alam
 (... 380 other copyright holders ...)
License: LGPL-2+ and LGPL-2.1+

(some files are LGPL-2+, some are LGPL-2.1+, keeping track of which ones
significantly increases the length and maintenance cost of the file for
what seems to be little or no benefit)

or even this?

Files: *
Copyright:
 2009-2010 A S Alam
 (... around 390 other copyright holders ...)
License: LGPL-2+ and LGPL-2.1+ and CC0-1.0 and (Expat or unlicense) and ...

(some files are LGPL-2+, some are LGPL-2.1+, some are under one of the
permissive licenses mentioned)

Rationale for wanting to do this: I suspect that for our purposes it
doesn't actually matter which individual files are under which permissive
licenses, as long as we document each license that is applicable, and
each copyright holder. The license that we, and our users, actually
have to obey when dealing with the source and binary packages is the
intersection of all the applicable licenses.

smcv

Re: length of Debian copyright files

[Keith Packard]

Simon McVittie  writes:

> One thing that the ftp team clarified somewhat recently is that in
> most cases, we must track all the copyright notices that exist in the
> upstream source, and copy them into d/copyright.

As an example, I've got a package in the new queue with a 5077 line
copyright file, with 75 'unique' licenses (BSD licensed software picks
up a lot of variation over thirty years).

-- 
-keith


signature.asc
Description: PGP signature

Bug#954951: ITP: libcompiler-lexer-perl -- Lexical Analyzer for Perl5

[Dominique Dumont]

Package: wnpp
Owner: Dominique Dumont 
Severity: wishlist
X-Debbugs-CC: debian-devel@lists.debian.org, debian-p...@lists.debian.org

* Package name: libcompiler-lexer-perl
  Version : 0.23
  Upstream Author : Masaaki Goshima (goccy) 
* URL : https://metacpan.org/release/Compiler-Lexer
* License : Artistic or GPL-1+
  Programming Lang: Perl
  Description : Lexical Analyzer for Perl5

Compiler::Lexer provides a lexical analyser for Perl code. Once the Perl
code is parsed, the lexer can provide:
 * a list of tokens
 * the list of module used by the Perl code.

This module is written in XS to be faster than a pure Perl
implementation.

This module is a dependency of libdist-zilla-plugin-minimumperlfast-perl which 
I plan to package.

The package will be maintained under the umbrella of the Debian Perl Group.

Re: length of Debian copyright files

[Jonas Smedegaard]

Quoting Keith Packard (2020-03-25 19:07:33)
> Simon McVittie  writes:
> 
> > One thing that the ftp team clarified somewhat recently is that in 
> > most cases, we must track all the copyright notices that exist in 
> > the upstream source, and copy them into d/copyright.
> 
> As an example, I've got a package in the new queue with a 5077 line 
> copyright file, with 75 'unique' licenses (BSD licensed software picks 
> up a lot of variation over thirty years).

Wauw, that sounds like a great challenge for licensecheck.

Looking at https://ftp-master.debian.org/new.html it seems to be 
picolibc.  Thanks for that!

If anyone else have good examples of complex source packages, please do 
let me know.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: length of Debian copyright files

[Sean Whitton]

Hello Simon,

Not speaking for the whole FTP Team in this mail, but maybe I can help a
bit.

On Wed 25 Mar 2020 at 05:47PM +00, Simon McVittie wrote:

> On Wed, 25 Mar 2020 at 09:04:52 -0700, Sean Whitton wrote:
>> On Wed 25 Mar 2020 at 08:58PM +05, Andrey Rahmatullin wrote:
>> > On Wed, Mar 25, 2020 at 03:43:17PM +, Simon McVittie wrote:
>> >> maintainers are incentivized
>> >> to dot every i and cross every t in the copyright file even if it isn't
>> >> strictly necessary
>> >>
>> > Do you mean it's not essential to track and list all licenses and
>> > copyrights?
>
> Policy just says "a verbatim copy of its copyright information and
> distribution license", which is not really enough to answer your question
> either way. The ftp team are responsible for interpreting this part
> of Policy in order to accept or reject packages, and have made various
> announcements about what is or isn't acceptable.
>
> One thing that the ftp team clarified somewhat recently is that in
> most cases, we must track all the copyright notices that exist in the
> upstream source, and copy them into d/copyright.
> https://lists.debian.org/debian-devel-announce/2018/10/msg4.html

I am not sure that it is quite right to understand the FTP Team as
interpreting that particular part of Policy (anymore than any reader of
Policy is involved in intrepreting it), because Policy currently
requires strictly more than the FTP Team require.

For example, if a package's license does not require all copyright
notices to be included in binary distributions, and some are missing, we
may well ACCEPT and file an RC bug, citing Policy.

>> > IIRC only one simplification is permitted and I don't even
>> > remember which one is it (maybe combining copyright years and names into
>> > one entry? or just years?).
>>
>> With the machine-readable format, you can combine copyright years and
>> names for files under the same license, yes.
>
> Strictly speaking, I am not aware of the ftp team having said this is
> acceptable - although I've had packages ACCEPTed where I did this, so
> presumably it must be (and the copyright file for non-trivial packages
> would become even larger, and presumably more frustrating for the ftp team
> to review, if this was considered to be unacceptable).
>
> Can you also combine licenses, like this real-life example from gtk+4.0?
>
> Files: *
> Copyright:
>  2009-2010 A S Alam
>  (... 380 other copyright holders ...)
> License: LGPL-2+ and LGPL-2.1+
>
> (some files are LGPL-2+, some are LGPL-2.1+, keeping track of which ones
> significantly increases the length and maintenance cost of the file for
> what seems to be little or no benefit)
>
> or even this?
>
> Files: *
> Copyright:
>  2009-2010 A S Alam
>  (... around 390 other copyright holders ...)
> License: LGPL-2+ and LGPL-2.1+ and CC0-1.0 and (Expat or unlicense) and 
> ...
>
> (some files are LGPL-2+, some are LGPL-2.1+, some are under one of the
> permissive licenses mentioned)
>
> Rationale for wanting to do this: I suspect that for our purposes it
> doesn't actually matter which individual files are under which permissive
> licenses, as long as we document each license that is applicable, and
> each copyright holder. The license that we, and our users, actually
> have to obey when dealing with the source and binary packages is the
> intersection of all the applicable licenses.

I do not believe that you would get a REJECT where the combination
involves a single license in the License: field.

Your case, to be specific, is
- using "and" to combine license shortnames in the License: field,
- where the code under different licenses is in different files.

(The uncontroversial case of using "and" in License: is when there is
code under different licenses in a single file.)

I am not sure there exists a consensus within the FTP Team about this
sort of case and I agree that it would be useful to develop such a
consensus.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: What to do when DD considers policy to be optional? [kubernetes]

[Bernd Zeimetz]

Hi,

> As a person who originally introduced Kubernetes to Debian I can say that it 
> is indeed a lot of work to maintain this package and to reuse packaged 
> libraries. But I've demonstrated that it is possible at least to some extent.

if you've maintained k8s you would know that it is only tested with the
exact versions of the libraries as it was released with.

Choosing different versions is prone to introduce subtle bugs and should
never be done and accepted if that k8s version is supposed to be ready
for production usage.

What you suggest is a nice idea, but hard for go sources and impossible
for packages like k8s.

The DD you called inexperienced has done everything right.


Bernd

-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: new kubernetes packaging

[Moritz Mühlenhoff]

Sean Whitton  wrote:
> I am not sure, however, that your argument applies to security updates
> to our stable releases.  These updates are almost always a matter of
> backporting small fixes, rather than updating to new upstream releases.
> And for backported fixes, vendoring makes things much harder.

In the case of kubernetes it will most certainly make security updates
easier, not more complex. For an application like kubernetes there'll
be a steady stream of security releases and if some of these also rebase
to a fixed, vendored Go "library" that doesn't any extra effort.

It's very similar to e.g. Chromium (and to some extent Firefox) which
also frequently fix issues in bundled libraries, but it's always just
one more bug in a bigger update pile.

I have some concerns whether the fast-paced kubernetes release cadence
will be workable for Debian's release cycles, but I think Janos' tradeoffs
seems fair for packaging kubernetes.

Cheers,
Moritz

Re: What to do when DD considers policy to be optional? [kubernetes]

[Moritz Mühlenhoff]

Russ Allbery  schrieb:
> Michael Lustfield  writes:
>
>> One last thing to consider... NEW reviews are already an intense
>> process. If this package hit NEW /and/ we allowed vendored libs, you
>> could safely expect me to never complete that particular review. I doubt
>> I'm the only one; that's essentially ~200 package reviews wrapped into
>> 1.
>
> I'll repeat a point that I made earlier but put a bit of a sharper point
> on it: We should thoughtfully question whether the current approach to
> license review that we as a project ask ftpmasters to do is a correct
> investment of project resources.

Full ack!

> We do not *have* to do a detailed file-by-file review of the correctness
> of upstream's license metadata when packaging.  This is a choice.  By
> choosing to do this, we absolutely catch bugs... just like we would catch
> bugs if we did a detailed file-by-file review of any other property of
> upstream code.

Or even replace it with automated license detection to spot such bugs (as
provided by tools like Fossology), which could even be an ongoing thing
for every upload instead of "once for the initial upload" and "randomly 
when new new binary  packages" appear. Plus everyone keen on reviewing
copyright files is always able to report bugs in the BTS.

Cheers,
Moritz

Re: email backend for fedmsg

[clime]

>
> 
> I work in telecom for meteorology, and we ended up with a general method
> for file copying (catchphrase: rsync on steroids*.) ( *every catchphrase is
> a distortion, no dis to rsync, but in certain cases we do work much faster,
> it just communicates the idea.) Sarracenia (
> https://github.com/MetPX/Sarracenia) is a GPL2 app (Python and C
> implementations) that use mozilla public license rabbitmq broker, as well
> as openssh and/or any web server to do fastish file synching, and/or
> processing/orchestration. The app is just json messages with file metadata
> sent through the broker. Then you daisy chain brokers through clients.  No
> centralization (every entity installs their own broker), No federated
> identity required (authentication is to each broker, but they can pass
> files/messages to each other.)
> A firstish thing to do with it would be to sync the debian mirrors in
> real-time rather than periodically.  Each mirror has a broker, they get
> advertisements (AMQP messages containing JSON file metadata) download the
> corresponding file, and re-advertise (publish on the local broker with the
> local file URL) for downstream clients. You can then make a mesh of
> mirrors, where, if each mirror is subscribed to at least two others, then
> it can withstand the failure of any node.  If you add more connections, you
> increase redundancy.
> Once you have that sort of anchor tenant for an AMQP message bus, people
> might want to use it to provide other forms of automation, but way quicker
> and in some ways much simpler than SMTP.  but yeah... SMTP is a lot more
> well-known/common. RabbitMQ is the industry dominant open solution for AMQP
> brokers. sounds like marketing bs, but if you look around it is what the
> vast majority are using, and there are thousands upon thousands of
> deployments. It's a much more viable starting point, for stability, and a
> lot less assembly required to get something going. Sarracenia makes it a
> bit easier again, but messages are kind of alien and different, so it takes
> a while to get used to them.
> 


Peter, I like the solution and for the mirrors it sounds great but I have a
few nitpicks:

- the file syncing part is makes a perfect sense for the debian mirrors but
in general case you might only want to send a message and skip the file
syncing part
- I am currently, personally more intrigued by even more standard
technologies than RabbitMQ and I believe that a good solution might lie
there

What I particularly like about Sarracenia is that it is decentralized
because each host has its own broker - that I think is cool and I would
like to potentially do something similar...

clime



On Wed, 25 Mar 2020 at 01:07, clime  wrote:

> On Wed, 25 Mar 2020 at 01:00, clime  wrote:
> >
> > On Tue, 24 Mar 2020 at 22:45, Nicolas Dandrimont 
> wrote:
> > >
> > > On Tue, Mar 24, 2020, at 21:51, clime wrote:
> > > > On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont 
> wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > On Sun, Mar 22, 2020, at 13:06, clime wrote:
> > > > > > Hello!
> > > > > >
> > > > > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html
> -
> > > > > > fedmsg usage in Debian.
> > > > > >
> > > > > > There is a note: "it seems that people actually like parsing
> emails"
> > > > >
> > > > > This was just a way to say that fedmsg never got much of a user
> base in the services that run on Debian infra, and that even the new
> services introduced at the time kept parsing emails.
> > > >
> > > > Hello Nicolas!
> > > >
> > > > Do you remember some such service and how it used email parsing
> specifically?
> > >
> > > I believe that tracker.debian.org was introduced around that time.
> > >
> > > At the point it was created, tracker.d.o was mostly consuming emails
> from packages.debian.org to update its data. These days tracker.d.o has
> replaced packages.d.o as "email router", in that it receives all the mails
> from services (e.g. the BTS, the archive maintenance software, buildds,
> salsa webhooks, ...) and forwards them to the public.
> > >
> > > > I am still a bit unclear how email parsing is used in Debian
> > > > infrastructure, don't get me wrong, I find it elegant
> > >
> > > Ha. I find that it's a big mess.
> > >
> > > Here's the set of headers of a message I received today from
> tracker.d.o, which are supposed to make parsing these emails better:
> > >
> > > X-PTS-Approved: yes
> > > X-Distro-Tracker-Package: facter
> > > X-Distro-Tracker-Keyword: derivatives
> > > X-Remote-Delivered-To: dispa...@tracker.debian.org
> > > X-Loop: dispa...@tracker.debian.org
> > > X-Distro-Tracker-Keyword: derivatives
> > > X-Distro-Tracker-Package: facter
> > > List-Id: 
> > > X-Debian: tracker.debian.org
> > > X-Debian-Package: facter
> > > X-PTS-Package: facter
> > > X-PTS-Keyword: derivatives
> > > Precedence: list
> > > List-Unsubscribe:  ?body=unsubscribe%20facter>
> > >
> > > I'll leave you to judge whether this

Re: email backend for fedmsg

[clime]

I was looking into the email approach more and maybe I found a few
improvements:

- each communicating agent has an exim instance
- there are a few dns servers that replicate their configuration between
each other (to provide redundancy)
- these servers store subscriptions of the agents for each topic probably
as TXT records
- they have rest API to manage those subscriptions from the agents (so that
a service can register itself at its startup)
- when a message should be sent to another agent, first dns lookup is made
to obtain recipients based on a topic of the message and the subscriptions
stored in dns for that topic
- when recipients are fetched emails are sent to recipients in peer-to-peer
fashion

Now, I am not sure if it would be possible to configure Exim for this
role...

On Wed, 25 Mar 2020 at 22:56, clime  wrote:

> 
>> I work in telecom for meteorology, and we ended up with a general method
>> for file copying (catchphrase: rsync on steroids*.) ( *every catchphrase is
>> a distortion, no dis to rsync, but in certain cases we do work much faster,
>> it just communicates the idea.) Sarracenia (
>> https://github.com/MetPX/Sarracenia) is a GPL2 app (Python and C
>> implementations) that use mozilla public license rabbitmq broker, as well
>> as openssh and/or any web server to do fastish file synching, and/or
>> processing/orchestration. The app is just json messages with file metadata
>> sent through the broker. Then you daisy chain brokers through clients.  No
>> centralization (every entity installs their own broker), No federated
>> identity required (authentication is to each broker, but they can pass
>> files/messages to each other.)
>> A firstish thing to do with it would be to sync the debian mirrors in
>> real-time rather than periodically.  Each mirror has a broker, they get
>> advertisements (AMQP messages containing JSON file metadata) download the
>> corresponding file, and re-advertise (publish on the local broker with the
>> local file URL) for downstream clients. You can then make a mesh of
>> mirrors, where, if each mirror is subscribed to at least two others, then
>> it can withstand the failure of any node.  If you add more connections, you
>> increase redundancy.
>> Once you have that sort of anchor tenant for an AMQP message bus, people
>> might want to use it to provide other forms of automation, but way quicker
>> and in some ways much simpler than SMTP.  but yeah... SMTP is a lot more
>> well-known/common. RabbitMQ is the industry dominant open solution for AMQP
>> brokers. sounds like marketing bs, but if you look around it is what the
>> vast majority are using, and there are thousands upon thousands of
>> deployments. It's a much more viable starting point, for stability, and a
>> lot less assembly required to get something going. Sarracenia makes it a
>> bit easier again, but messages are kind of alien and different, so it takes
>> a while to get used to them.
>> 
>
>
> Peter, I like the solution and for the mirrors it sounds great but I have
> a few nitpicks:
>
> - the file syncing part is makes a perfect sense for the debian mirrors
> but in general case you might only want to send a message and skip the file
> syncing part
> - I am currently, personally more intrigued by even more standard
> technologies than RabbitMQ and I believe that a good solution might lie
> there
>
> What I particularly like about Sarracenia is that it is decentralized
> because each host has its own broker - that I think is cool and I would
> like to potentially do something similar...
>
> clime
>
>
>
> On Wed, 25 Mar 2020 at 01:07, clime  wrote:
>
>> On Wed, 25 Mar 2020 at 01:00, clime  wrote:
>> >
>> > On Tue, 24 Mar 2020 at 22:45, Nicolas Dandrimont 
>> wrote:
>> > >
>> > > On Tue, Mar 24, 2020, at 21:51, clime wrote:
>> > > > On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont 
>> wrote:
>> > > > >
>> > > > > Hi!
>> > > > >
>> > > > > On Sun, Mar 22, 2020, at 13:06, clime wrote:
>> > > > > > Hello!
>> > > > > >
>> > > > > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html
>> -
>> > > > > > fedmsg usage in Debian.
>> > > > > >
>> > > > > > There is a note: "it seems that people actually like parsing
>> emails"
>> > > > >
>> > > > > This was just a way to say that fedmsg never got much of a user
>> base in the services that run on Debian infra, and that even the new
>> services introduced at the time kept parsing emails.
>> > > >
>> > > > Hello Nicolas!
>> > > >
>> > > > Do you remember some such service and how it used email parsing
>> specifically?
>> > >
>> > > I believe that tracker.debian.org was introduced around that time.
>> > >
>> > > At the point it was created, tracker.d.o was mostly consuming emails
>> from packages.debian.org to update its data. These days tracker.d.o has
>> replaced packages.d.o as "email router", in that it receives all the mails
>> from services (e.g. the BTS, the archive maintenance software, buildds,
>> salsa webhooks, ...) and forwards them to 

Re: email backend for fedmsg

[Peter Silva]

Most Sarrracenia stuff is tied to AMQP, but next-gen messages are called
v03 (version 3) they use a JSON payload
for all the information, and that makes it somewhat protocol independent.
There is also a 500 line MQTT demo
that implements a file replication network, using the same JSON messages,
and primed from an AMQP upstream.

https://github.com/MetPX/wmo_mesh

the peer code there is just a demonstration prototype, but it processes the
messages the same way as real Sarracenia.

That code has been run against mosquitto and EMQT, and I think another
broker, I forget... It worked without issues on all of them. MQTT interop
is flawless afaict.   note: we were using v3.  Have not played with v5.

Sarracenia essentially defines a JSON payload for advertising that a file
exists. That is a fairly popular problem, but if your problem isn´t that,
then you should define a different payload.  It could be used for file
replication, or orchestration/workload co-ordination, or other things in
the IFTTT style... but in the end, this is just one application of a
message bus, it doesn´t need to encompass all applications, but is a good
way to get a useful thing implemented with it, so people see that it is
useful.   I think applications need to define their messages, and trying to
be too general makes them harder to understand and apply.


On Wed, Mar 25, 2020 at 5:57 PM clime  wrote:

> 
>> I work in telecom for meteorology, and we ended up with a general method
>> for file copying (catchphrase: rsync on steroids*.) ( *every catchphrase is
>> a distortion, no dis to rsync, but in certain cases we do work much faster,
>> it just communicates the idea.) Sarracenia (
>> https://github.com/MetPX/Sarracenia) is a GPL2 app (Python and C
>> implementations) that use mozilla public license rabbitmq broker, as well
>> as openssh and/or any web server to do fastish file synching, and/or
>> processing/orchestration. The app is just json messages with file metadata
>> sent through the broker. Then you daisy chain brokers through clients.  No
>> centralization (every entity installs their own broker), No federated
>> identity required (authentication is to each broker, but they can pass
>> files/messages to each other.)
>> A firstish thing to do with it would be to sync the debian mirrors in
>> real-time rather than periodically.  Each mirror has a broker, they get
>> advertisements (AMQP messages containing JSON file metadata) download the
>> corresponding file, and re-advertise (publish on the local broker with the
>> local file URL) for downstream clients. You can then make a mesh of
>> mirrors, where, if each mirror is subscribed to at least two others, then
>> it can withstand the failure of any node.  If you add more connections, you
>> increase redundancy.
>> Once you have that sort of anchor tenant for an AMQP message bus, people
>> might want to use it to provide other forms of automation, but way quicker
>> and in some ways much simpler than SMTP.  but yeah... SMTP is a lot more
>> well-known/common. RabbitMQ is the industry dominant open solution for AMQP
>> brokers. sounds like marketing bs, but if you look around it is what the
>> vast majority are using, and there are thousands upon thousands of
>> deployments. It's a much more viable starting point, for stability, and a
>> lot less assembly required to get something going. Sarracenia makes it a
>> bit easier again, but messages are kind of alien and different, so it takes
>> a while to get used to them.
>> 
>
>
> Peter, I like the solution and for the mirrors it sounds great but I have
> a few nitpicks:
>
> - the file syncing part is makes a perfect sense for the debian mirrors
> but in general case you might only want to send a message and skip the file
> syncing part
> - I am currently, personally more intrigued by even more standard
> technologies than RabbitMQ and I believe that a good solution might lie
> there
>
> What I particularly like about Sarracenia is that it is decentralized
> because each host has its own broker - that I think is cool and I would
> like to potentially do something similar...
>
> clime
>
>
>
> On Wed, 25 Mar 2020 at 01:07, clime  wrote:
>
>> On Wed, 25 Mar 2020 at 01:00, clime  wrote:
>> >
>> > On Tue, 24 Mar 2020 at 22:45, Nicolas Dandrimont 
>> wrote:
>> > >
>> > > On Tue, Mar 24, 2020, at 21:51, clime wrote:
>> > > > On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont 
>> wrote:
>> > > > >
>> > > > > Hi!
>> > > > >
>> > > > > On Sun, Mar 22, 2020, at 13:06, clime wrote:
>> > > > > > Hello!
>> > > > > >
>> > > > > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html
>> -
>> > > > > > fedmsg usage in Debian.
>> > > > > >
>> > > > > > There is a note: "it seems that people actually like parsing
>> emails"
>> > > > >
>> > > > > This was just a way to say that fedmsg never got much of a user
>> base in the services that run on Debian infra, and that even the new
>> services introduced at the time kept parsing emails.
>> > > >
>> >

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Thursday, 26 March 2020 7:38:12 AM AEDT Bernd Zeimetz wrote:
> ... you would know that it is only tested with the
> exact versions of the libraries as it was released with.
> 
> Choosing different versions is prone to introduce subtle bugs and should
> never be done and accepted if that k8s version is supposed to be ready
> for production usage.

Given that logic even re-compiling using different compiler would not be 
trustworthy. And indeed some people make exactly that argument -- "use our 
tested binary" as one can't be sure if re-compiling introduces any bugs.

That questions the very usability of source code releases, whether you 
understand it or not.

With this and your next arguments you are questioning the very usability of 
packaging and I might agree that Kubernetes may not be worth packaged, 
especially if we can't do it properly.


> What you suggest is a nice idea, but hard for go sources and impossible
> for packages like k8s.

I don't know how to respond nicely when someone who did not maintain a 
complex Golang application tells me that the way myself and others maintain 
packages like docker,io, consul, nomad, vault, syncthing is "impossible"...


> The DD you called inexperienced has done everything right.

If you are not with me on technical reasoning, at least you have to recognise 
that he circumvented FTP-masters policy: I'm having a hard time getting a 
package with several vendored libraries accepted through NEW process (e.g. 
Podman was rejected due to "many embedded packages in vendor/" with only 6 or 
7 private libs versus 120 libraries removed in favour of packaged ones).

Thinking that compliance with NEW acceptance practices and standards is only 
a one time thing for a first upload (when subsequently you can re-introduce 
hundreds of private libraries at your discretion) is disrespectful to ftp-
masters team and to established practices.

We have to conclude that either we accept the terms for future uploads or not 
package at all.

-- 
Cheers,
 Dmitry Smirnov.

---

We have now sunk to a depth at which restatement of the obvious is the
first duty of intelligent men.
-- George Orwell


signature.asc
Description: This is a digitally signed message part.

Re: What to do when DD considers policy to be optional? [kubernetes]

[Dmitry Smirnov]

On Thursday, 26 March 2020 3:45:21 AM AEDT Marc Haber wrote:
> Software packages like kubernetes, docker, and many of the other "hip
> tools of the day" are moving way too fast for our release scheme.

It is worth remembering that Debian is not only a stable release.
Statically built Golang apps are easy to install from testing/unstable into 
stable and packaging still provides certain benefits.

Stable Debian release protects users from sudden unexpected and disruptive 
changes but some prefer to run their systems from "testing".

With rapidly changing upstream projects users can't have peace anywhere 
neither in Debian nor with upstream.

Also most complex Golang apps usually don't even get into "stable" due to 
massive removals from "testing" on bugs in any of their dependency libraries 
(direct or indirect). Stabilising the whole Golang ecosystem so few heavy 
packages could migrate is a long effort but we are getting there...

-- 
All the best,
 Dmitry Smirnov.

---

Few people are capable of expressing with equanimity opinions which
differ from the prejudices of their social environment. Most people are
even incapable of forming such opinion.
-- Albert Einstein, from "Aphorisms for Leo Baeck;
   Opinions of Albert Einstein"


signature.asc
Description: This is a digitally signed message part.