Re: Adding support for LZIP to dpkg, using that instead of xz, archive wide

[Thomas Goirand]

Guillem,

First, thanks for your reply and taking the time to reply on every
point. This really is helpful.

While I believe all of your argumentation is correct, I am still not
convince about the reproducibility, which is my main issue here. Could
you please reply to that point, and that one only? I've removed from the
quote all what doesn't concern it, because it is my feeling that it is a
distraction in this thread.

On 06/14/2015 05:46 AM, Guillem Jover wrote:
> Hi,
> 
> On Sun, 2015-06-14 at 01:08:29 +0200, Thomas Goirand wrote:
>> On 06/13/2015 10:55 AM, Paul Wise wrote:
>>> On Sat, Jun 13, 2015 at 4:23 PM, Thomas Goirand wrote:
 I've been using xz compression for a long time, but I see a big defect
 which is today pushing me to turn it off for the .orig.tar file. The
 issue is that depending on the version of xz-utils, it produces a
 different output.
> 
> Well if you want reproducible output, then use the same tool version.

That's not possible: Jessie, Sid and Trusty don't have the same version,
and we need to generate the orig.tar file in all of them. The
contributors for the Debian OpenStack packaging are mostly using Ubuntu,
and they need to keep a workflow with the orig.tar file in the Git
repository.

I did tell them to just get the file from the Debian archive, but it
doesn't work. One of the reason it doesn't is because sometimes, they
upload first to Ubuntu, and then I do in Debian, and we end up with
different orig.tar.xz files, meaning it's hard for them to sync back
with Debian. I would like this to not be an issue anymore.

> That's the equivalent of expecting that using a different gcc version
> will give you the same output.

I fail to see what gcc and a lossless compressor have in common.

> As long as the bitstream is compatible with previous versions, I don't
> see it as a problem

The problem, I just explained it: I can't use xz in a pristine-tar like
workflow, because it wouldn't reproduce the same output. And I'd like to
use something better than the 20 years old gzip.

 We use "git archive" within the PKG OpenStack team to generate this
 tarball (which is more or less the same as pristine-tar, except we use
 upstream tags rather than a pristine-tar branch). The fact that xz
 produces a different result makes it not reproducible. As a
 consequence, it is very hard for us to use this system across
 distributions (ie: use that in both Debian and Ubuntu, or in Sid &
 Jessie). We need consistency.
> 
> If you generate it once, as part of the release process, why do you
> need to generate it on different systems with different versions?

Because I'd like the Git repository to contain it, without the need to
pick-up the file from the Debian archive. And to be exact: that's mostly
a need from contributors, I could live with the issue, but they can't.
This is mostly a need expressed by Ubuntu/Canonical server team working
with me on OpenStack packaging on Alioth.

> And how does that have anything to do with what gets packaged in Debian.
> For Debian you only need to generate it once, why would you want to
> generate it anew every time you build a new Debian revision instead
> of just reusing the same tarball that is on the archive, if you don't
> keep source tarball releases around?

See above. It's a pristine-tar like workflow. Your question is
equivalent to: "why do people use pristine-tar?". The answer is: because
it's convenient to just use git, without having to look into the Debian
archive. And by the way, xz wouldn't be usable with pristine-tar for the
same reason.

 So it'd be super nice to have LZIP support in dpkg, and use that
 instead of xz, archive wide.

 Your thoughts everyone? Is there any reason why we wouldn't do that?
> 
> Yes, replacing xz with lzip on .deb or .dsc packages does not make any
> sense.

That isn't what I care about. I only care about the orig.tar file here.

> Adding lzip support for source packages *might* make some sense, as
> I pointed out in the bug report. But doing so does have a very high cost:
> 
>   
> 

I do understand the cost. But there's a valid reason. If you believe
there's something better than lz, with the same properties, and that we
had support for it, I'd happily adopt it. It is just that xz doesn't
work right now, and most likely will break again in future versions of
xz-utils.

> Whenever considering to add a new compressor, all surrounding tools need
> to be modified to support it as well:
> 
>   
>   
> 
> That's a non-zero amount of work and time, and that does not take into
> account external tools and users. It would also not be usable until the
> next stable release. Also notice that for example there are still tools
> that do not support data.tar.xz in .deb, which has b

Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Vincent Lefevre]

Normally, a well-designed dependency system should make sure that the
user cannot install an incorrect combination of packages (avoiding
segmentation faults and internal errors), e.g. during a partial
upgrade. But it appears that this is not the case, and users are
required to do "apt-get (dist-)upgrade" and can no longer rely on
"apt-get install " to upgrade just the wanted package
and dependencies:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788710#10

Note that the problem still occurs on an available set of packages:
just start with a Debian/stable system (jessie) and upgrade
libgnutls-deb0-28 to unstable (no dependencies/conflicts will
yield an upgrade of wget, which will occasionally segfault).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150614140332.ga...@xvii.vinc17.org

Re: Adding support for LZIP to dpkg, using that instead of xz, archive wide

[Felipe Sateler]

On Sun, 14 Jun 2015 14:08:24 +0200, Thomas Goirand wrote:

> And by the way, xz wouldn't be usable with pristine-tar for the same
> reason.

Ehm. pristine-xz(1) would beg to disagree.

In the multimedia team, we use it for over 40 packages (where upstream 
provides an xz file of course).

I guess you should have a script that does git archive ; pristine-tar 
commit.

-- 
Saludos,
Felipe Sateler


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/mlk1sh$v1b$1...@ger.gmane.org

Re: Adding support for LZIP to dpkg, using that instead of xz, archive wide

[Vincent Lefevre]

I'm currently using xz for my own files, but...

On 2015-06-14 05:46:00 +0200, Guillem Jover wrote:
> On Sun, 2015-06-14 at 01:08:29 +0200, Thomas Goirand wrote:
> > On 06/13/2015 10:55 AM, Paul Wise wrote:
> > > On Sat, Jun 13, 2015 at 4:23 PM, Thomas Goirand wrote:
> > >> As a friend puts it:
> > >> 
> > >> "This is a fundamental problem/defect with xz. This (and a lot of
> > >> other such defects, e.g. non-robustness of xz archives that easily
> > >> lead to file corruption etc) are the reason that there is lzip (and
> > >> which is why gnu.org has, on a technical basis, decided that lzip is
> > >> official gzip-successor for gnu software releases when they come in
> > >> tarballs).
> 
> TBH this smells like FUD. For example I've never heard of corruption in
> .xz files due to non-robustness, I'd expect that corruption to come from
> external forces, and that integrity would help or not detect it.

xz-utils (4.999.9beta-1) experimental; urgency=low

  [ Jonathan Nieder ]
  * New upstream release.
 - Fix a data corruption in the compression code. (Closes: #544872)
[...]

But of course, this is old, and any compression software can have the
same kind of bug (possibly unless proved formally).

However lzip compresses better, sometimes much better:

-rw-r- 1 vinc17 vinc17   822474 2015-04-26 00:45:51 mail.log.lz
-rw-r- 1 vinc17 vinc17   915544 2015-04-26 00:45:51 mail.log.xz

(this example is a postfix mail log) and uses much less memory for
compression:

$ sh -c 'ulimit -v 20; lzip -9 < mail.log > /dev/null'
$ sh -c 'ulimit -v 80; xz -9 < mail.log > /dev/null'
xz: (stdin): Cannot allocate memory
$ sh -c 'ulimit -v 80; xz -9 < /dev/null > /dev/null'
xz: (stdin): Cannot allocate memory

Note: see the 20 for lzip and 80 for xz.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150614144821.gb...@xvii.vinc17.org

Re: Adding support for LZIP to dpkg, using that instead of xz, archive wide

[Don Armstrong]

On Sun, 14 Jun 2015, Thomas Goirand wrote:
> Therefore, I'm tempted to raise this to the technical committee
> (putting their list as Cc). Does anyone see a reason why I am
> mistaking here?

Does a patch exist which can enable lz for orig.tar? 

Otherwise, I guess some of us could be involved to help clarify
communication, but anyone can do that, really.

-- 
Don Armstrong  http://www.donarmstrong.com



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150614151027.gb5...@qor.donarmstring.com

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Dominik George]

Hi,

> Note that the problem still occurs on an available set of packages:
> just start with a Debian/stable system (jessie) and upgrade
> libgnutls-deb0-28 to unstable (no dependencies/conflicts will
> yield an upgrade of wget, which will occasionally segfault).

well, then, obviously, the dependency on libgnutls-deb0-28 (>= 3.3.0) in
wget is a bit too optimistic. This could have been prevented by the wget
maintainer selecting a more restrictive set ot libgnutls versions,
probably just 3.3.0.

Then again, you wouldn't expect a library to break ABI between
revisions. So this might also be considered a bug in libgnutls or even
in its developers.

In any case, this is nothing any package dependency system could fix
unless told about the situation, because, as noted above, there even is
an expressly written rule stating that 3.3.15, being >= 3.3.0, is
perfectly ok, and that's what apt takes into account, and that's the
best it can do.

-nik



signature.asc
Description: OpenPGP digital signature

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Felipe Sateler]

On Sun, 14 Jun 2015 16:03:32 +0200, Vincent Lefevre wrote:


>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788710#10
> 
> Note that the problem still occurs on an available set of packages: just
> start with a Debian/stable system (jessie) and upgrade libgnutls-deb0-28
> to unstable (no dependencies/conflicts will yield an upgrade of wget,
> which will occasionally segfault).

I think either libgnutls-deb0-28 or libnettle6 should add a Breaks: 
libnettle4[1], to ensure all related packages are upgraded in lockstep.

I have filed bug #788735 for this.


[1] Maybe it could "only" break all rdeps of libnettle4 at the 
appropriate versions, but this is easier.
-- 
Saludos,
Felipe Sateler


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/mlk9i6$ih3$1...@ger.gmane.org

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Marc Haber]

On Sun, 14 Jun 2015 16:03:32 +0200, Vincent Lefevre
 wrote:
>Normally, a well-designed dependency system should make sure that the
>user cannot install an incorrect combination of packages (avoiding
>segmentation faults and internal errors), e.g. during a partial
>upgrade. But it appears that this is not the case, and users are
>required to do "apt-get (dist-)upgrade" and can no longer rely on
>"apt-get install " to upgrade just the wanted package
>and dependencies:
>
>  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788710#10
>
>Note that the problem still occurs on an available set of packages:
>just start with a Debian/stable system (jessie) and upgrade
>libgnutls-deb0-28 to unstable (no dependencies/conflicts will
>yield an upgrade of wget, which will occasionally segfault).

So gnutls-deb0-28 needs a versioned dependency on the correct wget
version. Is that a problem?

btw, please read up on bug severities. I consider filing this bug as
"grave" quite short of being offensive.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/e1z4b0p-s1...@swivel.zugschlus.de

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Simon McVittie]

On 14/06/15 17:19, Felipe Sateler wrote:
> I think either libgnutls-deb0-28 or libnettle6 should add a Breaks: 
> libnettle4[1], to ensure all related packages are upgraded in lockstep.

This is a recurring (anti-)pattern:

* an ABI-stable, high-level library, say libhigh0, links to a
  lower-level library, say liblow0
* we have an ABI transition from liblow0 to liblow1
* liblow0 and liblow1 do not both have versioned symbols
* binaries linked to libhigh0, but whose only reference to liblow* is
  via libhigh0, upgrade gracefully
* binaries that end up with both liblow0 and liblow1 in their address
  space crash

For instance, in this particular case, libhigh is gnutls and liblow is
nettle. The GNOME team sees this every few GNOME release cycles with
clutter (which is long-term ABI-stable) and its dependency cogl (which
isn't). I'm sure there are plenty more examples.

Linking with -Wl,--as-needed and making use of pkg-config's
Requires.private mitigates this problem by reducing the number of
binaries explicitly linked to both libhigh0 and liblow, but doesn't
solve it for anything that does explicitly use liblow functionality.

One solution is to give the lower-level library versioned symbols, with
at least one unique version per SONAME. That's how libjpeg and libpng
avoid breaking lots of GUIs every time they bump SONAME, for instance.

Another solution is to add enough Breaks to force a lockstep upgrade,
which is how the GNOME team deals with it for clutter/cogl AIUI.

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/557db0ba.2060...@debian.org

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Andreas Metzler]

Simon McVittie  wrote:
[...]
> One solution is to give the lower-level library versioned symbols, with
> at least one unique version per SONAME. That's how libjpeg and libpng
> avoid breaking lots of GUIs every time they bump SONAME, for instance.
[...]

FWIW this specific combination should not break again, since the new
nettle has versioned symbols.

It also should not hit us on the jessie->jessie+1 upgrade, because
this will include a GnuTLS soname-bump which automatically enforces
a lockstep nettle/gnutls update.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/eqs05c-f0l@argenau.downhill.at.eu.org

Re: ppp plugins and dependencies

[Chris Boot]

Hi all,

I'm emailing again because I realise I got the per-package QA email
addresses all wrong, but also because I don't think we came to any real
resolution on this.

My original message:

On 07/06/15 11:26, Chris Boot wrote:
> Hi all,
> 
> Apologies for the long email, but there's a lot to discuss on the topic.
> 
> Marco d'Itri and I recently agreed that I would take over maintenance of
> ppp. Thanks for all your hard work over the years keeping this package
> going, Marco.
> 
> One of the first tasks on my list is to resolve the issue with
> dependencies and ABI compatibility surrounding the building of ppp
> plugins. Several packages in the archive use the ppp-dev package to help
> them build ppp plugins that are loaded into pppd at run-time using
> dlopen(). The plugins are generally installed into a particular
> versioned directory on the filesystem (currently
> /usr/lib/pppd/${abi_version}/) where pppd looks for them by default.
> 
> There is currently no mechanism for binary packages to depend on a
> particular ppp plugin ABI version being available, other than manually
> creating (and maintaining) dependencies such as:
> 
> Depends: [...] ppp (>= 2.4.6~), ppp (<< 2.4.7~) [...]
> 
> This is easy to get wrong (in fact, only network-manager-pptp appears to
> get it even nearly right), is a pain to maintain by hand, and is
> impossible for the release team to track sensibly with the transition
> tracker.
> 
> I want to fix this. I'd like to upload the new upstream release 2.4.7
> soon, but when I do this all the packages that build plugins will
> silently break. Last time we went through this pain I had to work with
> the maintainers to get fixed versions uploaded; I know I can't get away
> with it this time either, but perhaps we can make it better for next
> time especially now that ppp has gained momentum upstream again.
> 
> The main problem that I see is that there isn't a built-in mechanism for
> tracking such a situation, as far as I can tell. There aren't any shared
> libraries involved, so I don't have the benefit of sonames, symbols
> files or symbol versioning.
> 
> It seems I the way to resolve this might well be by using a
> "pppapi-${upstream_version}" virtual package (or perhaps a newfangled
> versioned Provides / virtual package). This appears to work for Apache,
> Perl and PHP among others. The disadvantage of this is that currently
> pppd is not a Depends for all the packages that build ppp plugins,
> unlike Apache/Perl/PHP are for their plugins.
> 
> network-manager only has pppd as a Recommends despite shipping a pppd
> plugin. fso-gsmd ships ppp2fsogsmd.so and yet has no dependency on pppd
> whatsoever. Clearly, either those packages need to change or I cannot
> rely on a Depends relationship on pppapi-$foo.
> 
> A ppp plugin is no use on its own, but could easily be installed and
> simply go unused without pppd installed. What definitely doesn't work is
> a ppp plugin built for a different version of pppd - the result will
> either fail to load (due to pppd's built-in version check or being in
> the wrong directory) or crash/misbehave due to an ABI incompatibility.
> 
> What's the best way to handle this?
> 
> I was also considering writing a debhelper plugin and/or dh_ppp_plugin
> script to help with calculating the correct dependencies at build time,
> so that packages can simply invoke the script at build-time and Do The
> Right Thing. It could also be used to obtain the correct plugin
> directory to install plugins into, which seems like it would be useful
> for network-manager-pptp among others. Does this sound like a useful
> addition?
> 
> Lastly, pppd has a built-in mechanism for checking that loaded plugins
> are built for the correct version of pppd. It does this by looking up a
> pppd_version symbol and comparing it against its own version. Currently
> this check is optional, and is used by all the listed packages (below)
> except fso-gsmd. If it's not used, pppd prints a warning and carries on
> regardless. I wonder about making this version check a requirement to
> avoid silent ABI breakage in future. Thoughts?
> 
> I very much look forward to everyone's input on these issues.
> 
> If you'd like to see what's cooking so far for the next upload of ppp,
> please feel free to have a poke around the 'develop' branch in
> collab-maint/pkg-ppp.git [1]. All comments gratefully received.
> 
> 1:
> http://anonscm.debian.org/cgit/collab-maint/pkg-ppp.git/log/?h=develop
> 
> Thanks,
> Chris
> 
> Affected maintainers and source packages:
> 
> Christoph Biedl 
>pptpd
> 
> Debian FreeSmartphone.Org Team 
>fso-gsmd
> 
> Jan-Michael Brummer 
>isdnutils (U)
> 
> Michael Biebl 
>network-manager (U)
>network-manager-pptp (U)
> 
> Rico Rommel 
>fso-gsmd (U)
> 
> Rolf Leggewie 
>isdnutils
> 
> Sebastian Reichel 
>fso-gsmd (U)
> 
> Simon Busch 
>fso-gsmd (U)
> 
> Sjoerd Simons 
>network-manager (U)
> 
> Utopia Maintenance Team 
>network-m

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Vincent Lefevre]

On 2015-06-14 18:43:33 +0200, Marc Haber wrote:
> On Sun, 14 Jun 2015 16:03:32 +0200, Vincent Lefevre
>  wrote:
> >Normally, a well-designed dependency system should make sure that the
> >user cannot install an incorrect combination of packages (avoiding
> >segmentation faults and internal errors), e.g. during a partial
> >upgrade. But it appears that this is not the case, and users are
> >required to do "apt-get (dist-)upgrade" and can no longer rely on
> >"apt-get install " to upgrade just the wanted package
> >and dependencies:
> >
> >  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788710#10
> >
> >Note that the problem still occurs on an available set of packages:
> >just start with a Debian/stable system (jessie) and upgrade
> >libgnutls-deb0-28 to unstable (no dependencies/conflicts will
> >yield an upgrade of wget, which will occasionally segfault).
> 
> So gnutls-deb0-28 needs a versioned dependency on the correct wget
> version. Is that a problem?

The bug was closed without fixing the dependency (gnutls-deb0-28
should actually break libnettle4).

> btw, please read up on bug severities. I consider filing this bug as
> "grave" quite short of being offensive.

Well, I got segfaults and internal errors on various websites. A
package that is broken at this point is almost completely unusable,
so that it deserves a "grave" bug. Note that I hadn't thought that
this was a dependency problem precisely because libgnutls-deb0-28
allowed such a package combination.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150614201809.ga20...@xvii.vinc17.org

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Russ Allbery]

Simon McVittie  writes:

> This is a recurring (anti-)pattern:

> * an ABI-stable, high-level library, say libhigh0, links to a
>   lower-level library, say liblow0
> * we have an ABI transition from liblow0 to liblow1
> * liblow0 and liblow1 do not both have versioned symbols

And this point is the root of the problem.

When I'm in a particular tilting at windmills mood, I think we should just
stop accepting new shared libraries in Debian that don't use symbol
versioning, and make adding symbol versioning mandatory the next time the
SONAME changes.  I know this is a ton of work for a lot of edge packages
where the upstream maintainers are building shared libraries without
really understanding how they work, but it's so hard to properly manage
library upgrades without symbol versioning.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/87y4jm2g0d@hope.eyrie.org

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Vincent Lefevre]

On 2015-06-14 18:15:33 +0200, Dominik George wrote:
> Hi,
> 
> > Note that the problem still occurs on an available set of packages:
> > just start with a Debian/stable system (jessie) and upgrade
> > libgnutls-deb0-28 to unstable (no dependencies/conflicts will
> > yield an upgrade of wget, which will occasionally segfault).
> 
> well, then, obviously, the dependency on libgnutls-deb0-28 (>= 3.3.0) in
> wget is a bit too optimistic. This could have been prevented by the wget
> maintainer selecting a more restrictive set ot libgnutls versions,
> probably just 3.3.0.

Well, there are two things that one wants to avoid:

1. Upgrading libgnutls-deb0-28 to a version using libnettle6 without
   upgrading the packages that depend on libgnutls-deb0-28 and use
   libnettle4. This is the problem I've mentioned here. And there's
   now the following bug reported by Felipe Sateler:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788735

   (Bug 788710 shouldn't have been closed, but changed to something
   like what bug 788735 says.)

2. Upgrading wget to to a version using libnettle6 without upgrading
   libgnutls-deb0-28 to such a version too. For this point, I agree
   that the dependency on libgnutls-deb0-28 (>= 3.3.0) in wget is too
   optimistic. That's the following bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787942

BTW, if wget had the correct dependency, I would probably never have
seen problem (1).

> In any case, this is nothing any package dependency system could fix
> unless told about the situation, because, as noted above, there even is
> an expressly written rule stating that 3.3.15, being >= 3.3.0, is
> perfectly ok, and that's what apt takes into account, and that's the
> best it can do.

OK, so, that's more a problem with developers who close bugs without
fixing the dependencies.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150614203828.gb20...@xvii.vinc17.org

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Michael Banck]

Hi Siomn,

On Sun, Jun 14, 2015 at 05:50:02PM +0100, Simon McVittie wrote:
> On 14/06/15 17:19, Felipe Sateler wrote:
> > I think either libgnutls-deb0-28 or libnettle6 should add a Breaks: 
> > libnettle4[1], to ensure all related packages are upgraded in lockstep.
> 
> This is a recurring (anti-)pattern:
> 
> * an ABI-stable, high-level library, say libhigh0, links to a
>   lower-level library, say liblow0
> * we have an ABI transition from liblow0 to liblow1
> * liblow0 and liblow1 do not both have versioned symbols
> * binaries linked to libhigh0, but whose only reference to liblow* is
>   via libhigh0, upgrade gracefully
> * binaries that end up with both liblow0 and liblow1 in their address
>   space crash
> 
> For instance, in this particular case, libhigh is gnutls and liblow is
> nettle. The GNOME team sees this every few GNOME release cycles with
> clutter (which is long-term ABI-stable) and its dependency cogl (which
> isn't). I'm sure there are plenty more examples.
> 
> Linking with -Wl,--as-needed and making use of pkg-config's
> Requires.private mitigates this problem by reducing the number of
> binaries explicitly linked to both libhigh0 and liblow, but doesn't
> solve it for anything that does explicitly use liblow functionality.
> 
> One solution is to give the lower-level library versioned symbols, with
> at least one unique version per SONAME. That's how libjpeg and libpng
> avoid breaking lots of GUIs every time they bump SONAME, for instance.
> 
> Another solution is to add enough Breaks to force a lockstep upgrade,
> which is how the GNOME team deals with it for clutter/cogl AIUI.

Hrm, you're not coming to DebConf15, right?

I think a talk like "Best Practises for Debian Library Packaging" would
be well-received, I am not sure we had something like this decade...

(the CfP would be open for another 24 hours)


Michael


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20150614205538.ga12...@raptor.chemicalconnection.dyndns.org

Re: Adding support for LZIP to dpkg, using that instead of xz, archive wide

[Guillem Jover]

Hi!

On Sun, 2015-06-14 at 16:48:21 +0200, Vincent Lefevre wrote:
> On 2015-06-14 05:46:00 +0200, Guillem Jover wrote:
> > On Sun, 2015-06-14 at 01:08:29 +0200, Thomas Goirand wrote:
> > > On 06/13/2015 10:55 AM, Paul Wise wrote:
> > > > On Sat, Jun 13, 2015 at 4:23 PM, Thomas Goirand wrote:
> > > >> As a friend puts it:
> > > >> 
> > > >> "This is a fundamental problem/defect with xz. This (and a lot of
> > > >> other such defects, e.g. non-robustness of xz archives that easily
> > > >> lead to file corruption etc) are the reason that there is lzip (and
> > > >> which is why gnu.org has, on a technical basis, decided that lzip is
> > > >> official gzip-successor for gnu software releases when they come in
> > > >> tarballs).
> > 
> > TBH this smells like FUD. For example I've never heard of corruption in
> > .xz files due to non-robustness, I'd expect that corruption to come from
> > external forces, and that integrity would help or not detect it.
> 
> xz-utils (4.999.9beta-1) experimental; urgency=low
> 
>   [ Jonathan Nieder ]
>   * New upstream release.
>  - Fix a data corruption in the compression code. (Closes: #544872)
> [...]
> 
> But of course, this is old,

Yes, that was even before dpkg started to use xz-utils to handle .xz
files.

> and any compression software can have the
> same kind of bug (possibly unless proved formally).

And in any case I don't see how this is a "fundamental problem" with
the format, this is simply just a bug in a beta version, although an
unfortunate one.

> However lzip compresses better, sometimes much better:
> 
> -rw-r- 1 vinc17 vinc17   822474 2015-04-26 00:45:51 mail.log.lz
> -rw-r- 1 vinc17 vinc17   915544 2015-04-26 00:45:51 mail.log.xz

Oh, interesting, this didn't use to be the case when we added .xz
support to dpkg.

> (this example is a postfix mail log) and uses much less memory for
> compression:
> 
> $ sh -c 'ulimit -v 20; lzip -9 < mail.log > /dev/null'
> $ sh -c 'ulimit -v 80; xz -9 < mail.log > /dev/null'
> xz: (stdin): Cannot allocate memory
> $ sh -c 'ulimit -v 80; xz -9 < /dev/null > /dev/null'
> xz: (stdin): Cannot allocate memory
> 
> Note: see the 20 for lzip and 80 for xz.

The preset levels do not match between lzip and xz. For example for -9, xz
uses a dictionary size of 64 MiB, while lzip uses 32 MiB. Other parameters
are also probably quite different. In addition lzip seems to be
substantially slower (at least) when compressing compared to xz using the
same preset levels. With a small pdf file it took more than twice the time:

,---
$ cat Posix_1003.1e-990310.pdf >/dev/null
$ ls -la Posix_1003.1e-990310.pdf
-rw-r- 1 guillem guillem 3486116 Feb 20 16:43 Posix_1003.1e-990310.pdf
$ /usr/bin/time xz -9k Posix_1003.1e-990310.pdf
1.24user 0.07system 0:01.31elapsed 99%CPU (0avgtext+0avgdata 98748maxresident)k
0inputs+3520outputs (0major+24291minor)pagefaults 0swaps
$ rm -f Posix_1003.1e-990310.pdf.xz
$ /usr/bin/time xz -9k Posix_1003.1e-990310.pdf
1.25user 0.06system 0:01.31elapsed 99%CPU (0avgtext+0avgdata 98952maxresident)k
0inputs+3520outputs (0major+24295minor)pagefaults 0swaps
$ ls -la Posix_1003.1e-990310.pdf.xz
-rw-r- 1 guillem guillem 1801372 Feb 20 16:43 Posix_1003.1e-990310.pdf.xz
$ rm -f Posix_1003.1e-990310.pdf.xz
#
$ /usr/bin/time lzip -9k Posix_1003.1e-990310.pdf
2.93user 0.02system 0:02.96elapsed 99%CPU (0avgtext+0avgdata 37628maxresident)k
0inputs+3520outputs (0major+8957minor)pagefaults 0swaps
$ rm -f Posix_1003.1e-990310.pdf.lz
$ /usr/bin/time lzip -9k Posix_1003.1e-990310.pdf
2.94user 0.03system 0:02.98elapsed 99%CPU (0avgtext+0avgdata 37576maxresident)k
0inputs+3520outputs (0major+8955minor)pagefaults 0swaps
-rw-r- 1 guillem guillem 1798338 Feb 20 16:43 Posix_1003.1e-990310.pdf.lz
$ rm -f Posix_1003.1e-990310.pdf.lz
`---

With the linux sources:

,---
$ cat linux_4.0.4.orig.tar >/dev/null
$ ls -la linux_4.0.4.orig.tar
-rw-r--r-- 1 guillem guillem 582932480 May 26 20:15 linux_4.0.4.orig.tar
$ /usr/bin/time lzip -k9 linux_4.0.4.orig.tar
619.52user 1.27system 10:21.95elapsed 99%CPU (0avgtext+0avgdata 
363168maxresident)k
24inputs+156680outputs (0major+90387minor)pagefaults 0swaps
$ ls -la linux_4.0.4.orig.tar.lz
-rw-r--r-- 1 guillem guillem 80218126 May 26 20:15 linux_4.0.4.orig.tar.lz
$ rm -f linux_4.0.4.orig.tar.lz
$ /usr/bin/time lzip -k9 linux_4.0.4.orig.tar
618.94user 1.10system 10:21.02elapsed 99%CPU (0avgtext+0avgdata 
363180maxresident)k
8inputs+156680outputs (0major+90389minor)pagefaults 0swaps
$ rm -f linux_4.0.4.orig.tar.lz
#
$ /usr/bin/time xz -k9 linux_4.0.4.orig.tar
514.76user 1.53system 8:37.22elapsed 99%CPU (0avgtext+0avgdata 
691428maxresident)k
176inputs+156656outputs (1major+172417minor)pagefaults 0swaps
$ ls -la linux_4.0.4.orig.tar.xz 
-rw-r--r-- 1 guillem guillem 80205900 May 26 20:15 linux_4.0.4.orig.tar.xz
$ rm -f linux_4.0.4.orig.tar.xz
$ /usr/bin/time xz -k9 linux_4.0.4.orig.tar
515.96user 1.62system 8:38.60elapsed 99%CPU (0avgtext+0avgdata 
691328maxresident)k
5

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Neil Williams]

On Sun, 14 Jun 2015 13:26:26 -0700
Russ Allbery  wrote:

> Simon McVittie  writes:
> 
> > This is a recurring (anti-)pattern:
> 
> > * an ABI-stable, high-level library, say libhigh0, links to a
> >   lower-level library, say liblow0
> > * we have an ABI transition from liblow0 to liblow1
> > * liblow0 and liblow1 do not both have versioned symbols
> 
> And this point is the root of the problem.
> 
> When I'm in a particular tilting at windmills mood, I think we should
> just stop accepting new shared libraries in Debian that don't use
> symbol versioning, and make adding symbol versioning mandatory the
> next time the SONAME changes. 

With the proviso that "ignoring/fudging" a SONAME change to avoid this
step is an RC bug. A SONAME change going through NEW which doesn't
include symbol versioning should be a reject.

The work to generate the symbols has already been done but needs a new
maintainer (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543640).
That would need to be fixed first.

> I know this is a ton of work for a lot
> of edge packages where the upstream maintainers are building shared
> libraries without really understanding how they work, but it's so
> hard to properly manage library upgrades without symbol versioning.

Yet these are precisely the packages (and upstreams) which are most in
need of such a requirement.



-- 


Neil Williams
=
http://www.linux.codehelp.co.uk/



pgppkkShkwh1J.pgp
Description: OpenPGP digital signature

Re: Is the Debian dependency system broken? (wget vs libgnutls-deb0-28)

[Niels Thykier]

On 2015-06-15 08:04, Neil Williams wrote:
> On Sun, 14 Jun 2015 13:26:26 -0700
> Russ Allbery  wrote:
> 
>> Simon McVittie  writes:
>>
>>> This is a recurring (anti-)pattern:
>>
>>> * an ABI-stable, high-level library, say libhigh0, links to a
>>>   lower-level library, say liblow0
>>> * we have an ABI transition from liblow0 to liblow1
>>> * liblow0 and liblow1 do not both have versioned symbols
>>
>> And this point is the root of the problem.
>>
>> When I'm in a particular tilting at windmills mood, I think we should
>> just stop accepting new shared libraries in Debian that don't use
>> symbol versioning, and make adding symbol versioning mandatory the
>> next time the SONAME changes. 
> 
> With the proviso that "ignoring/fudging" a SONAME change to avoid this
> step is an RC bug. A SONAME change going through NEW which doesn't
> include symbol versioning should be a reject.
> 
> The work to generate the symbols has already been done but needs a new
> maintainer (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=543640).
> That would need to be fixed first.
> 
>> [...]

To avoid confusing myself further, Russ and Neil, are you both talking
about the "debian/symbols" files?  I thought Russ might have been
talking about "versioned symbols at DSO level" (e.g. symbol@LOW0 vs
symbol@LOW1).

~Niels



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/557e6ea2.4090...@thykier.net

Re: ppp plugins and dependencies

[Christoph Biedl]

Chris Boot wrote...

> The main problem that I see is that there isn't a built-in mechanism for
> tracking such a situation, as far as I can tell. There aren't any shared
> libraries involved, so I don't have the benefit of sonames, symbols
> files or symbol versioning.

(...)

disclaimer: I might not overlook the implications so just take this
as a small bit from a maintainer of one of the packages involved (pptpd).

In general I prefer solutions that require a lot of work but for
just one time over those that need some small attention every now and
then; and also those where work needs to be done just in one place
instead of many. Not just because I'm lazy but mostly since this
should be more error-prone.

Therefore, the debhelper/dh_ppp_plugin looks like a good idea unless
there is an even more generic approach as suggested by James McCoy.
AFAICT this could also create the lower and upper bound versioned
dependencies on ppp.

However, I'll follow any changes your decision will imply. Choose wisely.

my 2¢

Christoph


signature.asc
Description: Digital signature

Bug#788794: ITP: python-suds-jurko -- lightweight SOAP client (Jurko's fork)

[Thomas Goirand]

Package: wnpp
Severity: wishlist
Owner: Thomas Goirand 

* Package name: python-suds-jurko
  Version : 0.6
  Upstream Author : Jurko Gospodnetic
* URL : https://bitbucket.org/jurko/suds
* License : LGPL
  Programming Lang: Python
  Description : lightweight SOAP client (Jurko's fork)

 Based on the original 'suds' project by Jeff Ortel 
 dot com) hosted at http://fedorahosted.org/suds.
 .
 Suds is a lightweight SOAP-based web service client for Python.

Note from future maintainer: the OpenStack project decided to switch from Suds
to suds-jurko, as Suds is unmaintained. So suds-jurko becomes the new thing to
use now for OpenStack.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/20150615065527.26013.92490.report...@buzig2.mirantis.com