Re: .deb format: let's use 0.939, zstd, drop bzip2

[Ansgar]

Adam Borowski writes:
> On Wed, May 08, 2019 at 10:35:58PM +0200, Ansgar wrote:
>> Adam Borowski writes:
>> > I've recently did some research on how can we improve the speed of 
>> > unpacking
>> > packages.  There's a lot of other stages that can be improved, but let's
>> > talk about the .deb format.
>> >
>> > First, the 0.939 format, as described in "man deb-old".  While still being
>> > accepted by dpkg, it had been superseded before even the very first stable
>> > release.  Why?  It has at least two upsides over 2.0:
>> 
>> Switching to a different binary format will break various tools.
>
> The 0.939 format is already supported by most tools.
>
> No one sane digs into insides of the format, using a small number of
> low-level tools, thus we can reuse it with little effort.
>
> Of course, adding a new compressor _does_ break compat, but we added four
> compressors to 2.0 over the years already, and the sky didn't fall.

Well, it causes minor breakage which is fairly easy to fix.  A different
container format instead of tar would require more involved changes in
tools, so I'm not 100% convinced of my idea myself ;-)  The thread just
looked like the right time to consider such changes.

>> If we want to do this, I wonder if we shouldn't take the chance to move
>> away from tar?
>
> Any seekable format significantly reduces compression, although this can
> be reduced by managing split points.

Well, depending on how much splitting you do, the loss in compression
should be small enough to not care about?

>> We have various applications that only want to extract single members of
>> the package (changelog, NEWS, copyright, ...); tar is a really bad
>> format for such an operation.  Other formats (zip, 7z, ...) are more
>> suited for them.
>
> Perhaps such files could be considered metadata and moved to the control
> tarball?  Or merely just moved forward -- remember that tarballs are
> unordered.

I don't think that is a good idea: if someone wants to use another file
in a similar way, he couldn't and would have to fall back to the worse
solution.

As an example: I have a config-diff script which compares conffiles with
the pristine version included in the *.deb; it wants to access /etc/*.
(Though ideally dpkg would keep the pristine version accessible below
/usr; that would also be useful for other uses.)

Also dpkg keeps metadata in /var, but changelogs, NEWS, copyright
documentation isn't variable state data and should be below /usr...  The
same is really true for lists of files and maintainer scripts though.

Ansgar

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Mike Hommey]

On Wed, May 08, 2019 at 09:01:27PM -0400, Michael Stone wrote:
> On Thu, May 09, 2019 at 07:37:55AM +0800, Paul Wise wrote:
> > On Thu, May 9, 2019 at 1:38 AM Adam Borowski wrote:
> > 
> > > Thus, even though we'd want to stick with xz for the official archive, 
> > > speed
> > > gains from zstd are so massive that it's tempting to add support for it,
> > > at least for non-official uses, possibly also for common Build-Depends.
> > 
> > Could we use custom zstd dictionaries on a per-architecture basis to
> > further reduce the size of zstd packages, possibly allowing it to beat
> > xz?
> 
> In theory, sure. Have any test results? My gut tells me that wouldn't buy
> much but numbers matter more.

Another option is to use filters like xz does.
e.g.
https://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/x86.c;h=0b14807e900cdf4a85dc513c281892c2309bb454;hb=4ed339606156bd313ed99237485cb8ed0362d64f
https://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/arm.c;h=181d0e3b223220fa24cb5feb638b231357326905;hb=4ed339606156bd313ed99237485cb8ed0362d64f
https://git.tukaani.org/?p=xz.git;a=blob;f=src/liblzma/simple/armthumb.c;h=eab4862dd76dd03d4c0a0d8e4af7385866d9197d;hb=4ed339606156bd313ed99237485cb8ed0362d64f
etc.

Mike

Bug#928699: ITP: HelioPy -- Python for heliospheric and planetary physics

[Shreyas Bapat]

Package: heliopy
Owner: Shreyas Bapat 
Severity: wishlist
X-Debbugs-Cc: debian-devel@lists.debian.org,debian-as...@lists.debian.org,
debian-pyt...@lists.debian.org

* Package name: heliopy
  Version: 0.6.7
  Upstream Author : David Stansby 
* URL: https://github.com/heliopython/heliopy
* Licence   : GPL-3.0
  Programming Lang : Python
  Description  : Python for heliospheric and planetary
physics

A python library for heliospheric and planetary Physics. The primary goal
of HelioPy is to provide a set of tools to download and read in data, and
to carry out other common data processing tasks.


It will maintained within the Debian Astronomy Working Group. A git
repository will be created on salsa:

https://salsa.debian.org/debian-astro-team/heliopy.git

Best regards

Shreyas

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Anthony DeRobertis]

On May 8, 2019 9:43:50 PM UTC, Adam Borowski  wrote:

>I just checked Stretch: not a single .bz2, either control nor data. 
>I'm not
>going to download all of Jessie just to check -- but even assuming
>something
>was left by Jessie's time, by Bullseye trying to install such a .deb
>will
>mean mixing packages 3 releases apart.

dpkg-deb is used to examine debs too, and considering Jessie is still LTS and 
Wheezy is ELTS, you may well want to examine packages from several releases ago 
on a current system. I have a weird case at work where I need to examine 
packages from as far back as Sarge and Etch through Buster, but I'd fully 
understand not supporting that. 

Some local packages can be long-lived, too. E.g., at work I have one that 
installs an internal CA. That package hasn't needed changing in a while, it 
drops a file and calls update-ca-certificates. Wouldn't be a huge deal to 
rebuild it, of course.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Campbell]

On Thu, 2019-05-09 at 11:19 +1000, Ben Finney wrote:
> What I remain unconvinced of is the worth of abandoning the clean
> separation between an upstream source repository (whether Git repository
> or not) and a VCS repository for Debian packaging (typically in Git).

I think there's a common misconception here which has cropped up
several times in this thread. (NB: I've not used dgit in anger yet, but
I hope what follows is correct).

That misconception is that "dgit repo" == "maintainer's repo", which is
not actually the case. As a maintainer you can (if you want) just use
"dgit push" (instead of dput) while only caring about (and interacting
with) the "maintainer's repo", never touching or looking at dgit's view
of the world (apart from perhaps noticing and ignoring some `dgit/*`
branches in your _local_ repo, I don't beleive you are required to push
those to anywhere, including your maintainer repo).

The "dgit repo" provides a view onto precisely what has been uploaded
to the archive, no more no less. Where it can (i.e. where maintainers
are using "dgit push" etc) it tries to stitch in the "maintainer's
repo" (which I think is part #1 of where this misconception arises) as
a subset but it doesn't follow that using dgit implies any constraint
on the maintainer's repo (or at least that is the aim, AIUI there are
some bugs and wrinkles which make this less than 100% true in reality,
e.g. #908747)

There is a caveat though (which I think is part #2 of the
misconception) which is that dgit sometimes does need to understand
tooling which maintainer's are using in the "maintainer's repo" (i.e.
it needs to understand gbp pq and debrebase etc to work with repos
maintained that way). But "dgit needs to understand the maintainer's
repo" does not imply the inverse "the maintainer's repo must be shaped
somehow by the use of dgit". In fact AIUI not having the use of dgit
put constraints on how a maintainer structures their repo is an
explicit design goal of dgit.

It follows that there are some "maintainer's repo" layouts/strategies
which dgit is not currently aware of and is therefore unable to cope
with (and I beleive "packaging only repo" workflow is one of those).
However AIUI from what Ian has said elsewhere that this is purely down
to prioritisation and time, and given either big demand or adequate
time this workflow will be supported by dgit. There is no philosophical
objection (in dgit at least) to such workflows nor a desire to force
(nor nudge) maintainers away from using whatever workflow they wish to.

Ian.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Colin Watson]

On Wed, May 08, 2019 at 03:26:32PM +, Holger Levsen wrote:
> thanks for the pointer, but I don't see the string debcheckout in that
> message and vcs-git only once, where you write:
> 
> ---begin quote---
> 
> >From these we can conclude:
> 
>  * Debian should provide source code as git branches which:
>   - can be built using a standard set of runes
>   - will produce the same binaries as official Debian ones 
>   - can be reliably located
>   - can be easily modified (using standard git commands)
>   - contain the git histories we are actually using ourselves
> 
> There is only one way to do this.  It is `dgit push[-source]'.
> 
> Vcs-Git and Salsa do not provide this. 
> 
> ---end quote---
> 
> and I'm not sure I agree this is true, to me Vcs-Git and Salsa do provide all
> of this, *if* Vcs-Git is set.
> 
> And if it's not set, it's either because the package is not in git or because
> d/control is lacking information, aka the package is buggy.

I think the thing you're missing is Ian's "can be reliably located"
point.  As you say, debcheckout works if those conditions are met, but
they're very often not met.  Given how long debcheckout and Vcs-Git have
been around, it seems a reasonable conclusion that they're unlikely to
reach 100% any time soon (and there's the long-standing problem with
stale metadata in stable releases and so on that means it isn't even a
ratchet, and won't be unless some as-yet-unfinished work to make it be
an archive override or similar happens instead).  Without wishing to
speak for Ian, I think this is what he's abbreviating as "do not provide
this".

By contrast, "dgit clone" works regardless of whether these conditions
are met, although it results in a better git history if the maintainer
used "dgit push" or "dgit push-source".  That does seem like an
important distinction.

> So, IOW, I can see problems with individual packages here but not with the
> general workflow/tool of using vcs-git: and debcheckout.
> 
> (or what am I missing?)

Maybe another way to look at it is that sometimes it makes more sense to
look at a problem systemically, including new tools, than by annotating
thousands of source packages one by one.

[FWIW: I recently started using "dgit push-source" for my uploads,
partly as a result of this thread although I'd been meaning to look into
it for ages.  I've filed a couple of bugs and been confused about one or
two things that Ian has kindly helped me with, but by and large it's
been much more straightforward than I'd feared.]

-- 
Colin Watson   [cjwat...@debian.org]

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ansgar]

On Thu, 2019-05-09 at 10:33 +0100, Ian Campbell wrote:
> On Thu, 2019-05-09 at 11:19 +1000, Ben Finney wrote:
> > What I remain unconvinced of is the worth of abandoning the clean
> > separation between an upstream source repository (whether Git repository
> > or not) and a VCS repository for Debian packaging (typically in Git).
> 
> I think there's a common misconception here which has cropped up
> several times in this thread. (NB: I've not used dgit in anger yet, but
> I hope what follows is correct).
> 
> That misconception is that "dgit repo" == "maintainer's repo", which is
> not actually the case. As a maintainer you can (if you want) just use
> "dgit push" (instead of dput) while only caring about (and interacting
> with) the "maintainer's repo", never touching or looking at dgit's view
> of the world (apart from perhaps noticing and ignoring some `dgit/*`
> branches in your _local_ repo, I don't beleive you are required to push
> those to anywhere, including your maintainer repo).

I think that is very confusing, yes.

By now I have come to understand that whatever "dgit" produces should
be just regarded as a build artifact, just like binary packages or for
some people the .dsc or for some workflows debian/patches/*.

Though directing users to build artifacts when they request the source
seems counterproductive to me...

I wonder why not the "maintainer view" is published (for the part
making Git repositories "reliably locatable") and any mangling is
applied by "dgit clone" (instead of "dgit push")?  ("dgit push" can
check that the mangling works.)

Then whatever is published is what is the actual preferred form of
modification (as it is what the maintainer uses), but if so desired one
can still get a "dgit view".  (Though for contributing changes to the
maintainer, one should probably base them on the maintainer view...)

In this case the published history also matches the "git histories we
are actually using ourselves", a design goal not met currently; one
could also apply the mangling feature to repositories not published on
the dgit server.

Ansgar

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Adam Borowski]

On Thu, May 09, 2019 at 08:10:00AM +, Anthony DeRobertis wrote:
> On May 8, 2019 9:43:50 PM UTC, Adam Borowski  wrote:
> 
> >I just checked Stretch: not a single .bz2, either control nor data.  I'm
> >not going to download all of Jessie just to check -- but even assuming
> >something was left by Jessie's time, by Bullseye trying to install such a
> >.deb will mean mixing packages 3 releases apart.
> 
> dpkg-deb is used to examine debs too, and considering Jessie is still LTS
> and Wheezy is ELTS, you may well want to examine packages from several
> releases ago on a current system.  I have a weird case at work where I
> need to examine packages from as far back as Sarge and Etch through
> Buster, but I'd fully understand not supporting that.

Yeah -- and on any non-minimal system, unpacking such debs would work
without any action (be it via dlopen or exec|pipe).  libbz2 has enough
dependencies that it won't get out of default installs anytime soon.  On
minimal systems you'd get an error message telling you to install the
optional library.

> Some local packages can be long-lived, too.  E.g., at work I have one that
> installs an internal CA.  That package hasn't needed changing in a while,
> it drops a file and calls update-ca-certificates.  Wouldn't be a huge deal
> to rebuild it, of course.

You can "ar t foobar.deb" to see what kind of compression it uses.  I don't
think bzip2 was ever the default, though -- thus it's likely to happen only
in largest or overoptimized packages.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Did ya know that typing "test -j8" instead of "ctest -j8"
⢿⡄⠘⠷⠚⠋⠀ will make your testsuite pass much faster, and fix bugs?
⠈⠳⣄

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Adam Borowski]

On Thu, May 09, 2019 at 09:22:56AM +0200, Ansgar wrote:
> Also dpkg keeps metadata in /var, but changelogs, NEWS, copyright
> documentation isn't variable state data and should be below /usr...  The
> same is really true for lists of files and maintainer scripts though.

It's a mess:

* Most of the control tarball (to be exact: every file other than "control")
  goes to /var/lib/dpkg/info/$PACKAGE.$FILENAME; they're all (I've verified
  across all .debs in Buster) plain files of either mode 644 or 755.
* Except for "control" which is sort of concatenated and dumped into
  /var/lib/dpkg/status, with "Status:" added.
* On the other hand, /var/lib/dpkg/info/$PACKAGE.list is generated from
  the list of files in the data tarball.

Knowing $PACKAGE requires reading the control tarball: if Multi-Arch is
"same" (and no other value), $PACKAGE is "Package:Architecture", "Package"
otherwise.

The data tarball is unpacked into the filesystem mostly as-is, but you still
need to obey diverts, replaces, and symlinks.


Meow!

(I'm reverse-engineering dpkg instead of reading the specs on purpose: in
order to change the spec, what matters is current practice rather than the
letter of documentation, often written 25 years ago before we settled on a
subset of features.)
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Did ya know that typing "test -j8" instead of "ctest -j8"
⢿⡄⠘⠷⠚⠋⠀ will make your testsuite pass much faster, and fix bugs?
⠈⠳⣄

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

Ansgar writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> On Thu, 2019-05-09 at 10:33 +0100, Ian Campbell wrote:
> > I think there's a common misconception here which has cropped up
> > several times in this thread. (NB: I've not used dgit in anger yet, but
> > I hope what follows is correct).

Everything Ian said is completely correct.  Thanks, Ian :-).

> > That misconception is that "dgit repo" == "maintainer's repo", which is
> > not actually the case. As a maintainer you can (if you want) just use
> > "dgit push" (instead of dput) while only caring about (and interacting
> > with) the "maintainer's repo", never touching or looking at dgit's view
> > of the world (apart from perhaps noticing and ignoring some `dgit/*`
> > branches in your _local_ repo, I don't beleive you are required to push
> > those to anywhere, including your maintainer repo).
> 
> I think that is very confusing, yes.
> 
> By now I have come to understand that whatever "dgit" produces should
> be just regarded as a build artifact, just like binary packages or for
> some people the .dsc or for some workflows debian/patches/*.

As a maintainer you can think of it like, and treat it as, a build
product.  The user or downstream can think of it like, and use it as,
the source code.  It all depends on your point of view.

Fundamentally the circle being squared here is this:

 * Maintainers think "the source code" means some Debian-specific
   thing.  Often a strange or dangerous thing - even the most usual, a
   patches- unapplied git branch, is dangerous if you don't know how
   to handle it.  Furthermore the maintainers don't even agree.

 * Users think "the source code" means a normal git branch you can
   build and do `git grep' and `git commit' and `git merge' with.

Luckily it turns out that you can construct the 2nd from the 1st, in
most cases.  (Indeed `gbp pq import' sort of does that and actually
for maintainers using gbp pq, the patch queue branch is the preferred
form for most *modifications* and the master branch is an interchange
format used for distribution; luckily the two are interconvertibel.)

And you can do it in such a way that the 2nd *contains* the 1st, so
that anyone who has the converted version also has the unconverted
version, if they care.  In practice, it is IME very rare to want this.

> I wonder why not the "maintainer view" is published (for the part
> making Git repositories "reliably locatable") and any mangling is
> applied by "dgit clone" (instead of "dgit push")?  ("dgit push" can
> check that the mangling works.)

dgit push makes, and publishes, a maintainer view tag, as well as a
dgit view tag.  So you can indeed get the maintainer view.

So far no-one who is a user of `dgit clone' and `dgit fetch' has
requested a feature for presenting the maintainer view as a matter of
course.  Such a feature would have some difficulties.  What is the
`maintainer view' if the maintainer uses gbp pq, but the most recent
upload was an NMU not done with dgit ?

If you say "the .dsc import" then this "maintainer view" format
oscillates across versions.  If you say "whatever you get when
converting the .dsc back to the maintainer's format" then a
backwards-converter would be required.

Or to put it another way, while everyone is required, when uploading,
to provide something that looks like a .dsc, not everyone who uploads
is required to provide something in some other maintainer-determined
format.

> Then whatever is published is what is the actual preferred form of
> modification (as it is what the maintainer uses),

The key question about "preferred form for modification" is "preferred
by who", of course.  As I say above, maintainers and (at least a big
subset of) users/downstreams have different opinions.

Happily since the maintainer's view is always an ancestor of the
corresponding dgit view, publishing the dgit view implies publishing
the maintainer's view; and finding it in the history is
straightforward because there is an appropriate tag.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git

[Ian Jackson]

(adding #903392)

Ben Finney writes ("Re: Preferred git branch structure when upstream moves from 
tarballs to git"):
> What I remain unconvinced of is the worth of abandoning the clean
> separation between an upstream source repository (whether Git repository
> or not) and a VCS repository for Debian packaging (typically in Git).

I disagree with this but I'm not really interested in arguing you out
of it.  As Ian Campbell says, dgit doesn't want to change your
workflow.  It wants to help you help *users* and *downstreams* to
change theirs, from tarballs+diffs to git.

My interest is to make that possible, *without* maintainers having to
change their workflows.

For your workflow, I see broadly two possibilities, depending what
representation you use for upstream source code:

 (i) You use upstream tarballs as your representation of the upstream
 source code, and do not use upstream git branches.

 If this is the case then the benefits for everyone in me
 implementing #903392 for you, and you then using `dgit
 push-source', are - frankly - limited.  The resulting dgit git
 branch would still contain imports of .orig tarballs, rather than
 the actual upstream git history.

 The user of `dgit clone' would get your *packaging* history,
 sure, but this is a much more minor benefit.

 (ii) You use upstream git history, and generate .orig tarballs
 with (say) git-deborig.

 Implementing #903392 for this case should be straightforward,
 provided that your .orig tarballs and the git branch actually are
 identical.  Furthermore, it would be very useful to someone who
 `dgit clone's one of your packages: they would get a full and
 proper git history which would work exactly as they expect.

 The question here would be: how do you manage the upstream git
 branch ?  dgit push would need to automatically find (or be told,
 but that would be a UI nuisance) the upstream git commit.

 (And if that upstream git commit is in a separate tree somewhere,
 dgit would have to find that tree, too.)

If you are roughly in situation (ii) then I am quite interested in
supporting your use of dgit push, eg by working on #903392.  Please
engage in that bug, and encourage others in a similar position to do
the same.

If you are currently in situation (i) then your adoption of dgit is a
low priority for me, since to provide really useful output from `dgit
clone' it would be necessary for you to change your workflow.  (I do
have a wip branch but it is a mess and not tested.  So if someone
wants to work on this, let me know.)

I am not really interested in arguing that people's workflows are
wrong; it is a very religious kind of topic.  Let us save that for a
bar conversation some time ...

HTH.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Jeremy Stanley]

On 2019-05-09 06:27:36 +0900 (+0900), Mike Hommey wrote:
> On Wed, May 08, 2019 at 09:04:49PM +, Jeremy Stanley wrote:
[...]
> > Are you talking about source packages or binary packages here? The
> > latter use ar, not tar.
> 
> Binary packages use both.
> 
> $ ar t /var/cache/apt/archives/libgcc-9-dev_9.1.0-1_amd64.deb
> debian-binary
> control.tar.xz
> data.tar.xz

Ahh, yes, thanks I should have remembered that... so then my
question becomes: is the suggestion to replace *both* ar and tar in
the binary package format with a single flat archive, or to switch
out the tarballs inside the ar archive but continue using ar to
aggregate them?

(Also, no need to Cc me... as with basically everyone else on this
ML, I am subscribed.)
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Ansgar]

On Thu, 2019-05-09 at 12:32 +, Jeremy Stanley wrote:
> On 2019-05-09 06:27:36 +0900 (+0900), Mike Hommey wrote:
> > On Wed, May 08, 2019 at 09:04:49PM +, Jeremy Stanley wrote:
> [...]
> > > Are you talking about source packages or binary packages here?
> > > The
> > > latter use ar, not tar.
> > 
> > Binary packages use both.
> > 
> > $ ar t /var/cache/apt/archives/libgcc-9-dev_9.1.0-1_amd64.deb
> > debian-binary
> > control.tar.xz
> > data.tar.xz
> 
> Ahh, yes, thanks I should have remembered that... so then my
> question becomes: is the suggestion to replace *both* ar and tar in
> the binary package format with a single flat archive, or to switch
> out the tarballs inside the ar archive but continue using ar to
> aggregate them?

`ar` needs to be replaced for the file size limitation mentioned in the
initial mail: ar represents file size as a 10 digit decimal number[1]
which limits the members (control.tar.*, data.tar.*) to ~10G.

  [1] https://en.wikipedia.org/wiki/Ar_(Unix)#File_header

Replacing `ar` is an incompatible format change.  So if we already do
an incompatible change, it is an appropriate time to bundle any other
incompatible changes (if there are any).  That is why I suggested that
it might be useful to also replace the `tar` archives with another
format.

Ansgar

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git [and 1 more messages]"):
> On Wed, May 08, 2019 at 04:14:00PM +0100, Ian Jackson wrote:
> 
>  * Debian should provide source code as git branches which:
>   - can be built using a standard set of runes
>   - will produce the same binaries as official Debian ones 
>   - can be reliably located
>   - can be easily modified (using standard git commands)
>   - contain the git histories we are actually using ourselves
> 
> There is only one way to do this.  It is `dgit push[-source]'.
> Vcs-Git and Salsa do not provide this. 
> 
> ---end quote---
> 
> and I'm not sure I agree this is true, to me Vcs-Git and Salsa do provide all
> of this, *if* Vcs-Git is set.

No, sadly not.

   - can be built using a standard set of runes

This is not true of Vcs-Git because the format of the Vcs-Git
repository is not standardised.

   - will produce the same binaries as official Debian ones 

This is not true of Vcs-Git because it is not trivial (or even, in the
general case, systematically possible) to find the right git tags
corresponding to a particular upload.  DEP-14 does help here but not
everyone agrees with it: eg, git-dpm has slightly nonstandard version
mangling and its maintainer has resisted changing; packaging-only
monorepos cannot use DEP-14 tags because DEP-14 tags aren't
package-qualified.

debcheckout *certainly* doesn't do this.  It just gives you the
current master which may not have been uploaded anywhere.

   - can be reliably located

Vcs-Git does help find the repo, but finding the right branch/commit
is difficult (see above).  Also, Vcs-Git gives wrong answers if there
was an upload (eg an NMU) which was not pushed to salsa.  If you use
debcheckout you might miss an important security update this way!

   - can be easily modified (using standard git commands)

This is not true of Vcs-Git because of the repository format problem.
The most common tree format is patches-unapplied, which does not meet
this requirement.  git-grep does not work properly; git-blame does not
work properly; git-merge does not work properly; git-cherry-pick does
not work properly. etc. etc.

   - contain the git histories we are actually using ourselves

Vcs-Git does meet this requirement.

> So, IOW, I can see problems with individual packages here but not with the
> general workflow/tool of using vcs-git: and debcheckout.

See also my response to some similar points raised by Simon McVittie:

  https://lists.debian.org/debian-devel/2019/05/msg00085.html

Regards,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Holger Levsen]

On Thu, May 09, 2019 at 01:46:19PM +0100, Ian Jackson wrote:
> debcheckout *certainly* doesn't do this.  It just gives you the
> current master which may not have been uploaded anywhere.

nope:

$ debcheckout munin
declared git repository at https://salsa.debian.org/debian/munin.git -b debian
git clone https://salsa.debian.org/debian/munin.git -b debian munin ...
[...]

(not replying to the other points because enotime, sorry.)


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Ian Jackson]

Ansgar writes ("Re: .deb format: let's use 0.939, zstd, drop bzip2"):
> `ar` needs to be replaced for the file size limitation mentioned in the
> initial mail: ar represents file size as a 10 digit decimal number[1]
> which limits the members (control.tar.*, data.tar.*) to ~10G.
...
> Replacing `ar` is an incompatible format change.  So if we already do
> an incompatible change, it is an appropriate time to bundle any other
> incompatible changes (if there are any).  That is why I suggested that
> it might be useful to also replace the `tar` archives with another
> format.

As has been pointed out, we have done many incompatible format
changes.  Every new compression algorithm is one.  It isn't really a
big problem, when managed properly.

So I strongly disagree.  The archive size limit is getting more and
more annoying.  We should not let fixing that be entangled with some
random other nice-to-haves.

Personally I still like my multi-ar-member proposal here
  https://lists.debian.org/debian-dpkg/2016/05/msg00027.html
Guillem didn't seem entirely unreceptive but nothing came of it.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Campbell]

On Thu, 2019-05-09 at 12:51 +, Holger Levsen wrote:
> On Thu, May 09, 2019 at 01:46:19PM +0100, Ian Jackson wrote:
> > debcheckout *certainly* doesn't do this.  It just gives you the
> > current master which may not have been uploaded anywhere.
> 
> nope:
> 
> $ debcheckout munin
> declared git repository at https://salsa.debian.org/debian/munin.git
> -b debian
> git clone https://salsa.debian.org/debian/munin.git -b debian munin
> ...
> [...]

I don't think that one counterexample invalidates what Ian says in the
common case, it is certainly true that the most usual thing you get
from debcheckout is master regardless that some (minority of)
maintainers might have tried to be smarter.

In any case for me it gets the `debian-experimental` branch, not really
sure why and it's probably not what I expected/wanted[0]:

   $ debcheckout munin
   declared git repository at https://salsa.debian.org/debian/munin -b 
debian-experimental
   git clone https://salsa.debian.org/debian/munin -b debian-experimental munin 
...
   Cloning into 'munin'...
   warning: redirecting to https://salsa.debian.org/debian/munin.git/
remote: Enumerating objects: 53812, done.
remote: Counting objects: 100% (53812/53812), done.
remote: Compressing objects: 100% (13784/13784), done.
remote: Total 53812 (delta 37431), reused 53214 (delta 36851)
Receiving objects: 100% (53812/53812), 12.51 MiB | 1.31 MiB/s, done.
Resolving deltas: 100% (37431/37431), done.
warning: redirecting to https://salsa.debian.org/debian/munin.git/

I think at best all this example really does is mutate Ian's "It just gives you 
the
current master which may not have been uploaded anywhere" into "It just
gives you some branch which may not have been uploaded anywhere"?

Ian.

[0] and the manpage doesn't seem to suggest a way to request a
particular suite.

Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]

[Ian Jackson]

Holger Levsen writes ("Re: Preferred git branch structure when upstream moves 
from tarballs to git [and 1 more messages]"):
> On Thu, May 09, 2019 at 01:46:19PM +0100, Ian Jackson wrote:
> > debcheckout *certainly* doesn't do this.  It just gives you the
> > current master which may not have been uploaded anywhere.
> 
> nope:
> 
> $ debcheckout munin
> declared git repository at https://salsa.debian.org/debian/munin.git -b debian
> git clone https://salsa.debian.org/debian/munin.git -b debian munin ...
> [...]

Sorry, I was speaking loosely.

I meant it gives you the current head of the maintainer(s)' main
Debian sid packaging branch.  Which may be (and often is) ahead of the
archive, because things have been pushed to it and not uploaded yet.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Andrej Shadura]

Hi,


On Thu, 9 May 2019, 15:57 Ian Jackson, 
wrote:

> Ansgar writes ("Re: .deb format: let's use 0.939, zstd, drop bzip2"):
> > `ar` needs to be replaced for the file size limitation mentioned in the
> > initial mail: ar represents file size as a 10 digit decimal number[1]
> > which limits the members (control.tar.*, data.tar.*) to ~10G.
> ...
> > Replacing `ar` is an incompatible format change.  So if we already do
> > an incompatible change, it is an appropriate time to bundle any other
> > incompatible changes (if there are any).  That is why I suggested that
> > it might be useful to also replace the `tar` archives with another
> > format.
>
> As has been pointed out, we have done many incompatible format
> changes.  Every new compression algorithm is one.  It isn't really a
> big problem, when managed properly.
>
> So I strongly disagree.  The archive size limit is getting more and
> more annoying.  We should not let fixing that be entangled with some
> random other nice-to-haves.
>
> Personally I still like my multi-ar-member proposal here
>   https://lists.debian.org/debian-dpkg/2016/05/msg00027.html
> Guillem didn't seem entirely unreceptive but nothing came of it.
>


How about the format opkg used for some time, which is a .deb file but with
tar as the outer container format instead of ar?


-- 
Cheers,
  Andrej

>

Bug#928724: ITP: opensbi -- RISC-V Open Source Supervisor Binary Interface

[Vagrant Cascadian]

Package: wnpp
Severity: wishlist
Owner: Vagrant Cascadian 
X-Debbugs-Cc: debian-devel@lists.debian.org, mer...@debian.org, 
debian-ri...@lists.debian.org

* Package name: opensbi
  Version : 0.3+
  Upstream Author : Anup Patel/Western Digital, other contributors
* URL : https://github.com/riscv/opensbi
* License : BSD-2, Apache 2.0, GPL-2+
  Programming Lang: C
  Description : RISC-V Open Source Supervisor Binary Interface

The **RISC-V Supervisor Binary Interface (SBI)** is the recommended interface
between:

1. A platform-specific firmware running in M-mode and a bootloader, a
   hypervisor or a general-purpose OS executing in S-mode or HS-mode.
2. A hypervisor running in HS-mode and a bootloader or a general-purpose OS
   executing in VS-mode.

The *RISC-V SBI specification* is maintained as an independent project by the
RISC-V Foundation on [Github] (https://github.com/riscv/riscv-sbi-doc).

The goal of the OpenSBI project is to provide an open-source reference
implementation of the RISC-V SBI specifications for platform-specific firmwares
executing in M-mode (case 1 mentioned above). An OpenSBI implementation can be
easily extended by RISC-V platform and system-on-chip vendors to fit a
particular hardware configuration.

...

An SBI implementation is needed in order to boot RISC-V systems. This
package initially will at least enable loading u-boot in qemu
sufficient to boot a linux kernel and initramfs.


A similar project is the RISC-V Proxy Kernel and Boot Loader
(a.k.a. BBL):

  https://github.com/riscv/riscv-pk

But BBL requires a compilation step to embed the bootloader and/or
kernel into a payload every time you upgrade the kernel and/or
bootloader. It is possible with OpenSBI to load an arbitrary payload
without requiring a compilation step in some cases (e.g. qemu).


Karsten Merker has offered to co-maintain (who has also been
contributing upstream); not sure if we'll need a team just yet.


Initial rough cut of packaging:

  https://salsa.debian.org/vagrant/opensbi

It cross-compiles an arch:all firmware image usable with qemu+u-boot.

Help with improving the package description and a few remaining
lintian issues would be great!


live well,
  vagrant


signature.asc
Description: PGP signature

Bug#928731: ITP: ruby-has-secure-token -- provides an easy way to generate uniques random tokens for any model in ruby on rails

[Manas Kashyap]

Package: wnpp
Severity: wishlist
Owner: Manas Kashyap 
X-Debbugs-CC: debian-devel@lists.debian.org, debian-r...@lists.debian.org

* Package name: ruby-has-secure-token
  Version : 1.0.0
 Upstream Author : Roberto Miranda
* URL : https://github.com/robertomiranda/has_secure_token
* License : MIT
  Programming Lang: Ruby
  Description : It provides an easy way to generate uniques random
tokens for any model in ruby on rails
  .
SecureRandom::base58 is used to generate the 24-character unique tokens, so
collisions are highly unlikely.

Work-needing packages report for May 10, 2019

[wnpp]

The following is a listing of packages for which help has been requested
through the WNPP (Work-Needing and Prospective Packages) system in the
last week.

Total number of orphaned packages: 1413 (new: 14)
Total number of packages offered up for adoption: 160 (new: 0)
Total number of packages requested help for: 55 (new: 0)

Please refer to https://www.debian.org/devel/wnpp/ for more information.



The following packages have been orphaned:

   commando (#928541), orphaned 3 days ago
 Description: wrapper for argparse to define declaratively (Python 2)
 Installations reported by Popcon: 18
 Bug Report URL: https://bugs.debian.org/928541

   el-get (#928537), orphaned 3 days ago
 Description: install and manage elisp code for Emacs
 Installations reported by Popcon: 65
 Bug Report URL: https://bugs.debian.org/928537

   gsutil (#928528), orphaned 3 days ago
 Description: configure and manage Grandstream BudgeTone 100 VOIP and
   GX2000 phones
 Installations reported by Popcon: 26
 Bug Report URL: https://bugs.debian.org/928528

   libapache2-mod-defensible (#928535), orphaned 3 days ago
 Description: module for Apache2 which provides DNSBL usage
 Installations reported by Popcon: 33
 Bug Report URL: https://bugs.debian.org/928535

   pisg (#928534), orphaned 3 days ago
 Description: Perl IRC Statistics Generator
 Installations reported by Popcon: 56
 Bug Report URL: https://bugs.debian.org/928534

   python-first (#928536), orphaned 3 days ago
 Description: simple function that returns the first true value from
   an iterable
 Installations reported by Popcon: 8
 Bug Report URL: https://bugs.debian.org/928536

   python-fswrap (#928542), orphaned 3 days ago
 Description: unified object oriented interface to file system
   objects
 Installations reported by Popcon: 6
 Bug Report URL: https://bugs.debian.org/928542

   rebuildd (#928532), orphaned 3 days ago
 Description: build daemon aiming at rebuilding Debian packages
 Installations reported by Popcon: 8
 Bug Report URL: https://bugs.debian.org/928532

   sysrqd (#928529), orphaned 3 days ago
 Description: small daemon intended to manage Linux SysRq over
   network
 Installations reported by Popcon: 43
 Bug Report URL: https://bugs.debian.org/928529

   tetrinetx (#928533), orphaned 3 days ago
 Description: game server for Tetrinet
 Installations reported by Popcon: 183
 Bug Report URL: https://bugs.debian.org/928533

   varmon (#928531), orphaned 3 days ago
 Description: VA RAID monitor
 Installations reported by Popcon: 6
 Bug Report URL: https://bugs.debian.org/928531

   webcamd (#928540), orphaned 3 days ago
 Description: Capture images from video devices
 Installations reported by Popcon: 63
 Bug Report URL: https://bugs.debian.org/928540

   wmail (#928539), orphaned 3 days ago
 Description: WindowMaker docklet watching your inbox
 Installations reported by Popcon: 23
 Bug Report URL: https://bugs.debian.org/928539

   xpyb (#928538), orphaned 3 days ago
 Description: Python bindings to XCB
 Installations reported by Popcon: 58
 Bug Report URL: https://bugs.debian.org/928538

1399 older packages have been omitted from this listing, see
https://www.debian.org/devel/wnpp/orphaned for a complete list.



No new packages have been given up for adoption, but a total of 160 packages
are awaiting adoption.  See https://www.debian.org/devel/wnpp/rfa_bypackage
for a complete list.



For the following packages help is requested:

   autopkgtest (#846328), requested 890 days ago
 Description: automatic as-installed testing for Debian packages
 Reverse Depends: autodeb-worker debci-worker
 Installations reported by Popcon: 1165
 Bug Report URL: https://bugs.debian.org/846328

   balsa (#642906), requested 2783 days ago
 Description: An e-mail client for GNOME
 Installations reported by Popcon: 700
 Bug Report URL: https://bugs.debian.org/642906

   broadcom-sta (#886599), requested 486 days ago (non-free)
 Description: Broadcom STA Wireless driver (non-free)
 Installations reported by Popcon: 1851
 Bug Report URL: https://bugs.debian.org/886599

   cargo (#860116), requested 758 days ago
 Description: Rust package manager
 Reverse Depends: cargo-vendor dh-cargo
 Installations reported by Popcon: 982
 Bug Report URL: https://bugs.debian.org/860116

   cyrus-sasl2 (#799864), requested 1324 days ago
 Description: authentication abstraction library
 Reverse Depends: 389-ds-base 389-ds-base-legacy-tools
   389-ds-base-libs adcli autofs-ldap cairo-dock-mail-plug-in
   claws-mail claws-mail-acpi-notifie

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Guillem Jover]

Hi!

On Wed, 2019-05-08 at 19:38:26 +0200, Adam Borowski wrote:
> First, the 0.939 format, as described in "man deb-old".  While still being
> accepted by dpkg, it had been superseded before even the very first stable
> release.  Why?  It has at least two upsides over 2.0:

I'll try to detangle the discussion and address this first. Some of
what I'm going to write has already been writen in the thread, but
I'm just going to condense and give it some additional context and
lay down the direction I'd like to go with.

To recap, format 0.93x has multiple problems:

  - Cannot be handled with stock tools.
  - Not easily extensible.
  - Bad data alignment.
  - Bad commpression support.
  - Bad tool coverage (see below).

I don't think it's correct that most tools support that format, from the
list of programs that I've tracked that handle .deb directly, I'd even
say almost none do . This
list does not include many projects/programs not within Debian handling
.deb archives directly.


The size limit is indeed a problem, and was already known and tracked
in deb(5) and , see
the “.deb size limit” item there, and then later discussed in
. And
while I think the workarounds I listed there are probably still valid
in most cases, if this is affecting people then prioritizing fixing it
now would be good.

The crazy idea I came up with at the time was to use a dual-format PAX+ar
container (that would embed the ar(5) header in the first PAX name entry).
This would make old tools at least detect this is a .deb package, with a
higher major version.

  

But I guess I was never sold on it either, and thinking about it, the
tradeoff does not really look very good. file(1) does not even recognize
it out-of-the-box as a .deb anyway, and we'd just get a nicer error
message from some of the tools handling .debs, but all of them need to
be updated anyway to support any new format. It also destroys some of the
nice properties of the 2.x format, namely:

  - Not requiring special tools to build/extract.
  - Using a non-widespread format (PAX).

Getting rid of ar(5) also would make the format more portable, as the
ar(5) format does change depending on the Unix system! Even besides the
main common format and its BSD and GNU variants, there are other
wildly different layouts. It would also mean we do not need binutils
to analyze them when there is no dpkg-deb around.

For the same reason using PAX would probably be a bad idea, as it's a
format that has unfortunately not really caught up, and takes more
space due to the additional headers, and we do not really need xattr
in the contains. I went for that for its unlimited length metadata, but
since dpkg 1.18.24 that should not be an issue as I implemented GNU
large file metadata support which means we have pretty much "unlimited"
length metadata, and I'd say its encoding is more widespread than PAX
(for example star supports it).

So I think Andrej is on the spot, and we should just switch from ar(5)
to tar(5) as the container, but not to PAX, just the GNU extensions we
already support, which would only be used when necessary. And ignore
any crazy idea of embedding an ar header inside the first member, as
that will just complicate matters and be cruft once we have switched.
So given that we'd need to modify any program handling .debs directly
anyway, I'd go for the most straightforward and simple of the options.

I'll propose an actual diff I've got here of deb(5) tomorrow, but
otherwise if there are no great concerns, I'd like to start adding
support for this for dpkg 1.20.x.

Thanks,
Guillem

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Adam Borowski]

On Wed, May 08, 2019 at 09:53:19PM +0100, Ian Jackson wrote:
> (adding debian-dpkg)

Probably too late to move the thread :(
 
> Adam Borowski writes (".deb format: let's use 0.939, zstd, drop bzip2"):
> > First, the 0.939 format, as described in "man deb-old".  While still being
> > accepted by dpkg, it had been superseded before even the very first stable
> > release.  Why?  It has at least two upsides over 2.0:
> > * it's faster by a small but non-negligible factor.  A task "unpack all
> >   packages in default XFCE GUI install" gets done by stock dpkg (after
> >   repacking everything as gzip) 3% faster.
> 
> I'm not sure why it should be faster.

Now this is interesting; retested in a reproducible way:

Repack tools: http://ix.io/1HNB http://ix.io/1HNC
Corpus used: http://ix.io/1Iyg

Benchmark harness (preceded by warm-up to avoid thermal effects):
for x in gz0.93 gz2;do for i in {0..100};do rm -rf /tmp/target;/usr/bin/time -p 
-o /dev/stdout parallel -i dpkg-deb -x '{}' /tmp/target -- $x/*.deb;done|tee 
/tmp/$x;done

Results (median):
gz0.93: real 1.28   user 19.68
gz2:real 1.33   user 20.67

But, with -j1, I got real 16.14 for both in the first run (median of 11).

> As the person who deprecated deb-old in favour of the current format,
> my motives were:
>  - the old format was a real pain to unpack without a custom utility
> (this used to be a much more serious problem)

I understand that this was an issue when prototyping, but have you used ar
to manipulate .deb archives even once this millenium?  By now, the deb
format is common, while ar is an obscure implementation detail.

>  - the old format was not very extensible.

As in, can't change the compression used?

Only after looking at ways to add 0.93:zstd to real dpkg I noticed that it
relies on 2.0 ar's member filenames to select decompressor.  In my own
experiments, I used libarchive which doesn't look at file extensions at all.

> We use the extensibility for compression format changes, but
> compressors all have magic numbers and we could just use those.

Right.

> It would be much less convenient to change our archive format from tar
> to something else, as proposed by Ansgar, without this extensibility.
> (I don't necessarily think Ansgar's idea is a good one, but it makes
> an example here.)

The control tarball allows a lot of room for extensions, and it has been
used for them many times.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Have you accepted Khorne as your lord and saviour?
⠈⠳⣄

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Ondřej Surý]

> On 10 May 2019, at 10:39, Adam Borowski  wrote:
> 
> I understand that this was an issue when prototyping, but have you used ar
> to manipulate .deb archives even once this millenium?  By now, the deb
> format is common, while ar is an obscure implementation detail.

I did used ar to unpack .deb several times, usually when I had to pull a file
on a non-Debian system (or when I could not remember the dpkg-deb syntax
and I was lazy, but that doesn’t count :)

Ondrej
--
Ondřej Surý
ond...@sury.org

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Adam Borowski]

On Fri, May 10, 2019 at 05:18:18AM +0200, Guillem Jover wrote:
> On Wed, 2019-05-08 at 19:38:26 +0200, Adam Borowski wrote:
> > First, the 0.939 format, as described in "man deb-old".  While still being
> > accepted by dpkg, it had been superseded before even the very first stable
> > release.  Why?  It has at least two upsides over 2.0:

> To recap, format 0.93x has multiple problems:
> 
>   - Cannot be handled with stock tools.

"ar" is an obscure historical thing, akin to "cpio" or such.  It is used
deep within the format, but I wouldn't call this part an upside at all.

>   - Not easily extensible.

Huh?  Seems exactly as extensible as 2.0 (where all deployed extensions went
to control.tar).

>   - Bad data alignment.

Yeah, but it's still faster than 2.0.  And I don't expect decompressors to
care about alignment.  Might matter for "cat", though -- but I don't imagine
many uncompressed archives.  Heck, if we'd start 3.0, I'd recommend lz4
instead.

>   - Bad commpression support.

Trivial to add.

>   - Bad tool coverage (see below).
> 
> I don't think it's correct that most tools support that format, from the
> list of programs that I've tracked that handle .deb directly, I'd even
> say almost none do .

Most of those have no business looking at the format's details, just the
payload.

> The crazy idea I came up with at the time was to use a dual-format PAX+ar
> container (that would embed the ar(5) header in the first PAX name entry).
> This would make old tools at least detect this is a .deb package, with a
> higher major version.
[...]
> So I think Andrej is on the spot, and we should just switch from ar(5)
> to tar(5) as the container

I would heavily advise against archive-in-archive.  Especially not tar, with
its block madness.  The blocks disappear when compressed but you're not
going to compress the outer layer.  Also, you can't shed the outer layer of
tar without a filter.

According to the benchmarks I just posted, even less than 1/3 loaded
processor is already bottlenecked on passing data from layer to layer.  I
tried a zero-copy implementation with libarchive's callbacks, but it doesn't
seem to help:

gzip, median of 101, libarchive implementation:
0.93: real 0.97 user 14.74
2.0:  real 0.99 user 15.89

> I'll propose an actual diff I've got here of deb(5) tomorrow, but
> otherwise if there are no great concerns, I'd like to start adding
> support for this for dpkg 1.20.x.

Let's not be hasty -- unlike 0.93 which has an existing (if spotty) support,
a complete format break should be better researched.  Ansgar's concerns for
example should be at least considered.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Did ya know that typing "test -j8" instead of "ctest -j8"
⢿⡄⠘⠷⠚⠋⠀ will make your testsuite pass much faster, and fix bugs?
⠈⠳⣄

Re: .deb format: let's use 0.939, zstd, drop bzip2

[Ansgar Burchardt]

Ian Jackson writes:
> Ansgar writes ("Re: .deb format: let's use 0.939, zstd, drop bzip2"):
>> `ar` needs to be replaced for the file size limitation mentioned in the
>> initial mail: ar represents file size as a 10 digit decimal number[1]
>> which limits the members (control.tar.*, data.tar.*) to ~10G.
> ...
>> Replacing `ar` is an incompatible format change.  So if we already do
>> an incompatible change, it is an appropriate time to bundle any other
>> incompatible changes (if there are any).  That is why I suggested that
>> it might be useful to also replace the `tar` archives with another
>> format.
>
> As has been pointed out, we have done many incompatible format
> changes.  Every new compression algorithm is one.  It isn't really a
> big problem, when managed properly.
>
> So I strongly disagree.  The archive size limit is getting more and
> more annoying.  We should not let fixing that be entangled with some
> random other nice-to-haves.

Changing container formats is a more invasive change than just
compression algorithms (of which there are already several anyway);
I believe it is a bad idea to create too many new formats that
would require long-term support in dpkg.

Ansgar