Re: Preferred git branch structure when upstream moves from tarballs to git
Sam Hartman writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> Ian Jackson writes:
> > The latter point is because using dgit push is an ethical
> > imperative, not because the two somehow have some deep
> > technical linkage. IMO almost *any* tutorial being written now
> > about how to do Debian packaging, and which mentions git at
> > all, ought to say to use dgit.
>
> I think this is at the crux of our disagreement.
> I'm going to need to think about this for myself.
OK...
> I think it's safe to say there is not a project consensus that dgit is
> an ethical imperative.
Indeed.
> And yes understanding that you believe that's the case makes it a lot
> easier to understand the rest of your mail.
Good.
> As I said I'll need to think about whether I agree with that. A lot of
> my thinking in this discussion is that I consider dgit a nice to have,
> not a requirement.
I guess I have stated the reasons already, but to recap:
* Users should be able to easily change the software that runs
on their computer.
* "Easily" requires source code which:
- can be built using a standard set of runes
- will produce the same binaries as official Debian ones
- can be reliably located
- can be easily modified
apt provides this, if you think tarballs+diffs are enough, but:
* Nowadays people want the source code as the git history.
When upstream and Debian are both using git, then arguably
tarballs+diffs is not really the preferred form for modification.
I.e. the .dsc source package is not really the source code.
>From these we can conclude:
* Debian should provide source code as git branches which:
- can be built using a standard set of runes
- will produce the same binaries as official Debian ones
- can be reliably located
- can be easily modified (using standard git commands)
- contain the git histories we are actually using ourselves
There is only one way to do this. It is `dgit push[-source]'.
Vcs-Git and Salsa do not provide this. Occasionally people propose
alternative technical approaches. So in theory there might be some
other way of achieving the goal. But people have been proposing
technical approaches to this question for a very long time - a decade
at least. Joey Hess and I came up with a design which was actually
implementable, politically practical, deployable and operable, and I
have implemented it, negotiated for it, deployed it, and am now
operating it.
dgit push involves minimal change to maintainer workflows and does not
gore anyone's ox, unlike all most of the (at best vapourware)
proposals before and since.
> If you accept the premis that everyone should be using dgit, I agree
> with you.
> I don't think the gbp documentation has adopted that premis.
Indeed.
> > The only reason dgit needs to get involved in your build
> > process is because of the .gitignore bug. (#908747)
>
> Well, and dgit wants to construct my dsc, which means it wants to
> construct my changes, which... means it wants to call sbuild or
> something else for me.
The only reason dgit wants to construct your dsc is #908747. If you
can avoid #908747 some other way then you can build things however you
like and dgit push will work just fine. [1]
Of course if you are doing source-only uploads, as we should be doing
all the time (we're still shipping maintainers' binaries?!!11eleven)
then you do not need to really "build" anything and `dgit push-source'
suffices. Of course it makes a .dsc but that is an affordance rather
than an inconvenience.
I recognise that your message was in some sense "I will go away and
think" rather than an invitation for me to further engage in advocacy
to you. I hope you don't mind that I've done so anyway. Feel free
not to reply. But I did want to say these things for the benefit of
other readers.
And yes, do please file bugs if you disagree with dgit about things.
I welcome inchoate bug reports, although I don't promise not to reply
with "that doesn't sound right, please send steps to reproduce" :-).
Regards,
Ian.
[1] I would love it if dgit never needed to get involved in building.
There's tons of code in dgit to deal with the quirks of everyone's
favourite build tools. I appreciate that wrapping the build tools in
yet another layer is very undesirable.
All of this is only necessary because of #908747.
Why didn't I try to get #908747 fixed ? Well, it was one of dgit's
original principles that it would not require anyone else's permission
or assistance, at least not for anything remotely controversial.
Without that principle the project would have been totally stalled.
Looking at the history of #908747 you can see that I was right.
The result is that dgit contains many many ugly workarounds. Each of
these workarounds generated a bug report against some other Debian
package. For the most annoying bugs, requiring the worst
workarounds, I don't think any
Re: Preferred git branch structure when upstream moves from tarballs to git
On Tue, May 07, 2019 at 07:25:39PM +0100, Ian Jackson wrote: > A maintainer who is currently using gbp pq can (and IMO generally > should) use `dgit push' to do their uploads, rather than dput/dupload. as a data point: I never really got my head around gbp, too many times it came in the way, did things I didnt expect to do (or couldnt easily figure out what it did), so I basically (try to) avoid packages which require gbp in some way. dgit *is* another layer of complexity on top of debuild and git (and other tools), as is gbp, and frankly (or sadly?), for the dgit case I dont see much benefit why I should use it rather than dput: my packages are maintained in git and have the correct Vcs-Git* headers set, so debcheckout will do the right thing and give you the full git history. I *do* appreciate the idea of dgit and the work put into it, I think it goes into the right direction, but to me it's not sliced bread yet, but rather a fancy new machine for slicing bread which is more complicated and harder to grasp, so I'm still on that old slicing machine, which I know since many years and which I can easily fix if the bread gets stuck or the knifes need resharpening or some such. -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: Preferred git branch structure when upstream moves from tarballs to git
Holger Levsen writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> as a data point: I never really got my head around gbp, too many times
> it came in the way, did things I didnt expect to do (or couldnt easily
> figure out what it did), so I basically (try to) avoid packages which
> require gbp in some way.
Fair enough.
I'm not sure what git branch format you are using. (Maybe you said it
earlier in this thread.) Packaging-only ? git-merge ?
> dgit *is* another layer of complexity on top of debuild and git (and other
> tools), as is gbp, and frankly (or sadly?), for the dgit case I dont see
> much benefit why I should use it rather than dput: my packages are
> maintained in git and have the correct Vcs-Git* headers set, so
> debcheckout will do the right thing and give you the full git history.
You should use dgit for the benefit of users. See my other mail which
answers why Vcs-Git and debcheckout is not enough.
Indeed, you yourself say you avoid gbp but for many packages, the
Vcs-Git header gives you a patches-unapplied format which requires[1]
gbp pq to switch to a patches-applied view before you build it.
([1] there are other ways to apply the patches but this is the
easiest.)
> I *do* appreciate the idea of dgit and the work put into it, I think it
> goes into the right direction, but to me it's not sliced bread yet, but
> rather a fancy new machine for slicing bread which is more complicated
> and harder to grasp, so I'm still on that old slicing machine, which I
> know since many years and which I can easily fix if the bread gets stuck
> or the knifes need resharpening or some such.
Yes, I quite understand this point of view.
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Preferred git branch structure when upstream moves from tarballs to git
On Wed, May 08, 2019 at 12:18:41PM +0100, Ian Jackson wrote: > I'm not sure what git branch format you are using. (Maybe you said it > earlier in this thread.) Packaging-only ? git-merge ? both. > You should use dgit for the benefit of users. See my other mail which > answers why Vcs-Git and debcheckout is not enough. could you be so kind and provide a pointer, this thread is rather long already? (Maybe this is also worth an FAQ entry somewhere..) > > I *do* appreciate the idea of dgit and the work put into it, I think it > > goes into the right direction, [...] > Yes, I quite understand this point of view. thanks. (also for the work and thoughts you put into dgit!) -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Ważne Pytanie
Dzień dobry, zajmujemy się tworzeniem nowoczesnych, responsywnych*_ Stron i Sklepów internetowych. _* Jeśli chcieliby Państwo otrzymać bezpłatną propozycję w tym zakresie dla Państwa firmy prosimy o odpowiedź *TAK.* - - - - Pozdrawiamy Serdecznie, Twórcy Stron i Sklepów WWW
Re: Preferred git branch structure when upstream moves from tarballs to git
On 2019/05/08 13:23, Holger Levsen wrote: >> You should use dgit for the benefit of users. See my other mail which >> answers why Vcs-Git and debcheckout is not enough. > > could you be so kind and provide a pointer, this thread is rather long > already? (Maybe this is also worth an FAQ entry somewhere..) I'd like to second that request, I mean to go through this thread again when I have some free moments, but a FAQ would help a lot to assimilate all this information. (I'm in a similar boat to Holger with my experience to gbp, I tried going through the documentation on the wiki a few times and just got frustrated. I just use packaging and Vcs-fields and tags and nothing fancier than that, which works for the few dozen packages I have and I think the workflow is trivial enough so that downstream consumers wouldn't have any problem whatsoever figuring out how to make use of it, but I suppose as that list grows it will become important to have worflows that scale up a bit more.) > thanks. (also for the work and thoughts you put into dgit!) Ditto. -Jonathan -- ⢀⣴⠾⠻⢶⣦⠀ Jonathan Carter (highvoltage) ⣾⠁⢠⠒⠀⣿⡁ Debian Developer - https://wiki.debian.org/highvoltage ⢿⡄⠘⠷⠚⠋ https://debian.org | https://jonathancarter.org ⠈⠳⣄ Be Bold. Be brave. Debian has got your back.
Bug#928657: ITP: golang-github-naegelejd-go-acl -- Golang POSIX 1e ACL bindings
Package: wnpp Severity: wishlist Owner: Mikhail f. Shiryaev * Package name: golang-github-naegelejd-go-acl Version : 0.0~git20160113.f33f861-1 Upstream Author : Joseph Naegele * URL : https://github.com/naegelejd/go-acl * License : MIT Programming Lang: Go Description : Golang POSIX 1e ACL bindings go-acl Golang POSIX.1e ACL bindings. Essentially bindings to /usr/include/sys/acl.h notesmac os x Mac OS X does not seem to support basic POSIX1.e ACLs. They do provide the POSIX API for NFSv4 ACLs. It would be nice for this package to also support NFSv4 ACLs. freebsd By default, FreeBSD does not enable POSIX1.e ACLs on the root partition. To enable them, reboot into single-user mode and execute: $ tunefs -a enable $ reboot . Source: https://www.freebsd.org/doc/handbook/fs-acl.html info The IEEE POSIX.1e specification describes five security extensions to the base POSIX.1 API: Access Control Lists (ACLs), Auditing, Capabilities, Mandatory Access Control, and Information Flow Labels. The specificaiton was abandoned before finalization, however most UNIX-like operating systems have some form of ACL implementation. . Source: http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=posix1e copying Copyright (c) 2015 Joseph Naegele. See LICENSE file.
Re: Preferred git branch structure when upstream moves from tarballs to git
On Wed, 08 May 2019 at 12:18:41 +0100, Ian Jackson wrote: > Indeed, you yourself say you avoid gbp but for many packages, the > Vcs-Git header gives you a patches-unapplied format which requires[1] > gbp pq to switch to a patches-applied view before you build it. This step is not required: for 3.0 (quilt) packages, dpkg-source is happy with either patches-applied or patches-unapplied source directories, as long as you're at one extreme or the other (applying half the patch series doesn't work). Here's a smallish package that I happen to maintain using gbp-pq, built without using gbp: ~/tmp$ debcheckout bubblewrap ... ~/tmp$ cd bubblewrap ~/tmp/bubblewrap$ dpkg-buildpackage -b ... dpkg-source --before-build . dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying debian/Use-Python-3-for-test-demo-code.patch ... a build happens ... dpkg-genchanges --build=binary >../bubblewrap_0.3.3-1_amd64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-source: info: unapplying debian/Use-Python-3-for-test-demo-code.patch dpkg-buildpackage: info: binary-only upload (no source included) (This probably only works for 3.0 (quilt) source packages, but gbp pq makes little sense for any other format.) > ([1] there are other ways to apply the patches but this is the > easiest.) Easier than the tool you have to use anyway (directly or indirectly) to build any package? smcv
Re: Preferred git branch structure when upstream moves from tarballs to git
Scott Kitterman writes: > On May 7, 2019 8:50:57 PM UTC, Sam Hartman wrote: >>> "Ian" == Ian Jackson writes: >>Ian> The latter point is because using dgit push is an ethical >>Ian> imperative, not because the two somehow have some deep >> Ian> technical linkage. IMO almost *any* tutorial being written now >>Ian> about how to do Debian packaging, and which mentions git at >>Ian> all, ought to say to use dgit. >> >>I think this is at the crux of our disagreement. >>I'm going to need to think about this for myself. >>I think it's safe to say there is not a project consensus that dgit is >>an ethical imperative. >>And yes understanding that you believe that's the case makes it a lot >>easier to understand the rest of your mail. >> >>As I said I'll need to think about whether I agree with that. A lot of >>my thinking in this discussion is that I consider dgit a nice to have, >>not a requirement. > > For what it's worth, when you start calling me names (unethical) > because I'm not using your shiny new toy, my immediate reaction is to > decide to ignore further discussion on the topic. I think there's a considerable difference between: "I think ethically, everyone should do X, and I will try and persuade you that this is correct" And "If you disagree with me that everyone should do X then you are unethical" ...and that conflating the two as you appear to be doing so here is quite unhelpful. Regards, Matthew -- "At least you know where you are with Microsoft." "True. I just wish I'd brought a paddle." http://www.debian.org
Configure your PC to contribute to Debian community
Hey there, I've been using Debian from couples of years but haven't contributed yet back to community. I want to contribute to Debian by maintaining packages and fixing bugs. Since I'm using Debian for work purpose also so, I don't want to mess-up with my system by installing unstable packages or libraries. Is there a way to get isolation for work & contribution purpose to keep yourself organized? I can get isolation by using Docker image or install one more copy of Debian in PC and switch between them but that would be painful. I want to hear from contributors & maintainers Which method they are using or prefer to get isolation? Sorry, if this a wrong place to ask this question, then where should I ask? Cheers, VIpul PS: I'm not subscribed to this mailing. signature.asc Description: OpenPGP digital signature
Re: Tell me about your salsa experience
Dmitry Bogatov 于2019年4月22日周一 下午5:56写道： > > > [ I know, it month and half late ] > [ I did my best to recover thread. Sorry, If I failed. ] > > [ Please, CC me if you want me to reply. I'm not subscribed to debian-devel@ ] > > [ Alexander Wirt ] > > Thats where you come in, please tell me how tools like salsa, alioth, > > git, tracker and so on changed to way you work. I want to know > > everything, the good, the bad and so on. > > Glad you asked. So I have excuse for stating my opinion, which is quite > unpopular. > > Summary: Introduction of salsa is nothing short of disaster. > One problem of salsa is that the speed of fork is quite slow. I guess we need something like CoW. > I started working with Debian in mid-2014, when all code lived on > alioth. The best thing about alioth is that I did not interact with it. > > Alioth did not get in my way. I had ssh access to alioth.debian.org, all > operations was simple and intuitive. I had choice of VCS to use, git > hooks were easy to setup, every chore was easy to script. I participated > in two teams -- Haskell and Emacs. Everything was smooth, it is just > matter of file permissions. > > Maybe developers, who granted me access to teams, had to deal with > something more terrible I can imagine. Maybe administering Alioth for > DSA team was nightmare. No idea, I am telling about my experience. > > And then came yet another tragic day for Debian, and Gitlab replaced > alioth.debian.org. It brought pain, inconvenience and friction. > > Performing basic operations with repository now either impossible > (salsa forces foo/bar naming, instead of flexibility of proper > filesystem on Alioth), or requires learning new useless stuff. > > There is no longer proper git hooks. > > Other version control systems are gone. > > In an instant, I became second-class citizen, now everything -- > documentation, processes, defaults -- everything is optimized for > running "modern" browser and pushing buttons. Scriptability is pain. > > That is not all, folks. Salsa brought own issue tracker and concept of > pull-request. So now I can't just mail a patch with reportbug(1) -- > there are chances, that maintainer will either ignore it, because he > only follows salsa issue tracker, or that he will ask you to make a > pull-request on salsa. > > git was step forward from svn/cvs -- now we can work on our version > control system offline. Salsa issue tracker is a same step backward from > debbugs -- it disables offline working with bugs. > > To be fair, there is very minor positive thing in salsa -- Gitlab CI. > Its usefulness is limited, since there is no API to capture output in > real time, so I still have to use sbuild on my local machine, but > ability to rebuild package once in a week and get email on failure is > good thing to prevent package rot. > > I know, Alioth was unmaintained, but just having box with sudo rules > about adduser/usermod and Apache, running cgit would be much better > replacement. Well, this ship sailed; I have been writing scripts to deal > with madness around for a while, and it seems I will have to continue to > do so. > -- > Note, that I send and fetch email in batch, once every 24 hours. > If matter is urgent, try https://t.me/kaction > > -- -- YunQiang Su
Re: Preferred git branch structure when upstream moves from tarballs to git
On Tue, 07 May 2019 at 19:25:39 +0100, Ian Jackson wrote: > What I am primarily advocating for in this thread is that maintainers > should choose `dgit push-source' over `dput' (where this is possible). > This is the only way for a maintainer to provide users with the git > history in a sensible form (ie, a form which does not require the user > to know what special git practices the maintainer has adopted). I think you're implicitly asserting here that the tree that exists in a dgit commit (upstream source as conveyed by the orig tarball, with debian/ added, with patches applied if any) is the only sensible form for the git history of a Debian source package, and I don't think that's something that has consensus. Alternative "shapes" that other people advocate include: debian/ only (a straightforward translation of typical packaging-in-svn to git); patches-unapplied trees (as seen in e.g. the Perl, Python, GNOME teams); debian/ overlaid onto upstream's git repository (as opposed to upstream's tarball code-drops), with any differences between the orig tarball and upstream's git repository papered over by debian/source/local-options (as seen in the Xorg team). I realise that you think (some or all of) those other "shapes" are not sensible, and I realise that dgit's design requires a single canonical result of importing a Debian source package, and I agree that as a general operating principle it's undesirable that different packages have their contents represented in different ways; but I don't think it's necessarily helpful to assert that the representation chosen for dgit is the only one that can be considered sensible, and I think you're in danger of causing people who disagree with you on this point to not consider dgit, which seems like the opposite of what you want. I think the reason people are conflating the contents of the tree with the workflow that was used to arrive at that tree is that the workflow is not completely decoupled from the tree. This is most obvious with git-dpm, which leaves a mechanically-generated file in debian/ that it cannot work without (and that is only practical to maintain by using git-dpm), but it's also true for other representations: for instance gbp-pq normally uses a patches-unapplied tree as the interchange format (requiring dgit to compensate for this difference in assumptions), whereas git-debrebase requires patches-applied if I understand correctly. The major interoperating "families" are: * patches-applied: git-debrebase or dgit-maint-merge (and git-dpm would be in this category if it didn't require extra metadata in debian/). * patches-unapplied: gbp pq, or patches managed out-of-band with quilt or by dropping git format-patch output into d/patches, either with full source in git or with only debian/ in git (built with gbp buildpackage --overlay, or by unpacking the orig tarball into the source directory and avoiding committing it to git). (Native packages, and non-native packages with no patches, can be seen as a special case of either family - if you don't have any patches then it doesn't matter whether you have applied them or not.) I don't think it's coincidence that many of the larger teams in Debian, in particular Perl and Python, have ended up with patches-unapplied. It's a straightforward continuation of how we used to maintain packages in non-git VCSs like svn, it only requires learning to use a tool like gbp pq or quilt if you have enough non-orthogonal patches that they need to be rebased on each other to apply cleanly, and it's a fairly close match for how most non-Debian source packaging formats work (for instance Fedora SRPMs, Arch Linux PKGBUILDs and Gentoo ebuilds all involve committing unified diffs to their respective VCSs alongside the equivalent of debian/control). smcv
Re: Preferred git branch structure when upstream moves from tarballs to git
On Sat, 04 May 2019 at 21:10:34 +0100, Ian Jackson wrote: > So far I have gained the impression that > the kind of people who are using packaging-only git trees tend to be > very conservative, and very comfortable with .dsc/tarball/patch/quilt > based tools, and not really convinced of the usefulness of git. A less common, but IMO valid, reason to use packaging-only git trees is that your upstream source code consists of large non-text objects that aren't managed efficiently by git. The openarena-data family of packages are (I think) the only place I still use packaging-only git trees, and the nexuiz-data package is an example of the sort of monster that will typically be created if you don't. I suppose that's a form of not really being convinced of the usefulness of git: I'm convinced that git is useful for program source code and metadata, but I'm not convinced that it's useful for multiple gigabytes of models, textures etc. that are added and deleted more often than they are edited. If there was a convenient way to use a lookaside object store like git-lfs or git-annex for source packages, then I'd use that. Arguably this is an instance of the same problem as some of those we have around DFSG interpretation and whether "software" refers to programs or to all blobs of bytes: our processes are designed for program source code, and non-program content doesn't always fit very well into those processes; but we can't just not ship the non-program content, because we're trying to provide a self-contained software distribution with no external dependencies. smcv
Re: Preferred git branch structure when upstream moves from tarballs to git
Matthew Vernon writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> Scott Kitterman writes:
> > For what it's worth, when you start calling me names (unethical)
> > because I'm not using your shiny new toy, my immediate reaction is to
> > decide to ignore further discussion on the topic.
I'm sorry, that was not my intention.
> I think there's a considerable difference between:
>
> "I think ethically, everyone should do X, and I will try and persuade
> you that this is correct"
>
> And
>
> "If you disagree with me that everyone should do X then you are
> unethical"
I meant the former. I definitely don't mean to say that all the
people not doing what I think is best, are bad people.
Regards,
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Preferred git branch structure when upstream moves from tarballs to git
Holger Levsen writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> On Wed, May 08, 2019 at 12:18:41PM +0100, Ian Jackson wrote:
> > I'm not sure what git branch format you are using. (Maybe you said it
> > earlier in this thread.) Packaging-only ? git-merge ?
>
> both.
For packaging-only, dgit does not properly support that. Do you have
the upstream source in git form too, and if so how do you combine
them ? Do you manipulate patches using quilt ? [1]
If you use git-merge and have all the source (including upstream
source) in a single git tree there is indeed no need to use gbp pq.
The manpage dgit-maint-merge(7) explains how to publish such a thing
with dgit.
> > You should use dgit for the benefit of users. See my other mail which
> > answers why Vcs-Git and debcheckout is not enough.
>
> could you be so kind and provide a pointer, this thread is rather long
> already? (Maybe this is also worth an FAQ entry somewhere..)
Sorry, I should have done that the first time. I'll reply to
Jonathan and you.
Regards,
Ian.
[1] I have to say that I think quilt is just awful. Perhaps your
mileage varies, but I thought quilt was awful by comparison with
*CVS*, when I first encountered it...
I know it can be work to learn new stuff, and of course it's a pain if
lots of people are advising you to learn different and mutually
incompatible new stuff, but anyone who is still using quilt will
probably benefit a lot from learning how to use gitish tools instead.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Bug#928665: ITP: python-pybigwig -- read/write chromosomal sequence data in bigWig files
Package: wnpp Severity: wishlist Owner: Steffen Moeller * Package name: python-pybigwig Version : 0.3.15 * URL : https://github.com/deeptools/pyBigWig * License : MIT Programming Lang: Python Description : read/write chromosomal sequence data in bigWig files Team maintained on https://salsa.debian.org/med-team/python-pybigwig
Re: Preferred git branch structure when upstream moves from tarballs to git
On Wed, May 08, 2019 at 03:57:08PM +0100, Ian Jackson wrote: > > both. > For packaging-only, dgit does not properly support that. Do you have > the upstream source in git form too, and if so how do you combine > them ? Do you manipulate patches using quilt ? [1] hm/i'm sorry, I cannot really think of such an example of a package I'm maintaining. I've definitly encountered such packages though... > > could you be so kind and provide a pointer, this thread is rather long > > already? (Maybe this is also worth an FAQ entry somewhere..) > Sorry, I should have done that the first time. I'll reply to > Jonathan and you. thank you! > [1] I have to say that I think quilt is just awful. Perhaps your > mileage varies, [...] well, only sometimes. but then, it also does work. > I know it can be work to learn new stuff, and of course it's a pain if > lots of people are advising you to learn different and mutually > incompatible new stuff, but anyone who is still using quilt will > probably benefit a lot from learning how to use gitish tools instead. most of the time I deal with quilt is when I want to modify existing packages, usually not from sid, but from stable or older, and then it feels more straightforward to deal with quilt. often I just dget the sources (from an old release (*)), run git init ; git add * : git commit -m init and fiddle qith quilt & git until it applies nicely. that way, git really supports me nicely, without forcing me to learn anything. there's so much to learn all the time... (*) debcheckout is largely unusable for stretch and older as alioth is gone, so... meh. -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]
Holger Levsen writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> could you be so kind and provide a pointer, this thread is rather long
> already? (Maybe this is also worth an FAQ entry somewhere..)
Jonathan Carter writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> I'd like to second that request, I mean to go through this thread again
> when I have some free moments, but a FAQ would help a lot to assimilate
> all this information.
A FAQ is a really good idea. Thank you for the suggestion.
In the meantime,
> > [Ian Jackson:]
> > > You should use dgit for the benefit of users. See my other mail which
> > > answers why Vcs-Git and debcheckout is not enough.
> >
> > could you be so kind and provide a pointer, this thread is rather long
> > already? (Maybe this is also worth an FAQ entry somewhere..)
I meant this message:
https://lists.debian.org/debian-devel/2019/05/msg00065.html
> (I'm in a similar boat to Holger with my experience to gbp, I tried
> going through the documentation on the wiki a few times and just got
> frustrated. I just use packaging and Vcs-fields and tags and nothing
> fancier than that, which works for the few dozen packages I have and I
> think the workflow is trivial enough so that downstream consumers
> wouldn't have any problem whatsoever figuring out how to make use of it,
> but I suppose as that list grows it will become important to have
> worflows that scale up a bit more.)
I confess I always find these kind of comments (which I hope you won't
mind me parodying as "I don't use anything") a bit confusing because
they don't explain how you perform the key task that tools like gbp pq
are intended to help with.
I think what is going on here is this: to you, what you do for that is
so completely obvious it doesn't occur to you that I don't know what
it is. There is such a gulf here between different people's workflows
that we are having trouble communicating.
If you are maintaining a package in "3.0 (quilt)", you are maintaining
a series of patches to the upstream part of the package. The question
is: what tool(s) do you use to change what changes a patch makes to
the upstream source, edit a patch's accompanying message (if you give
them messages), add a new patch, drop a patch, rebase the patches onto
a new upstream version, and so on.
The answer clearly isn't "nothing" since you're not fiddling with the
individual bits in your computer's RAM by your electrotelekinetic
superpowers :-). At least if you are then awesome but maybe you are
beyond need for version control software.
I guess the answer probably isn't even "I edit the diffs in the
patches, in my text editor". At least I hope not since editing diffs
in a text editor is still horrific even with a really good text
editor. I think that probably many people who say "I don't use
anything fancy" are using quilt, or maybe raw diff and patch ? Maybe
you are editing debian/series with your text editor and occasionally
using rm and mv on patch files ? Or git-mv and git-rm ?
I realise the above might sound facetious. That's not my intent.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Ckghfwssm Ckghfwssm
Ckghfwssm Ckghfwssm, I am Donna Baker. I am a recruiter. Our company is recruiting a outgoing, responsible, punctual person to fill the position of the online manager. After examining a lot of CVs, we would like to see you as our employee. We offer this work as underemployment at the moment. We will allow you to fulfill yourself as a full-time employee after probation. We are also have an interest in you staying with us for a full-day position after probation. Please keep in mind that for getting this vacant post you must be a US citizen or have permission to work in the US. As the correct and prompt fulfillment of your duties will affect the profit of our company straightforward, it is very important for us to take a good candidate. Furthermore, rewards are provided for good performance. You neednât be tied to a specific place â you may work from home. We are waiting for your decision on this invitation. Let us know when you will be ready to start your duties. Please contact us to receive a full set of required documents. Sincerely, Donna Baker
Re: Preferred git branch structure when upstream moves from tarballs to git [and 1 more messages]
On Wed, May 08, 2019 at 04:14:00PM +0100, Ian Jackson wrote: > > > [Ian Jackson:] > > > > You should use dgit for the benefit of users. See my other mail which > > > > answers why Vcs-Git and debcheckout is not enough. > > > > > > could you be so kind and provide a pointer, this thread is rather long > > > already? (Maybe this is also worth an FAQ entry somewhere..) > > I meant this message: > https://lists.debian.org/debian-devel/2019/05/msg00065.html thanks for the pointer, but I don't see the string debcheckout in that message and vcs-git only once, where you write: ---begin quote--- >From these we can conclude: * Debian should provide source code as git branches which: - can be built using a standard set of runes - will produce the same binaries as official Debian ones - can be reliably located - can be easily modified (using standard git commands) - contain the git histories we are actually using ourselves There is only one way to do this. It is `dgit push[-source]'. Vcs-Git and Salsa do not provide this. ---end quote--- and I'm not sure I agree this is true, to me Vcs-Git and Salsa do provide all of this, *if* Vcs-Git is set. And if it's not set, it's either because the package is not in git or because d/control is lacking information, aka the package is buggy. So, IOW, I can see problems with individual packages here but not with the general workflow/tool of using vcs-git: and debcheckout. (or what am I missing?) -- tschau, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: Preferred git branch structure when upstream moves from tarballs to git
Simon McVittie writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> On Wed, 08 May 2019 at 12:18:41 +0100, Ian Jackson wrote:
> > Indeed, you yourself say you avoid gbp but for many packages, the
> > Vcs-Git header gives you a patches-unapplied format which requires[1]
> > gbp pq to switch to a patches-applied view before you build it.
>
> This step is not required: for 3.0 (quilt) packages, dpkg-source is happy
> with either patches-applied or patches-unapplied source directories,
> as long as you're at one extreme or the other
Yes. Perhaps indeed that's what the person I was responding to,
there, does.
However, this is actually quite dangerous. You definitely do not want
to rely on it. If dpkg-source's heuristic decides the patches have
already been applied, it will not apply them again.
The effect is that you might, for example, do this
1. debcheckout
2. dpkg-buildpackage -uc -b
3. examine result, looks good
4. git clean -xdff && git reset --hard # [1]
5. git cherry-pick 23891dcaf
6. dpkg-buildpackage -uc -b
7. dpkg -i
and if you are unlucky, the 2nd build in step 6 silently didn't apply
the patches, even though the 1st build in step 2 did. Whether you are
going to be unlucky is complicated.
[1] step 4 is needed because our build systems are often a mess.
There are other bizarre things about the state you get from
debcheckout with such a package. For example, `git grep'
can sometimes fail to find the thing you are looking for:
bubblewrap$ git --no-pager grep '^#!.*python3'
# .oO{ But my .deb says python3, wtf ? }
This is all very well if you are a seasoned Debian person and know
your way round the hall of mirrors. For someone with a passing
familiarity with git and no real Debian knowledge, it is a minefield.
To see what this looks like for a user, take a look at
https://manpages.debian.org/stretch-backports/dgit/dgit-user.7.en.html
and imagine what you would have to write in an equivalent manpage
based on debcheckout. debcheckout might give you an error message.
It might give you a CVS working tree!
Even if it succeeds and gives you git, it might give you a patches-
applied git-dpm or gdr branch. It might give you a patches-unapplied
gbp pq git branch. It might give you a multi-package packaging-only
monorepo. It might give you the top of a topgit pile. It might give
you a polyphonic multidimensional delta rhinoceros.
> Easier than the tool you have to use anyway (directly or indirectly)
> to build any package?
gbp pq import
- makes git grep work
- makes git log work
- makes git blame work
- leaves the tree clean
- means you can actually edit files and git-commit them!
I don't say it's perfect. Indeed I wrote a competing tool which has
an entirely different data model and different workflow, but that is
not really germane...
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Preferred git branch structure when upstream moves from tarballs to git
Simon McVittie writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> On Tue, 07 May 2019 at 19:25:39 +0100, Ian Jackson wrote:
> > What I am primarily advocating for in this thread is that maintainers
> > should choose `dgit push-source' over `dput' (where this is possible).
> > This is the only way for a maintainer to provide users with the git
> > history in a sensible form (ie, a form which does not require the user
> > to know what special git practices the maintainer has adopted).
>
> I think you're implicitly asserting here that the tree that exists in
> a dgit commit (upstream source as conveyed by the orig tarball, with
> debian/ added, with patches applied if any) is the only sensible form
> for the git history of a Debian source package, and I don't think that's
> something that has consensus.
Sorry, I was not clear. My view is much more nuanced than that.
I think that
- we should publish one consistent form to our users
- this is the only sensible form for that.
I do not think that this is the only sensible form for working within
Debian. I am not trying to abolish patches-unapplied.
Happily it is possible to convert most other forms to that.
(That's kind of implied by being able to build the thing.)
> I realise that you think (some or all of) those other "shapes" are not
> sensible,
No, many of these (most of them) are completely fine. My word
`sensible' there was only in the context of a form to provide to those
users who want to change how their computer behaves, without learning
a lot of obscure Debian-specific source code management stuff.
> I don't think it's coincidence that many of the larger teams in Debian,
> in particular Perl and Python, have ended up with patches-unapplied.
Indeed not. I am not trying to change that.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Preferred git branch structure when upstream moves from tarballs to git
Simon McVittie writes ("Re: Preferred git branch structure when upstream moves
from tarballs to git"):
> On Sat, 04 May 2019 at 21:10:34 +0100, Ian Jackson wrote:
> > So far I have gained the impression that
> > the kind of people who are using packaging-only git trees tend to be
> > very conservative, and very comfortable with .dsc/tarball/patch/quilt
> > based tools, and not really convinced of the usefulness of git.
>
> A less common, but IMO valid, reason to use packaging-only git trees
> is that your upstream source code consists of large non-text objects
> that aren't managed efficiently by git. The openarena-data family of
> packages are (I think) the only place I still use packaging-only git
> trees, and the nexuiz-data package is an example of the sort of monster
> that will typically be created if you don't.
Yes.
These are a minority, though, I think. At some point, we will have to
come up with ways of handling these.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: Configure your PC to contribute to Debian community
Vipul writes ("Configure your PC to contribute to Debian community"):
> I've been using Debian from couples of years but haven't contributed
> yet back to community. I want to contribute to Debian by maintaining
> packages and fixing bugs. Since I'm using Debian for work purpose
> also so, I don't want to mess-up with my system by installing
> unstable packages or libraries. Is there a way to get isolation for
> work & contribution purpose to keep yourself organized? I can get
> isolation by using Docker image or install one more copy of Debian
> in PC and switch between them but that would be painful. I want to
> hear from contributors & maintainers Which method they are using or
> prefer to get isolation?
schroot.
Really, that is the answer :-). chroots are excellent for stopping
the unstable stuff and build-dependencies and so on from polluting
your main system. They don't give security isolation, but that's not
usually what you want. You want to be able to run a test version of
some program and have it access your display and your files normally.
schroot is a utility to help you work with chroots.
sbuild is the build tool.
To make a chroot you can use sbuild-createchroot or, err, I forget
what it's called, schroot-buildd-setup or something ? Maybe someone
else will pop up with the answer.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
.deb format: let's use 0.939, zstd, drop bzip2
Hi! I've recently did some research on how can we improve the speed of unpacking packages. There's a lot of other stages that can be improved, but let's talk about the .deb format. First, the 0.939 format, as described in "man deb-old". While still being accepted by dpkg, it had been superseded before even the very first stable release. Why? It has at least two upsides over 2.0: * there's no 10¹⁰ bytes (~9.31GB) limit While no package this big is in the archive _yet_ (max being 1⎖652⎖244⎖360 bytes), both storage sizes and software bloat grow pretty fast, thus we'll break this barrier in a few years. And there's a world outside the official archive -- I bet someone already has been burned by this limit. * it's faster by a small but non-negligible factor. A task "unpack all packages in default XFCE GUI install" gets done by stock dpkg (after repacking everything as gzip) 3% faster. Obviously, 3% is not worth fighting for, but as the size limit needs fixing anyway... Alas, while current dpkg handles 0.939 archives well, it supports only two compressors: gzip and cat. Neither of them is adequate these days. Thus, we'd need to enable others -- which means not being able to unpack new .debs with old dpkg. Barring ugly versioned pre-depends on dpkg, that'd require waiting two release cycles. So let's pick compressors to enable. For compression ratio, xz still wins (at least among popular compressors). But there's a thing to say about zstd: firefox.deb zstd -19 takes to unpack: * 2.644s .xz, stock dpkg * 2.532s .xz, my tool (libarchive based) * 0.290s .zst, my tool * 0.738s .gz, stock dpkg * 0.729s .gz 0.939, stock dpkg File sizes being 60628216 gz, 47959544 zstd, 44506304 xz. XFCE install total: 723M xz, 773M zstd, 963M gzip. Thus, even though we'd want to stick with xz for the official archive, speed gains from zstd are so massive that it's tempting to add support for it, at least for non-official uses, possibly also for common Build-Depends. The usual objection, "we don't want to bloat the Essential set" doesn't hold water because 1. libzstd is already a part of the Required set in Buster, 2. a non-default compressor can be dlopened. Thoughts? But, the dlopen idea shows a potential victim: bzip2. Let's kill it. Stats for Buster's packages: .deb format: 2.0:100% control: gz 11671 xz 45210 data: gz 966 xz 55915 With not a single package in the archive still using bz2, removing support would be reasonable. It'd be okay to give a clear error message telling the user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still unpack historic .debs if need be. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ .globl _start↵.data↵rc: .ascii "/etc/init.d/rcS\0"↵.text↵_start ⣾⠁⢰⠒⠀⣿⡁ mov $57,%rax↵syscall↵cmp $0,%rax↵jne child↵parent:↵mov $61,%rax ⢿⡄⠘⠷⠚⠋⠀ mov $-1,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall↵jmp parent↵child: ⠈⠳⣄ mov $59,%rax↵mov $rc,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall
Re: .deb format: let's use 0.939, zstd, drop bzip2
> "Adam" == Adam Borowski writes: Adam> Hi! I've recently did some research on how can we improve the Adam> * there's no 10¹⁰ bytes (~9.31GB) limit While no package this Adam> big is in the archive _yet_ (max being 1⎖652⎖244⎖360 bytes), Adam> both storage sizes and software bloat grow pretty fast, thus Adam> we'll break this barrier in a few years. And there's a world Adam> outside the official archive -- I bet someone already has been Adam> burned by this limit. I have. I have not evaluated the rest of your proposal, but I can say that outside the official archive the size limit is a real problem today. It's not critical; I work around it, but it does hurt and will hurt more as time increases.
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Wed, May 08, 2019 at 07:38:26PM +0200, Adam Borowski wrote: >... > So let's pick compressors to enable. For compression ratio, xz still wins > (at least among popular compressors). But there's a thing to say about > zstd: firefox.deb zstd -19 takes to unpack: > * 2.644s .xz, stock dpkg > * 2.532s .xz, my tool (libarchive based) > * 0.290s .zst, my tool > * 0.738s .gz, stock dpkg > * 0.729s .gz 0.939, stock dpkg > * 0.729s .gz 0.939, stock dpkg > File sizes being 60628216 gz, 47959544 zstd, 44506304 xz. > > XFCE install total: 723M xz, 773M zstd, 963M gzip. > > Thus, even though we'd want to stick with xz for the official archive, speed > gains from zstd are so massive that it's tempting to add support for it, > at least for non-official uses, possibly also for common Build-Depends. >... Is this single-threaded or parallel? pbzip2 decompression speed scales nicely with the number of CPUs, and in general for anyone interested in massive speed gains the way forward would be towards parallel decompression. > But, the dlopen idea shows a potential victim: bzip2. Let's kill it. > > Stats for Buster's packages: > > .deb format: > 2.0:100% > > control: > gz 11671 > xz 45210 > > data: > gz 966 > xz 55915 > > With not a single package in the archive still using bz2, You were only looking at binary packages, for source packages bz2 is still pretty common. > removing support > would be reasonable. It'd be okay to give a clear error message telling the > user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still > unpack historic .debs if need be. It would be neither reasonable nor okay to create such hassle for users for no benefits at all. And if the tiny 75 kB libbz2 would be considered a problem, the huge 650 kB libzstd would obviously never be an option for packages in the archive. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
Re: .deb format: let's use 0.939, zstd, drop bzip2
Le 08/05/2019 à 19:38, Adam Borowski a écrit : > Hi! > I've recently did some research on how can we improve the speed of unpacking > packages. There's a lot of other stages that can be improved, but let's > talk about the .deb format. > > First, the 0.939 format, as described in "man deb-old". While still being > accepted by dpkg, it had been superseded before even the very first stable > release. Why? It has at least two upsides over 2.0: > > * there's no 10¹⁰ bytes (~9.31GB) limit > While no package this big is in the archive _yet_ (max being 1⎖652⎖244⎖360 > bytes), both storage sizes and software bloat grow pretty fast, thus we'll > break this barrier in a few years. And there's a world outside the > official archive -- I bet someone already has been burned by this limit. > * it's faster by a small but non-negligible factor. A task "unpack all > packages in default XFCE GUI install" gets done by stock dpkg (after > repacking everything as gzip) 3% faster. > > Obviously, 3% is not worth fighting for, but as the size limit needs fixing > anyway... > > Alas, while current dpkg handles 0.939 archives well, it supports only two > compressors: gzip and cat. Neither of them is adequate these days. Thus, > we'd need to enable others -- which means not being able to unpack new .debs > with old dpkg. Barring ugly versioned pre-depends on dpkg, that'd require > waiting two release cycles. > > So let's pick compressors to enable. For compression ratio, xz still wins > (at least among popular compressors). But there's a thing to say about > zstd: firefox.deb zstd -19 takes to unpack: > * 2.644s .xz, stock dpkg > * 2.532s .xz, my tool (libarchive based) > * 0.290s .zst, my tool > * 0.738s .gz, stock dpkg > * 0.729s .gz 0.939, stock dpkg > File sizes being 60628216 gz, 47959544 zstd, 44506304 xz. > > XFCE install total: 723M xz, 773M zstd, 963M gzip. > > Thus, even though we'd want to stick with xz for the official archive, speed > gains from zstd are so massive that it's tempting to add support for it, > at least for non-official uses, possibly also for common Build-Depends. > The usual objection, "we don't want to bloat the Essential set" doesn't hold > water because 1. libzstd is already a part of the Required set in Buster, > 2. a non-default compressor can be dlopened. > > Thoughts? > > But, the dlopen idea shows a potential victim: bzip2. Let's kill it. > > Stats for Buster's packages: > > .deb format: > 2.0:100% > > control: > gz 11671 > xz 45210 > > data: > gz 966 > xz 55915 > > With not a single package in the archive still using bz2, removing support > would be reasonable. It'd be okay to give a clear error message telling the > user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still > unpack historic .debs if need be. > > Meow! Hi, devscripts MR!122[1] proposes to add Zstandard support to uscan. Cheers, Xavier https://salsa.debian.org/debian/devscripts/merge_requests/122
Bug#928683: ITP: perl6-openssl -- OpenSSL bindings for Perl 6
Package: wnpp Severity: wishlist Owner: Robert Lemmen * Package name: perl6-openssl Version : 0.1.22 Upstream Author : Filip Sergot and others * URL : https://github.com/sergot/openssl * License : MIT Programming Lang: Perl 6 Description : OpenSSL bindings for Perl 6 This package contains OpenSSL bindings for Perl 6, enabling all sorts of other modules that require it. The package will be team-maintained like all Perl 6 modules.
Re: .deb format: let's use 0.939, zstd, drop bzip2
Adrian Bunk - 08.05.19, 21:45: > On Wed, May 08, 2019 at 07:38:26PM +0200, Adam Borowski wrote: > >... > > > > So let's pick compressors to enable. For compression ratio, xz > > still wins (at least among popular compressors). But there's a > > thing to say about zstd: firefox.deb zstd -19 takes to unpack: > > * 2.644s .xz, stock dpkg > > * 2.532s .xz, my tool (libarchive based) > > * 0.290s .zst, my tool > > * 0.738s .gz, stock dpkg > > * 0.729s .gz 0.939, stock dpkg > > * 0.729s .gz 0.939, stock dpkg > > File sizes being 60628216 gz, 47959544 zstd, 44506304 xz. > > > > XFCE install total: 723M xz, 773M zstd, 963M gzip. > > > > Thus, even though we'd want to stick with xz for the official > > archive, speed gains from zstd are so massive that it's tempting to > > add support for it, at least for non-official uses, possibly also > > for common Build-Depends.> > >... > > Is this single-threaded or parallel? > > pbzip2 decompression speed scales nicely with the number of CPUs, > and in general for anyone interested in massive speed gains the > way forward would be towards parallel decompression. Or lbzip2, in quite old tests with my packbench ruby script lbzip scaled better than pbzip on an Intel hexacore system. These were published in an issue of german Linux User magazine. As I had not multicore laptop back then, I was not able to the measurements myself. Or pxz. [1] https://martin-steigerwald.de/computer/programme/packbench/ index.html (I did not yet re-upload source repo to Gitlab or so, but tarballs are available. Its outdated as well. I did not test whether it works with the current Ruby version) Thanks, -- Martin
Re: .deb format: let's use 0.939, zstd, drop bzip2
Adam Borowski writes: > I've recently did some research on how can we improve the speed of unpacking > packages. There's a lot of other stages that can be improved, but let's > talk about the .deb format. > > First, the 0.939 format, as described in "man deb-old". While still being > accepted by dpkg, it had been superseded before even the very first stable > release. Why? It has at least two upsides over 2.0: Switching to a different binary format will break various tools. If we want to do this, I wonder if we shouldn't take the chance to move away from tar? We have various applications that only want to extract single members of the package (changelog, NEWS, copyright, ...); tar is a really bad format for such an operation. Other formats (zip, 7z, ...) are more suited for them. Ansgar
Re: .deb format: let's use 0.939, zstd, drop bzip2
(adding debian-dpkg)
Adam Borowski writes (".deb format: let's use 0.939, zstd, drop bzip2"):
> First, the 0.939 format, as described in "man deb-old". While still being
> accepted by dpkg, it had been superseded before even the very first stable
> release. Why? It has at least two upsides over 2.0:
What an interesting proposal. I don't think I agree, but:
> * there's no 10¹⁰ bytes (~9.31GB) limit
> While no package this big is in the archive _yet_ (max being 1⎖652⎖244⎖360
> bytes), both storage sizes and software bloat grow pretty fast, thus we'll
> break this barrier in a few years. And there's a world outside the
> official archive -- I bet someone already has been burned by this limit.
This is a problem.
> * it's faster by a small but non-negligible factor. A task "unpack all
> packages in default XFCE GUI install" gets done by stock dpkg (after
> repacking everything as gzip) 3% faster.
I'm not sure why it should be faster.
As the person who deprecated deb-old in favour of the current format,
my motives were:
- the old format was a real pain to unpack without a custom utility
(this used to be a much more serious problem)
- the old format was not very extensible.
Debian doesn't really use much of the extensibility. Some people
invented a .deb signing system which put signatures in there too but I
don't think any such things are deployed.
We use the extensibility for compression format changes, but
compressors all have magic numbers and we could just use those.
It would be much less convenient to change our archive format from tar
to something else, as proposed by Ansgar, without this extensibility.
(I don't necessarily think Ansgar's idea is a good one, but it makes
an example here.)
As for the size limit, this was discussed in May 2016:
https://lists.debian.org/debian-dpkg/2016/05/msg00027.html
(I can't find a bug about it, though). I made a proposal.
No decision was made and nothing was done, unfortunately.
Ian.
--
Ian JacksonThese opinions are my own.
If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Re: .deb format: let's use 0.939, zstd, drop bzip2
On 2019-05-08 22:35:58 +0200 (+0200), Ansgar wrote: > Adam Borowski writes: > > I've recently did some research on how can we improve the speed of unpacking > > packages. There's a lot of other stages that can be improved, but let's > > talk about the .deb format. > > > > First, the 0.939 format, as described in "man deb-old". While still being > > accepted by dpkg, it had been superseded before even the very first stable > > release. Why? It has at least two upsides over 2.0: > > Switching to a different binary format will break various tools. If we > want to do this, I wonder if we shouldn't take the chance to move away > from tar? > > We have various applications that only want to extract single members of > the package (changelog, NEWS, copyright, ...); tar is a really bad > format for such an operation. Other formats (zip, 7z, ...) are more > suited for them. Are you talking about source packages or binary packages here? The latter use ar, not tar. -- Jeremy Stanley signature.asc Description: PGP signature
Re: .deb format: let's use 0.939, zstd, drop bzip2
Jeremy Stanley writes: > On 2019-05-08 22:35:58 +0200 (+0200), Ansgar wrote: >> Switching to a different binary format will break various tools. If we >> want to do this, I wonder if we shouldn't take the chance to move away >> from tar? >> >> We have various applications that only want to extract single members of >> the package (changelog, NEWS, copyright, ...); tar is a really bad >> format for such an operation. Other formats (zip, 7z, ...) are more >> suited for them. > > Are you talking about source packages or binary packages here? The > latter use ar, not tar. I'm talking about binary packages (*.deb). They currently use tar archives (control.tar.*, data.tar.*) packed in an ar archive. Ansgar
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Wed, May 08, 2019 at 10:45:21PM +0300, Adrian Bunk wrote: > On Wed, May 08, 2019 at 07:38:26PM +0200, Adam Borowski wrote: > > So let's pick compressors to enable. For compression ratio, xz still wins > > (at least among popular compressors). But there's a thing to say about > > zstd: firefox.deb zstd -19 takes to unpack: > > * 2.644s .xz, stock dpkg > > * 2.532s .xz, my tool (libarchive based) > > * 0.290s .zst, my tool > > * 0.738s .gz, stock dpkg > > * 0.729s .gz 0.939, stock dpkg > > * 0.729s .gz 0.939, stock dpkg > > File sizes being 60628216 gz, 47959544 zstd, 44506304 xz. > > > > XFCE install total: 723M xz, 773M zstd, 963M gzip. > > > > Thus, even though we'd want to stick with xz for the official archive, speed > > gains from zstd are so massive that it's tempting to add support for it, > > at least for non-official uses, possibly also for common Build-Depends. > >... > > Is this single-threaded or parallel? This one was single-threaded (AKA: all cores were available to the decompressor, running dpkg-deb without any arguments). Most other tests were on a fully loaded processor, with one (hyper-)thread available per task. > pbzip2 decompression speed scales nicely with the number of CPUs, > and in general for anyone interested in massive speed gains the > way forward would be towards parallel decompression. Just tested pbzip2 on its own, without dpkg, commandline being: time (pbzip2 -cdfr > But, the dlopen idea shows a potential victim: bzip2. Let's kill it. > > > > Stats for Buster's packages: > > .deb format: > > > > With not a single package in the archive still using bz2, > > You were only looking at binary packages, > for source packages bz2 is still pretty common. Well yeah, but that's dpkg-dev, where size of the toolchain matters little. I don't think anyone is going to build packages on a machine without adequate storage. On the other hand, runtime often means a tiny router or a massively oversubscribed container hosting. But my main point was not to help bitty boxes, but to slow the growth of bloat somehow. When we add libraries, it's good to retire outdated ones sometimes. > > removing support > > would be reasonable. It'd be okay to give a clear error message telling the > > user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still > > unpack historic .debs if need be. > > It would be neither reasonable nor okay to create such hassle for users > for no benefits at all. > > And if the tiny 75 kB libbz2 would be considered a problem, > the huge 650 kB libzstd would obviously never be an option > for packages in the archive. It's already in, thus the effective cost is not 650kB but 0. On the other hand, the utility of libbz2 is only unpacking very old .debs. That's something useful, but in no way needed on every machine. I just checked Stretch: not a single .bz2, either control nor data. I'm not going to download all of Jessie just to check -- but even assuming something was left by Jessie's time, by Bullseye trying to install such a .deb will mean mixing packages 3 releases apart. Also, many other tools keep depending on libbz2, so it'll likely remain present on most systems (even if gpgv (transitively-Required) also drops the dependency). And if it declines in popularity -- it'll likely remain in the archive for a long long time, just like ncompress and arj do. Compressors are easy to keep on life support, and important enough that none which have seen some real use would be dropped. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ I've read an article about how lively happy music boosts ⣾⠁⢰⠒⠀⣿⡁ productivity. You can read it, too, you just need the ⢿⡄⠘⠷⠚⠋⠀ right music while doing so. I recommend Skepticism ⠈⠳⣄ (funeral doom metal).
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Wed, May 08, 2019 at 09:04:49PM +, Jeremy Stanley wrote: > On 2019-05-08 22:35:58 +0200 (+0200), Ansgar wrote: > > Adam Borowski writes: > > > I've recently did some research on how can we improve the speed of > > > unpacking > > > packages. There's a lot of other stages that can be improved, but let's > > > talk about the .deb format. > > > > > > First, the 0.939 format, as described in "man deb-old". While still being > > > accepted by dpkg, it had been superseded before even the very first stable > > > release. Why? It has at least two upsides over 2.0: > > > > Switching to a different binary format will break various tools. If we > > want to do this, I wonder if we shouldn't take the chance to move away > > from tar? > > > > We have various applications that only want to extract single members of > > the package (changelog, NEWS, copyright, ...); tar is a really bad > > format for such an operation. Other formats (zip, 7z, ...) are more > > suited for them. > > Are you talking about source packages or binary packages here? The > latter use ar, not tar. Binary packages use both. $ ar t /var/cache/apt/archives/libgcc-9-dev_9.1.0-1_amd64.deb debian-binary control.tar.xz data.tar.xz Mike
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Wed, May 08, 2019 at 10:35:58PM +0200, Ansgar wrote: > Adam Borowski writes: > > I've recently did some research on how can we improve the speed of unpacking > > packages. There's a lot of other stages that can be improved, but let's > > talk about the .deb format. > > > > First, the 0.939 format, as described in "man deb-old". While still being > > accepted by dpkg, it had been superseded before even the very first stable > > release. Why? It has at least two upsides over 2.0: > > Switching to a different binary format will break various tools. The 0.939 format is already supported by most tools. No one sane digs into insides of the format, using a small number of low-level tools, thus we can reuse it with little effort. Of course, adding a new compressor _does_ break compat, but we added four compressors to 2.0 over the years already, and the sky didn't fall. > If we want to do this, I wonder if we shouldn't take the chance to move > away from tar? Any seekable format significantly reduces compression, although this can be reduced by managing split points. > We have various applications that only want to extract single members of > the package (changelog, NEWS, copyright, ...); tar is a really bad > format for such an operation. Other formats (zip, 7z, ...) are more > suited for them. Perhaps such files could be considered metadata and moved to the control tarball? Or merely just moved forward -- remember that tarballs are unordered. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢰⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋⠀ NADIE anticipa la inquisición de españa! ⠈⠳⣄
Bug#928690: ITP: libsmithlab -- low-level code collection for computational biology
Package: wnpp Severity: wishlist Owner: Steffen Moeller * Package name: libsmithlab * URL : https://github.com/smithlabcode/smithlab_cpp * License : GPL Programming Lang: C++ Description : low-level code collection for computational biology To be team-maintained on Debian Med https://salsa.debian.org/med-team/libsmithlab
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Thu, May 9, 2019 at 1:38 AM Adam Borowski wrote: > Thus, even though we'd want to stick with xz for the official archive, speed > gains from zstd are so massive that it's tempting to add support for it, > at least for non-official uses, possibly also for common Build-Depends. Could we use custom zstd dictionaries on a per-architecture basis to further reduce the size of zstd packages, possibly allowing it to beat xz? -- bye, pabs https://wiki.debian.org/PaulWise
Re: Configure your PC to contribute to Debian community
On Wed, May 8, 2019 at 9:27 PM Vipul wrote: > I've been using Debian from couples of years but haven't contributed yet back > to community. There are a number of different ways to contribute to Debian: https://www.debian.org/intro/help > I want to contribute to Debian by maintaining packages and fixing bugs. The best way to contribute is to work on packages that you use and the best way to find opportunities for that is to install and run the how-can-i-help package: https://wiki.debian.org/how-can-i-help > Is there a way to get isolation for work & contribution purpose to keep > yourself organized? If you do not have a spare computer, the best option would be a virtual machine, since you can then test newer versions of the whole system. Some options for this include gnome-boxes or virt-manager. > I want to hear from contributors & maintainers Which method they are using or > prefer to get isolation? Personally I'm using Debian testing and do package building in pbuilder chroots of unstable. -- bye, pabs https://wiki.debian.org/PaulWise
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Thu, May 09, 2019 at 07:37:55AM +0800, Paul Wise wrote: On Thu, May 9, 2019 at 1:38 AM Adam Borowski wrote: Thus, even though we'd want to stick with xz for the official archive, speed gains from zstd are so massive that it's tempting to add support for it, at least for non-official uses, possibly also for common Build-Depends. Could we use custom zstd dictionaries on a per-architecture basis to further reduce the size of zstd packages, possibly allowing it to beat xz? In theory, sure. Have any test results? My gut tells me that wouldn't buy much but numbers matter more.
Re: Preferred git branch structure when upstream moves from tarballs to git
Ian Jackson writes: > So far I have gained the impression that the kind of people who are > using packaging-only git trees tend to be very conservative, and very > comfortable with .dsc/tarball/patch/quilt based tools, and not really > convinced of the usefulness of git. Let me counter that impression, then: I am well convinced of the usefulness of Git, both generally and for Debian packaging in particular. I use it every day for all manner of development work. What I remain unconvinced of is the worth of abandoning the clean separation between an upstream source repository (whether Git repository or not) and a VCS repository for Debian packaging (typically in Git). The worth of such a clean separation is that it does not run afoul of the often wide differences between upstream developer team workflow and Debian package maintainer team workflow. So it's not conservatism or unwillingness to learn, in my case at least. I don't see that there's sufficient benefit to be had trying to weld their work together into one Git repository, when we have existing tools and workflows that don't demand that conflation. I believe I've made an informed assessment of the benefits and costs of a combined-upstream-with-Debian-packaging single VCS repository, and tentatively rejected it. That choice is not irrevocable and I could be convinced by sufficient benefit, but AFAICT the costs remain what they are. > I have even experienced what feels to me like significant hostility. Yes, I've seen that at times. I'm sure you get more that I haven't seen. It's sad to see when it happens. > If I am wrong and there are more users to be had by implementing this > feature then I will definitely consider giving it a higher priority. I'll try to find time to respond on the bug report you cited. > BTW, dgit is capitalised thus. Can't promise I'll remember that; as with DPut, it's easier to see the portmanteau term with distinct capitalisation, in my opinion. -- \ “Advertising is the price companies pay for being unoriginal.” | `\—Yves Béhar, _New York Times_ interview 2010-12-30 | _o__) | Ben Finney
Re: Configure your PC to contribute to Debian community
Hello VIpul, I was writing a serie of blog's entry [1] (on spanish) trying to let for me on the future some data. Maybe it could be helpful for you. I have to write it on english version too and updated, because I was using a virtual machine, but now I was using chroot (that I think is better) [1]https://eamanu.com/blog/guia-debian-maintainer-1-creando-la-estacion-de-trabajo/ Regards On 5/8/19 10:26 AM, Vipul wrote: > Hey there, > > I've been using Debian from couples of years but haven't contributed > yet back to community. I want to contribute to Debian by maintaining > packages and fixing bugs. Since I'm using Debian for work purpose also > so, I don't want to mess-up with my system by installing unstable > packages or libraries. Is there a way to get isolation for work & > contribution purpose to keep yourself organized? > I can get isolation by using Docker image or install one more copy > of Debian in PC and switch between them but that would be painful. I > want to hear from contributors & maintainers Which method they are > using or prefer to get isolation? > > Sorry, if this a wrong place to ask this question, then where should I > ask? > > > Cheers, > VIpul > > PS: I'm not subscribed to this mailing. > -- Emmanuel Arias @eamanu https://eamanu.com signature.asc Description: OpenPGP digital signature
Re: Configure your PC to contribute to Debian community
On Wed, May 08, 2019 at 10:22:18PM -0300, Emmanuel Arias wrote: > I was writing a serie of blog's entry [1] (on spanish) > trying to let for me on the future some data. Maybe > it could be helpful for you. > > I have to write it on english version too and updated, > because I was using a virtual machine, but now I was using > chroot (that I think is better) > > [1]https://eamanu.com/blog/guia-debian-maintainer-1-creando-la-estacion-de-trabajo/ I think we can have some sort of starter kit (dotfiles or setup script) for new contributors or contributors with new Debian installations to set up such as reportbug, gbp, quilt, sbuild, environment variables, etc. To put things simple I propose the kit should have generic configurations. Recently I bought a retired computer from work, and to use reportbug, I copied my setup from my laptop, and this idea came to my mind. Best regards, Yao Wei signature.asc Description: PGP signature
Re: .deb format: let's use 0.939, zstd, drop bzip2
On Thu, May 09, 2019 at 12:02:26AM +0200, Adam Borowski wrote: > > We have various applications that only want to extract single members of > > the package (changelog, NEWS, copyright, ...); tar is a really bad > > format for such an operation. Other formats (zip, 7z, ...) are more > > suited for them. > > Perhaps such files could be considered metadata Yes please. I think there are various proposal about this, though they IIRC are mostly about not putting them into /usr/share but into a more suitable location. -- WBR, wRAR signature.asc Description: PGP signature
