Hi Alec, Really great reading about packaging and the current challenges. This would be great to be posted in Gentoo blog!
--- To all community: I would like to commend the know how of Gentoo community and their principles. I would like to evidence relatively to other distributions some points that IMHO turns Gentoo the greatest (win in all points): - QA - freedom - tooling - know how - velocity - security - slotting - learning - sharing - straightforwardness - objectivity - mirroring - package system - **extensibility** Overlays, ebuild and profile definitions in Gentoo extends the freedom of the ingenious distro ever made. There are no barriers, so please don't forget the overlays when developing Gentoo code base that would help everyone to extend their needs and follow the good QA practices. There is nothing like Gentoo! Thank you all that contribute to Gentoo! On 4/1/20 4:49 PM, Alec Warner wrote: > > On Wed, Apr 1, 2020 at 5:14 AM Samuel Bernardo > <samuelbernardo.m...@gmail.com <mailto:samuelbernardo.m...@gmail.com>> > wrote: > > Forgive my noobishness in this matter that let Alec to comment > over my own statement. > > Alec pointed out some very important issues in go development that > break copyright infringement and security vulnerabilities, but I'm > sure that is not related to the good work done in go-module.eclass > to surpass all go mess. npm is worst and I take from go-module as > a good pattern to apply also into there. > > I am antarus, not mjo (but more on that below!) I don't believe > bundling presents many challenges with regards to copyright > infringement. As a package maintainer you should know the licenses > used in your packages. You are required to reflect any licenses used > in the LICENSE ebuild variable. Obviously this becomes more work if > you are using a bundle due to the fact that bundling will include more > code. In the golang ecosystem there is a tool to help maintainers do > this (https://packages.gentoo.org/packages/dev-go/golicense). I get > that with bundling we cannot share the work from previous packages > because packages are not shared in a bundled environment but I expect > the golicense tool to have good coverage in practice. If the tool does > the work, sharing the work becomes moot. > > I think licensing can be more challenging in other bundling scenarios > where tooling is not provided; but note that this is not significantly > different from the unbundled scenario in terms of license discovery. > If I am packaging a new program (A) and it depends on (B,C,D) I have > two options. I can either package [A,B,C,D] (normal gentoo way) or I > can package [A] (with B,C,D bundled). The intersection of the LICENSE > variables is the same effort for both here. The benefit of the > multiple packages is that future users of B,C,D can re-use the license > discovery work and that isn't nothing. > > Going back to my overlay use case, will go-modules download all > modules to distfiles directory? The naming convention will assure > that there will be no modules repetition? > > What about eclean-dist, will it work as expected for those modules > dependencies? > > I think some of this answers would worth mention in documentation. > > Sorry for anything I wrongly stated and thank you very much for > your help, > > Samuel > > I've chosen this part to write my treatise on packaging, but rest > assured it's mostly intended as a response to mgorny and mjo; not > specifically in response to you. > > The very long answer is that Gentoo was designed around a paradigm of > programs written primarily in C. In C programs you have the ability to > link to libraries which offer APIs and in the ideal case, each API is > offered via a unique SONAME[0]. Upstream packages were written and > built in this way (with dynamic linking). So in the case of package A, > that uses libraries B, C, and D; the result in many distributions is 4 > packages (A,B,C,D) and users who want A will get B, C, and D > installed. This in fact was a major selling point of package managers > at the time because finding these dependencies by hand and building > and merging them all was painful. > > Many applications break this trend; I don't think golang or nodejs are > particularly new (python and ruby have had (pip, venv) and rubygems[1] > for years, for example, which are similar bundling paradigms.) The > struggle as packagers and distribution managers is when upstream > decides "my software should be installed via a bundling solution > (golang, node, pip, rubygems, and so on)" we are left to decide both > whether to map this to the ebuild paradigm (no bundling of > dependencies) or omit ebuilds entirely. In the former case we are > often left working at odds with upstream (who are confused by our > decomposition of their application) and in the latter case, users > often use the bundle anyway (e.g. they install the packages by hand or > use the ruby gems or whatever.) I assert this is somewhat of a false > choice. Bundling isn't all bad and we can learn from past mistakes[2] > to try to avoid problems. > > Another challenge with bundling is that often bundling systems > (bundler, pip, venv, golang, etc.) specify specific versions, commits, > or tags. This is fine when bundling (because each bundle has its own > version of a dependency in the bundle) but when you are trying to > share a system wide package between N packages, you either need to > SLOT the dependencies or have a looser dependency specification. The > fine-grained nature of the upstream dependency specification can make > this challenging[3]. > > Unbundling then made it easier for system operators to operate a > system; and you see this often in the security space. A security > notice will come out saying "foo-X-Y-Z is vulnerable, move to > foo-Y.1." So operators want to know "do I have foo-x-y-z installed?" > When every package is in the package manager this is a trivial > question. When software is bundled inside of a package, this > visibility is lost. I haven't seen any tooling for Gentoo to this problem. > > In addition to the above, bundling can present exciting resource > challenges for some deployments. Imagine a common dep > (CommonFoo-x-y-z) has a security problem, so we must upgrade to > CommonFoo-y-z. In the scenario where CommonFoo is a dynamically linked > package we can recompile it once[4] and new consumers will just use > the new dynamic shared object. In a bundling scenario, we will be > forced to rebuild[5] all consumers. This can take a lot of time and > resources depending on the deployment. Is the deployment using a build > farm? A binary packages host? How many disparate platforms are in use? > > Which is to say many people in Gentoo dislike bundling for various > reasons; many of them legitimate. I wish to present a narrative where > bundling is an engineering trade-off, rather than a decision that is > settled engineering law. This doesn't mean Gentoo needs to support all > the bundling (clearly most people don't want to) but not supporting it > means that many packages will not be in Gentoo at all (because > unbundling is too costly) and so you end up at this exciting > discussion which happens every couple of years. > > -A > > [0] I understand this is not always true in practice, but let's assume > spherical cows momentarily. > [1] Gentoo has a rubygems-fakegem eclass that makes it pretty > streamlined to make an ebuild for a particular gem, but of course if > my application depends on 20 gems I still need to make 20 ebuilds in > this scheme and merge them all. Rubygems-fakegem is still pretty good > though! > [2] On windows https://en.wikipedia.org/wiki/DLL_Hell was common. In > the .NET ecosystems assemblies addressed some of these problems. > [3] Similar to DLL hell but more > generically: https://en.wikipedia.org/wiki/Dependency_hell > [4] Practice of course, leads to all kinds of weird edge cases where > upgrading your shared lib causes dependencies to break for various > reasons; which is one reason why application authors like to bundle; > because their application ends up being perceived as more reliable and > less finicky. > [5] The number of package rebuilds in Gentoo is a fairly common > complaint, from my personal observation. Obviously binary packages > make this problem worse (not better.) I dunno if its something the > community should put more effort into or not though; my expectation is > that rebuilds are common and making them more common is not a > strategic problem; but I'm also not compiling on some single core > atom, so what do I know, eh? :)
signature.asc
Description: OpenPGP digital signature
