Le 08/05/2019 à 19:38, Adam Borowski a écrit : > Hi! > I've recently did some research on how can we improve the speed of unpacking > packages. There's a lot of other stages that can be improved, but let's > talk about the .deb format. > > First, the 0.939 format, as described in "man deb-old". While still being > accepted by dpkg, it had been superseded before even the very first stable > release. Why? It has at least two upsides over 2.0: > > * there's no 10¹⁰ bytes (~9.31GB) limit > While no package this big is in the archive _yet_ (max being 1⎖652⎖244⎖360 > bytes), both storage sizes and software bloat grow pretty fast, thus we'll > break this barrier in a few years. And there's a world outside the > official archive -- I bet someone already has been burned by this limit. > * it's faster by a small but non-negligible factor. A task "unpack all > packages in default XFCE GUI install" gets done by stock dpkg (after > repacking everything as gzip) 3% faster. > > Obviously, 3% is not worth fighting for, but as the size limit needs fixing > anyway... > > Alas, while current dpkg handles 0.939 archives well, it supports only two > compressors: gzip and cat. Neither of them is adequate these days. Thus, > we'd need to enable others -- which means not being able to unpack new .debs > with old dpkg. Barring ugly versioned pre-depends on dpkg, that'd require > waiting two release cycles. > > So let's pick compressors to enable. For compression ratio, xz still wins > (at least among popular compressors). But there's a thing to say about > zstd: firefox.deb zstd -19 takes to unpack: > * 2.644s .xz, stock dpkg > * 2.532s .xz, my tool (libarchive based) > * 0.290s .zst, my tool > * 0.738s .gz, stock dpkg > * 0.729s .gz 0.939, stock dpkg > File sizes being 60628216 gz, 47959544 zstd, 44506304 xz. > > XFCE install total: 723M xz, 773M zstd, 963M gzip. > > Thus, even though we'd want to stick with xz for the official archive, speed > gains from zstd are so massive that it's tempting to add support for it, > at least for non-official uses, possibly also for common Build-Depends. > The usual objection, "we don't want to bloat the Essential set" doesn't hold > water because 1. libzstd is already a part of the Required set in Buster, > 2. a non-default compressor can be dlopened. > > Thoughts? > > But, the dlopen idea shows a potential victim: bzip2. Let's kill it. > > Stats for Buster's packages: > > .deb format: > 2.0: 100% > > control: > gz 11671 > xz 45210 > > data: > gz 966 > xz 55915 > > With not a single package in the archive still using bz2, removing support > would be reasonable. It'd be okay to give a clear error message telling the > user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still > unpack historic .debs if need be. > > Meow!
Hi, devscripts MR!122[1] proposes to add Zstandard support to uscan. Cheers, Xavier https://salsa.debian.org/debian/devscripts/merge_requests/122
