On Wed, May 29, 2019 at 5:53 PM Josh Boyer <[email protected]> wrote:
>
> On Wed, May 29, 2019 at 4:20 PM Ben Cotton <[email protected]> wrote:
> >
> > https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression
> >
> > = Switch RPMs to zstd compression =
> >
> > == Summary ==
> > Binary RPMs are currently compressed with xz level 2.
> > Switching to zstd would increase decompression speed significantly.
> >
> > == Owner ==
> > * Name: [[User:dmach| Daniel Mach]]
> > * Email: [email protected]
> >
> > == Detailed Description ==
> > * The change requires setting a new compression algorithm in rpm
> > macros. Then a mass rebuild of all packages is required.
>
> The gcc team often does mass rebuilds on the side prior to updating
> gcc in Fedora. Would it be possible to do the same or leverage their
> rebuild work with the default changed in RPM to see what the true
> overall savings is? That would get us a lot more data to see if it's
> truly going to benefit the distro in terms of size and installation
> speed.
>
This is news to me, as I've never heard of any "side mass rebuilds".
They're prohibitively expensive to do, which is why we do only one per
release anyway.
> > * The macro for setting the compression is: %define _binary_payload
> > w19.zstdio
> > * The recommended compression level is 19. The builds will take
> > longer, but the additional compression time is negligible in the total
> > build time and it pays off in better compression ratio than xz lvl2
> > has.
> > * SRPM payload compression should stay at gzip (there's almost no
> > benefit in changing the compression, because SRPM's contents is
> > compressed already)
> >
> > === Use case: Firefox installation ===
> > I rebuilt firefox-66.0.5-1.fc30 with zstd level19.
> > Then I compared installation times with the original (xz compressed)
> > package:
> >
> > {| class="wikitable"
> > |-
> > ! Compression !! Target File System !! Time
> > |-
> > | xz level 2 || tmpfs || 8s
> > |-
> > | xz level 2 || ext4 on nvme || 11s
> > |-
> > | zstd level 19 || tmpfs || 2s
> > |-
> > | zstd level 19 || ext4 on nvme || 4s
> > |-
> > |}
> >
> >
> > === Comparison of compression algorithms and levels ===
> > Following table shows '''cpio''' and '''compressed cpio''' extraction
> > times into a tmpfs. Actual times in decompressing RPMs will differ due
> > to extracting on an actual disk and also some overhead in the RPM tool
> > (checks, scriptlets).
> >
> > {| class="wikitable"
> > |-
> > ! Compression !! Level !! Size B !! Size GiB
> > !! Compression time !! Compression time, 4 threads !!
> > Decompression time !! Comment
> > |-
> > | CPIO || - || 5016785692 || 4,7
> > || - || - || -
> > ||
> > |-
> > | xz || 2 || 1615017616 || 1,6
> > || 9m55s || - || 1m36s
> > || slow decompression
> > |-
> > | pxz || 2 || 1631869880 || 1,6
> > || - || 6m11s || 1m38s
> > || slow decompression
> > |-
> > | gzip || 9 || 2086354992 || 2,0
> > || 10m23s || - || 31s
> > || insufficient compression ratio
> > |-
> > | bzip2 || 9 || 1889161565 || 1,8
> > || 8m || - || 2m50s
> > || very slow decompression; compression ratio could be
> > better
> > |-
> > | zstd || 3 || 1913536587 || 1,8
> > || 31s || 29s || 6,5s
> > ||
> > |-
> > | zstd || 10 || 1737928978 || 1,7
> > || 3m27s || 2m34s || 6,3s
> > ||
> > |-
> > | zstd || 15 || 1717303256 || 1,7
> > || 9m37s || 6m34s || 6,3s
> > || identical compression speed to xz; fast decompression;
> > slightly worse compression ratio than xz
> > |-
> > | zstd || 17 || 1635525492 || 1,6
> > || 16m16s || 11m20s || 6,7s
> > ||
> > |-
> > | zstd || 19 || 1575843696 || 1,5
> > || 24m2s || 18m55s || 7,7s
> > ||
> > |-
> > |}
> >
> > == Benefit to Fedora ==
> > * Faster installations/upgrades of user systems
> > * Faster koji builds (installations in build roots)
> > * Faster container builds
> > * Lower bandwidth on mirrors if we choose the highest compression level
> >
I'm pretty sure this would break DeltaRPMs, since none of the drpm
software has been updated to handle zstd compression. Neither drpm nor
deltarpm handle it today.
> > == Scope ==
> > * Proposal owners: submit a patch to redhat-rpm-config
> > * Other developers: redhat-rpm-config maintainer: include the patch
> > and make a new build
> > * Release engineering: [https://pagure.io/releng/issue/8345 #8345]
> > mass rebuild is needed
> >
> > == Upgrade/compatibility impact ==
> > * RPM in Fedora supports zstd compression already (from Fedora 28,
> > rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected.
> > * Fedora <= 27 and some other distros will not be able to decompress
> > zstd-compressed RPMs.
>
> If we did this, wouldn't it make it very difficult to use tools like
> mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM
> support zstd?
>
We're pretty much screwed here. Also, since RHEL 8's rpm package does
not have zstd support compiled in, it too cannot handle the RPMs.
Cf. https://git.centos.org/rpms/rpm/blob/c8/f/SPECS/rpm.spec#_17-18
> Does MBS's concept of platform modules help us build a module across
> the RPM zstd-support boundary? I think it does, but I honestly can't
> remember for sure and I'm not aware of the details that go into MBS
> performing the build.
>
Why would this help? MBS does nothing useful in this regard. It just
calls Koji to make builds. When built for a specific platform, it'll
use the definitions of that platform. And since the platform maps to
the distro release, it's effectively the same as normal packages.
> > == How To Test ==
> > * dnf install <package>
> > * rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
> > * expected output: zstd 19
> >
> > Also the overall system installation time should decrease significantly.
> >
> > == User Experience ==
> > See '''Benefit to Fedora'''
> >
> > == Dependencies ==
> > N/A
> >
> > == Contingency Plan ==
> > * Contingency mechanism: Not needed, Fedora will stay at current
> > compression.
> > * Contingency deadline: N/A
>
> This seems wrong. If we get through a mass rebuild (or partial mass
> rebuild) and find some ugly unknown issue with zstd compression, we're
> going to have to do another mass rebuild to revert everything back,
> correct? That should be listed as the Contingency, even if it's
> unlikely.
>
Yeah, if this turns out bad, we'd need a second mass build to
eliminate packages with zstd compression.
> > * Blocks release? No
>
> Are we not advocating for a fully successful mass rebuild? Would we
> ship the distribution with only a portion (significant or otherwise)
> switched to zstd?
>
I'd hope not...
--
真実はいつも一つ！/ Always, there's only one truth!
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
