I tested it a bit more today. I used the standard poky's local.conf and
added those lines. You can see that at zstd level 9 there is still a
significant difference in the compression ratio with xz for gcc-dbg which
is a big file. At zstd level 19, gcc-dbg is 241MB big instead of 214MB with
xz (12.6% more).
PACKAGE_CLASSES = "package_ipk"
ZSTD_DEFAULTS = "-T0 -19"
PACKAGECONFIG:append:pn-opkg-native = " zstd"
OPKGBUILDCMD = "opkg-build -Z zstd -a ${ZSTD_DEFAULTS}"
Zstd at level 9:
build$ ls -lh tmp-zstd-level9/deploy/ipk/core2-64/ --sort=size | head -n10
total 1.5G
-rw-r--r-- 3 ecordonnier ecordonnier 302M Sep 28 18:33
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 249M Sep 28 18:33
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 106M Sep 28 18:33
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 44M Sep 28 18:33
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 26M Sep 28 18:33
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 23M Sep 28 18:33
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:32
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:33
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 28 18:33
coreutils-dbg_9.1-r0_core2-64.ipk
zstd at level 19:
build$ ls -lh tmp-zstd-level19/deploy/ipk/core2-64/ --sort=size | head -n10
total 1.1G
-rw-r--r-- 3 ecordonnier ecordonnier 241M Sep 28 18:04
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 213M Sep 28 18:03
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 36M Sep 28 17:58
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 30M Sep 28 18:00
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 22M Sep 28 17:59
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 28 17:58
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 14M Sep 28 17:57
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:58
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:57
g++_12.2.0-r0_core2-64.ipk
xz at level 9 (the default):
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n10
total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 14 10:44
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 19M Sep 14 10:44
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 14 10:44
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 14 10:44
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 11M Sep 14 10:44
g++_12.2.0-r0_core2-64.ipk
On Wed, Sep 14, 2022 at 5:42 PM Khem Raj <[email protected]> wrote:
> On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <[email protected]> wrote:
> >
> > Thanks for checking.
> >
> > I'd be interested to know if setting a higher compression level for zstd
> > can get us to a similar compression ratio to xz. If so, then I think it
> > could be some real value to distro maintainers to be able to *tune*
> > their compression.
>
> yeah it will be interesting to say try something like level 9 but I think
> times
> might regress with that but it might be good to know the balance and
> perhaps
> suggest size mode and performance mode of zstd instead of xz
>
> >
> > That's not blocking for your new PR though.
> >
> >
> > On 9/14/22 05:08, Etienne Cordonnier wrote:
> > > Also note that zstd's default compression level is 3 per default (from
> > > a 1 to 19 scale). I did not test other compression levels.
> > >
> > > On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier
> > > <[email protected]> wrote:
> > >
> > > I ran a build of core-image-full-cmdline using xz and zstd, using
> > > pre-populated downloads and sstate-cache directories but with
> > > empty tmp directory. Here are the numbers:
> > > zstd:
> > > bitbake core-image-full-cmdline took 2m52.768s (real), the
> > > resulting directory tmp/deploy/ipk is 1.6GB big.
> > > xz:
> > > bitbake core-image-full-cmdline took 4m14.214s (real), the
> > > resulting directory tmp/deploy/ipk/ is 996M big
> > >
> > > So the build with xz is 47% slower (254/172) than with zstd for
> > > this "incremental build" use-case.
> > >
> > > See the 5 biggest packages, the difference in compression-ratio
> > > increases with big files:
> > > build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> > > total 1.6G
> > > -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
> > > gcc-dbg_12.2.0-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
> > > openssl-dbg_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
> > > openssl-ptest_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
> > > binutils-dbg_2.39-r0_core2-64.ipk
> > > build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> > > total 963M
> > > -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
> > > gcc-dbg_12.2.0-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
> > > openssl-dbg_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
> > > openssl-ptest_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
> > > binutils-dbg_2.39-r0_core2-64.ipk
> > > ...
> > >
> > > I think for use-cases where the size of the ipk packages matters,
> > > xz is better. For use-cases where it does not matter (ipk packages
> > > not deployed), build-time matters more than compression-ratio and
> > > zstd is better.
> > >
> > > Regarding the enablement of zstd in opkg per default, I'll send a
> > > new version of the patch without this line.
> > > My thinking for enabling zstd per default in opkg was that zstd is
> > > already enabled per default in libarchive's PACKAGECONFIG, and
> > > disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
> > > code from opkg (opkg uses libarchive for doing the actual
> > > compression/decompression).
> > >
> > > On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
> > > <[email protected]> wrote:
> > >
> > >
> > >
> > > On 9/13/22 15:20, Alex Feinman wrote:
> > > > I do have some numbers. When I was selling this change
> > > internally, I
> > > > did a comparison on our internal build.
> > > > Combined write IPK times (Σ t
> do_package_write_ipk)
> > > > xz 162m 35s
> > > > gz 52m 13s
> > > > zstd 33m 49s
> > > > Compression rate for zstd was closer to xz than to gz but
> > > not as good
> > > > as xz. For systems that have to cache packages on the device
> > > with
> > > > limited storage xz might be a better option, but for the
> > > bulk of
> > > > projects zstd is the best choice
> > > > Additionally, zstd offers much faster decompression than xz
> > > so the
> > > > rootfs build step that includes unpacking all of the ipks,
> > > takes 3m
> > > > 58s with xz and 2m 44s with zstd.
> > > > One other thing of note - if your build includes debug
> > > packages, some
> > > > may be quite large. E.g. one of our components produces a
> > > 2.2 GB debug
> > > > package (uncompressed). On large files xz requires a
> > > disproportionally
> > > > large amount of time resulting in 15 minutes needed to
> > > simply write
> > > > ipk for the abovementioned packages, whereas zstd took about
> > > 45 sec.
> > > > For frequent tasks like bitbaking a single package this
> > > translates in
> > > > a lot of saved time.
> > >
> > > Those are certainly compelling performance improvements.
> > > Assuming that
> > > the final data-segment size is within 5%-ish of xz, then I
> > > would agree
> > > with the rest of the thread that it should probably be the
> > > contemporary
> > > default.
> > >
> > > And if we make it the default compressor for OE IPKs, then
> > > obviously my
> > > criticism in the original PR is satisfied.
> > >
> > > > Bottom line - I think making xz a default package compressor
> > > was not
> > > > entirely thought through. gzip or zstd is what the default
> > > should be.
> > >
> > > ZStandard support was only added to opkg last September [1].
> > > Before
> > > that, xz was the new hotness that replaced gzip. :)
> > >
> > > [1]
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
> >
> > >
> > >
> > > > One final note: I could not find a reasonable explanation
> > > for why
> > > > opkg-tools require code changes to support a different
> > > compressor. BSD
> > > > tar and GNU tar both can easily accept compressors that they
> > > have no
> > > > idea about (via -I option) because all of them provide a
> > > unified
> > > > command line interface for use in pipes. If this were done
> > > similar to
> > > > tar, we could have used any compressor we wanted, including
> the
> > > > multithreaded versions (zstdmt)
> > >
> > > Well, presumably IPK creation tools can only support the
> > > matrix of
> > > compression algorithms which your opkg binary can decompress.
> > > I suppose
> > > someone could try to implement a plugable compression module
> > > system for
> > > opkg. But given that nearly everyone uses opkg in an embedded
> > > context,
> > > I'm not sure it would get much use.
> > >
> > > >
> > > > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
> > > <[email protected]> wrote:
> > > >
> > > > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> > > > <[email protected]> wrote:
> > > > >
> > > > > ACK from me - apart from enabling zstd by default.
> > > > >
> > > > > On 9/13/22 07:37, Etienne Cordonnier via
> > > lists.openembedded.org
> > > <
> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$
> >
> > > > wrote:
> > > > > > This allows the use of zstd for opkg packages by
> using
> > > > OPKGBUILDCMD:
> > > > > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > > > > >
> > > > > > Signed-off-by: Alex Feinman <[email protected]>
> > > > > > Signed-off-by: Etienne Cordonnier <
> [email protected]>
> > > > > > ---
> > > > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > | 3 ++-
> > > > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git
> > > a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > > > index 7b351e8123..e38d9d6f3f 100644
> > > > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > > > > target_localstatedir := "${localstatedir}"
> > > > > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > > > > >
> > > > > > -PACKAGECONFIG ??= "libsolv"
> > > > > > +PACKAGECONFIG ??= "libsolv zstd"
> > > > >
> > > > > Building in zstd support by default is a little
> > > suspect to me.
> > > > >
> > > > > Unless I'm mistaken, OE-core will only build
> > > xz-compressed IPKs by
> > > > > default. So zstd support would be unnecessary for a
> distro
> > > > integrator
> > > > > who just uses upstream OE-core.
> > > > >
> > > > > For distros which use zstd compression in their
> > > packages, I think it
> > > > > would be more appropriate to overwrite the opkg
> > > PACKAGECONFIG in a
> > > > > .bbappend.
> > > > >
> > > >
> > > > This is perhaps fine. I do wonder if there is some
> > > performance
> > > > comparison data between xz and zstd compressed ipks
> > > > with opkg, it might help users on making this choice and
> > > also if we
> > > > should consider using
> > > > zstd by default at some point or not.
> > > >
> > > > > Is there something I'm not considering here?
> > > > >
> > > > > >
> > > > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > > > > gnupg gpgme libgpg-error,\
> > > > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> > > > "--enable-gpg,--disable-gpg,\
> > > > > > PACKAGECONFIG[curl] =
> > > "--enable-curl,--disable-curl,curl"
> > > > > > PACKAGECONFIG[ssl-curl] =
> > > > "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > > > > PACKAGECONFIG[sha256] =
> > > "--enable-sha256,--disable-sha256"
> > > > > > +PACKAGECONFIG[zstd] =
> > > "--enable-zstd,--disable-zstd,zstd"
> > > > > > PACKAGECONFIG[libsolv] =
> > > > "--with-libsolv,--without-libsolv,libsolv"
> > > > > >
> > > > > > EXTRA_OECONF:class-native =
> > > > "--localstatedir=/${@os.path.relpath('${localstatedir}',
> > > > '${STAGING_DIR_NATIVE}')}
> > > > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> > > > '${STAGING_DIR_NATIVE}')}"
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Alex Stewart
> > > > > Software Engineer - NI Real-Time OS
> > > > > NI (National Instruments)
> > > > >
> > > > > [email protected]
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > > --
> > > Alex Stewart
> > > Software Engineer - NI Real-Time OS
> > > NI (National Instruments)
> > >
> > > [email protected]
> > >
> >
> > --
> > Alex Stewart
> > Software Engineer - NI Real-Time OS
> > NI (National Instruments)
> >
> > [email protected]
> >
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171151):
https://lists.openembedded.org/g/openembedded-core/message/171151
Mute This Topic: https://lists.openembedded.org/mt/93654146/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
