Hello Michael,
On 25-09-06 13:09, Michael Tokarev wrote:
> Control: tag -1 + unreproducible
>
> On 04.09.2025 18:29, Jakub Ružička wrote:
> > Source: qemu
> > Version: 1:10.0.2+ds-2+deb13u1
> > Severity: important
> >
> > Dear Maintainer,
> >
> > when using arm64 emulation on amd64 host with
> >
> > pbuilder build --architecture arm64 --distro trixie
> >
> > it's incredibly slow to a point where tests fail due to timeouts and
> > installation of texlive-latex-extra takes tens of minutes.
> >
> > This problem is present:
> >
> > * only on Trixie GUESTS (Bookworm, Bullseye, and even Buster guests work
> > fine)
> > * on both Bookworm and Trixie amd64 HOSTS (tested on 2 Bookworm and 1 Trixie
> > host)
>
> So, from this we can conclude it is the GUEST which behaves differently,
> not qemu. I guess you can file a bug against arm64 trixie :)
>
> I guess pbuilder means qemu-user package (as I noted before, I never
> used pbuilder and don't know how to use it).
Yeah, sorry for not being more helpful with in the initial report, but
I, too, don't know how pbuilder uses qemu in particular. I tried to
elaborate on that below.
But the fact is that I'm unable to build trixie arm64 packages now with
a standard process that works for all other DIST/ARCH combinations
for both Debian and Ubuntu guests so something seems amiss.
The same build works on physical arm64 runners so it's related to
emulation of trixie.
> I can't confirm this observation, at least not generally.
>
> Some operations has indeed become slower. For example, I ran openssl
> speed benchmark on bookworm guest and trixie guest, here are some
> results on x86-64 host (ops/sec, more is better):
>
> Operation Bookworm Trixie
> md5/16 1496014 981482
> md5/64 1382894 967298
> md5/256 1158511 883699
> md5/1024 526236 469784
> md5/8192 107971 130183
> md5/16384 58668 72427
> sha1/16 794488 571021
> sha1/64 601573 466057
> sha1/256 362402 309965
> sha1/1024 141595 132793
> sha1/8192 21644 21597
> sha1/16384 11106 10937
> ...
The biggest slowdown here isn't even close to 100 % but I'm seeing orders
of magnitude slowdown.
> [...]
>
> I also tried to compile a few packages in a foreign arm64 chroot on
> an x86_64 machine - I don't see dramatic speed difference here, more
> like gradual difference due to compiler being "smarter", doing more
> optimizations and checking for more cases in the code - maybe 5% the
> difference overall. I observe similar difference in speed when
> building the same package on trixie and bookworm natively.
>
> So.. this is an 'unreproducible' for now.
>
> It might be particular source package you're building, or particular
> toolchain difference, - I dunno. It looks like it is not a qemu
> problem.
It's true I only observe this with bird3 and bird2 packages, knot and
knot-resolver compile and test fine which is really strange. But it's
been a consistent issue for months and it's reproducable across systems
for me.
I have both qemu-user-static and qemu-system-* packages installed.
When looking at processes:
/usr/libexec/qemu-binfmt/aarch64-binfmt-P
from qemu-user-static is used which implies user/process emulation as
opposed to system emulation, right? I don't use custom pbuilder
config, this is default.
The slowest reproducible case I've hit is the installation of
texlive-latex-extra package which took several hours (takes several
seconds normally):
# create pbuilder image (basetgz)
sudo pbuilder create --distribution trixie --architecture arm64
--basetgz /var/cache/pbuilder/trixie-arm64-base.tgz
# login to the image
sudo pbuilder login --distribution trixie --architecture arm64
--basetgz /var/cache/pbuilder/trixie-arm64-base.tgz
# install problematic packages
apt install texlive-latex-extra
This stuck on
Processing triggers for tex-common (6.19) ...
Running updmap-sys. This may take some time... done.
Running mktexlsr /var/lib/texmf ... done.
Building format(s) --all.
This may take some time...
the, processes like the following consumes 100 % CPU for eons:
/usr/libexec/qemu-binfmt/aarch64-binfmt-P /usr/bin/luatex luatex -ini
-jobname=luatex -progname=luatex luatex.ini
/usr/libexec/qemu-binfmt/aarch64-binfmt-P /usr/bin/pdftex pdftex -ini
-jobname=latex-dev -progname=latex-dev -translate-file=cp227.tcx *latex.ini
This takes a while on native (several seconds) but it takes upto hours (!) on
trixie/arm64 :(
Can you reproduce that?
If one gets past this point, bird2 / bird3 package build finishes, but
tests fail due to timeouts. So I could workaround by not running tests
probably, but something is obviously wrong as the tests pass on all
other DIST/ARCH.
Thanks for your help with the mystery.
Cheers,
Jakub Ružička
