On Fri, 16 May 2025 at 12:45:43 +0200, Santiago Vila wrote:
Regarding the flakiness itself, I get a failure rate around 20%
on machines with 1 CPU and 30% on machines with 2 CPUs. This is
greater than the reference thresholds given by Paul in one
of the gcr bugs.
I'd like to propose a patch, but the tests which fail are
different every time. On a sample of 200 build tries on
different machines, I get the following failures these many times:
26 multithread-test
23 proxy-test
22 range-test
22 connection-test
22 auth-test
6 server-test
1 timeout-test
1 hsts-test
Is this still the same failure mode described in the bug title, with
"Address already in use" and "could not bind to address ..." being
reported by Apache?
Last time I looked at the libsoup* test suite, the actual tests were
each reasonably reliable, but the reliability issue was with their
setup/teardown. They run a temporary Apache web server, in order to
have a realistic server to test against. I think what's happening is
that sometimes, the web server port from one test (let's say test number
5) is still considered by the kernel to be in use by the time we reach
the setup stage of the next test (let's say test number 6).
As a result, the Apache for test number 6 can't listen on the port it
has been configured to use, and testing fails at that point. This is
rare on a per-test basis, therefore difficult to reproduce on-demand -
but running the whole test suite involves several setup/teardown cycles,
resulting in a higher failure rate for the test suite as a whole. For
example if you're seeing a 30% failure rate, that might be more like a
2% failure rate for each of 15 test executables, or perhaps even a 0.2%
failure rate for each of 150 smaller test-cases.
If that's still what is happening, then it's expected that you will see
failures in different tests (and even in different test-cases within
those larger tests) on different occasions.
Unfortunately, if that's the case, then skipping any specific test-case
is not going to be a viable workaround, because it's the common
setup/teardown done for each test-case that is the problem.
If it's possible to configure Apache to set options like SO_REUSEADDR
and/or SO_REUSEPORT then that might help (but I don't know whether
that's possible).
Or if it's possible to make the test suite use a different port for each
test then that might help (but I don't know whether that will be
feasible).
smcv
