Quoting Diederik de Haas (2022-01-11 16:39:47) > When version 1.21-1 was uploaded, the buildd failed for all the > release architecures. Looking into several of them, I found that they > failed on the same test: > "test-dpp: unit/test-dpp.c:114: test_key_derivation: Assertion `m' > failed." > I went then on OFTC:/#iwd to ask whether they'd (instantly) know what > may have caused it. While an immediate cause wasn't found, it did > bring some interesting aspects to light. > One of the upstream maintainers ran a build and reported that 22 tests > ran and they all succeeded. On buildd 1 test out of *20* failed.
The Debian package currently disables 2 tests, due to them (succeeding locally using cowbuild, but) failing on Debian build daemons using sbuild: https://salsa.debian.org/debian/iwd/-/blob/debian/master/debian/rules#L5 Mentioning both as that explains the 20 versus 22 tests, but also because it sounds like you might have revealed the reason not only for the newly introduced test failing but also those other two. > One of the great features (IMO) of iwd is that it reuses functionality > available in the kernel. But that also means it is dependent on the > kernel configuration. > In Debian kernels the rule is to build kernel modules as loadable > modules (=m) as much as possible and not builtin (=y). > > One of the triggers for a kernel module to get loaded is when the kernel > detects a certain piece of hardware present on the system, like a soundcard > or a HWRNG. AFAIK this works quite well. > But it also means that the loaded kernel modules differ from machine to > machine as their hardware differs. Or in case of Virtual Machines, which > (virtual) hardware is configured in that VM. > Another trigger (AFAIK) is when a certain process needs a certain kernel > module, it gets loaded (ondemand). > And the user also has the possibility to explicitly load kernel modules, > either through a direct `modprobe` or via `/etc/modules` (f.e.) > > All this means that a test on my laptop with 5.15.0-2-amd kernel may > succeed, while that same test may fail on my PC with the same kernel. > For example I noticed SHA512 kernel modules were loaded on my laptop, > while they were not on my PC. > The buildds run Debian Stable with a 5.10.0-10 kernel, which is already > different, but its hardware is very likely also different. > Different architectures also have different kernel configurations and > it's quite likely that caused the same test to succeed for several other > (non-release) architectures. And as they're different machines, the > hardware is also different and therefor the loaded kernel modules as > well. > > It can also mean that a build succeeds or fails because of other things > that happen concurrently or previously on *that same buildd machine*, > just because another build/process triggered a certain kernel module to > be loaded, which may stay loaded until that buildd machine gets rebooted. > > The (high) preference is that builds are reproducible and not dependent > on variable hardware/kernel config/other builds that may have happened. > > One possible solution could be making sure certain kernel modules are > loaded before all or a certain test is run. This page may be helpful: > https://iwd.wiki.kernel.org/gettingstarted#kernel_dependencies > This would need to be implemented/applied on the Debian side as Debian > has its own kernel configuration (per architecture). > Another possibility is to conditionally run a test based on whether a > kernel module is loaded. Note that checking `lsmod` is insufficient as > a kernel module can be builtin, like CONFIG_CRYPTO_SHA256 is. > > I don't know what's possible or desirable to address the issues I've > raised above (and there may be other issues as well), but I hope this > bug report is a useful start for such a discussion. In itself your report helps point to a likely cause of test failures that had me puzzled, as mentioned above. So thanks for that! Simplest is to just skip failing tests - as I did already. Next would be to share that upstream, and (as they deem sensible) allow them to implement gracefully skipping tests generally, for the benefit of other users as well - as you did already for one of the 3 tests. Maybe the kernel modules needed for (some of) these 3 failing tests would be generally beneficial for Debian users to have compiled built-in for the default Debian kernel. If that seems likely, it would be helpful to open a bug against the package "linux" suggesting that to the linux kernel maintainers. I guess (but don't know for sure) the official build daemons cannot be told to load modules - i.e. that builds are executes inside some container not permitting that. As you also elaborate on, that might open a can of worm related to reproducibility. So I kinda think it is better to *not* check tests requiring non-eneric hardware features. Somewhat related to that is this recent discussion about a package depending on certain GPU features for its testsuite: https://lists.debian.org/202346fb-b2a2-f393-9c01-afa0871d8...@codeweavers.com Seems to me that this case is similar, and we should simply enable only the parts of the testsuite that can _generally_ be expected to succeed (and work with upstream to improve tests to gracefully handle kernel features missing, so that we can enable the full testsuite). > PS: Based on the discussion I had on IRC, the following patch was > proposed (and applied) upstream: > https://lists.01.org/hyperkitty/list/i...@lists.01.org/thread/USXHQD27OI7RJEC52MPJLNGQCKEBBJKO/ Great. If I read that patch correctly, that means we can re-enable unit/test-dpp when that patch lands (i.e. from next upstream release). Would be helpful if you could suggest upstream to try identify and similarly patch tests unit/test-wsc unit/test-eap-sim - assuming the cause of failure is similar (since the pattern - failure not locally but on build daemons - is same). Thanks a lot, - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private
signature.asc
Description: signature
