On Fri, 18 Oct 2024 at 11:00, Daniel P. Berrangé <berra...@redhat.com> wrote: > > On Fri, Oct 18, 2024 at 10:46:55AM +0100, Peter Maydell wrote: > > We do not have infinite CI resources, unfortunately. Migration > > is competing with everything else for time on CI. You have to > > find a balance between "what do we run every time" and "what > > do we only run when specifically testing a migration pullreq". > > Similarly, there's a lot of iotests but we don't run all of them > > for every block backend for every CI job via "make check". > > The combos we don't run for iotests are a good source of > regressions too :-( > > > Long test times for tests run under "make check" are also bad > > for individual developers -- if I'm running "make check" to > > test a target/arm change I've made I don't really want that > > to then spend 15 minutes testing the migration code that > > I haven't touched and that is vanishingly unlikely to be > > affected by my patches. > > Migration-test *used* to take 15 minutes to run, but that was a > very long time ago. A run of it today is around 1m20. > > That said, if you are building multiple system emulators, we > run the same test multiple times, and with the number of > targets we have, that will be painful.
Yeah. Here's a recent s390 job, and not one that was running slow: https://gitlab.com/qemu-project/qemu/-/jobs/8112195449 The migration tests it ran were: 95/954 qemu:qtest+qtest-ppc64 / qtest-ppc64/migration-test OK 196.95s 50 subtests passed 96/954 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test OK 202.47s 50 subtests passed 99/954 qemu:qtest+qtest-i386 / qtest-i386/migration-test OK 203.54s 52 subtests passed 107/954 qemu:qtest+qtest-s390x / qtest-s390x/migration-test OK 193.65s 50 subtests passed 156/954 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test OK 164.44s 52 subtests passed So that's 192s (over 3 minutes) average, and over 16 minutes total spent on migration testing on this one CI job. If the s390 VM has a noisy-neighbour problem then the migration tests can hit their 8 minute timeout, implying 40 minutes spent on migration testing alone... thanks -- PMM
