On 10 October 2018 at 18:52, Cleber Rosa <cr...@redhat.com> wrote: > > > On 10/10/18 12:23 PM, Peter Maydell wrote: >> On 10 October 2018 at 16:47, Cleber Rosa <cr...@redhat.com> wrote: >>> To make sure we're on the same page, we're still going to have default >>> machine types, based on the arch, for those targets that don't provide >>> one (aarch64 is one example). Right? >> >> Does it make sense to define a default? The reason arm >> doesn't specify a default machine type is because you >> can't just run any old guest on any old machine type. >> You need to know "this guest image will run on machine >> type X", and run it on machine type X. This is like >> knowing you need to run a test on x86 PC and not >> on PPC spapr. >> > > While requiring tests to specify every single aspect of the environment > that will be used may be OK for low level unit tests, it puts a lot of > burden on higher level tests (which is supposed to be the vast majority > under tests/acceptance). > > From a test writer perspective, working on these higher level tests, it > may want to make sure that feature "X", unrelated to the target arch, > machine type, etc, "just works". You man want to look at the "vnc.py" > test for a real world example.
OK, if it doesn't have a dependency on machine at all, it should state that somehow. > Eduardo has suggested that "make check-acceptance" runs all (possible) > tests on all target archs by default. Yeah; or we have some mechanism for trimming down the matrix of what we run. But I think it's better coverage if we have 3 tests ABC that don't depend on machine and 3 machines XYZ to run AX BY CZ than AX BX CX by specifying X as an arbitrary "default". It looks like the 'vnc' test is just testing QEMU functionality, not anything that involves interacting with the guest or machine model? There's a good argument that that only really needs to be run once, not once per architecture. You might also want to consider the "none" machine, which exists for bits of test infrastructure that aren't actually trying to run guests. >> Would it make more sense for each test to specify >> which machine types it can work on? >> > > I think it does, but I believe in the black list approach, instead of > the white list. > > The reason for that is that I believe that majority of the tests under > "tests/acceptance" can be made to work on every target (which would be > the default). So far, I've made sure tests behave correctly on the 5 > arches included in the "archs.json" file in this series (x86_64, ppc64, > ppc, aarch64, s390x). > > To give a full disclosure, "boot_linux.py" (boots a linux kernel) is > x86_64 specific, and CANCELS when asked to be run on other archs. But, > on the work I've done top of these series, it already works with ppc64 > and aarch64. Also, "boot_linux.py" sent in another series, (which boots > a full linux guest) is also being adapted to work on most of the target > archs. Right, "boot Linux" is machine specific. The kernel/disk /etc that boots on aarch64 virt is probably not going to boot on the 64-bit xilinx board; and on 32-bit arm you definitely are going to want a different kernel in some places. This is likely to be true of most tests that actually try to run code in the guest. We should aim to test the machines we care about (regardless of what architectures they are), rather than thinking about it in terms of "testing architectures X, Y, Z", I think. I think you're going to need at least some whitelist functionality; otherwise half the tests are going to break every time we add a new machine (and "add every new machine to the blacklist for half the tests" doesn't scale very well). thanks -- PMM
