On 5/21/21 7:14 PM, Thomas Huth wrote: > On 21/05/2021 16.29, Peter Maydell wrote: >> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <[email protected]> >> wrote: >>> If you think these tests belong to tests/tcg/, I am OK to put >>> them they, but I don't think adding the Avocado buildsys >>> machinery to the already-complex tests/tcg/ Makefiles is going >>> to help us... >> >> This does raise the question of what we're actually trying >> to distinguish. It seems to me somewhat that what tests/acceptance/ >> actually contains that makes it interestingly different from other >> tests/ stuff is that it's specifically "tests using the Avocado >> framework". On that theory we might name it tests/avocado/. > > I think there are two aspects: > > 1) These tests are using the avocado framework > > 2) These tests are downloading other stuff from the internet (unlike the > other tests that we have) > >> Or we could just leave it as it is -- is the current naming >> actually confusing anybody? :-) > > Yes, I think "acceptance" is rather confusing. So far they haven't been > part of your PR acceptance tests (well, now they are part of the > gitlab-CI, though), and it's also not about tests that have been set up > by customers, which is what you normally think of when hearing > "acceptance tests". So a different name would be adequate.
IIUC the current "acceptance tests" are the ones Peter runs, which are *gating* the merge process. They can not fail. The current tests in tests/acceptance/ use Avocado (as said Thomas, to easily download artifacts) and shouldn't be considered gating; they could fail. This is my confusion so far. It should be OK to add tests using the Avocado framework which might fail and aren't gating.
