On 2/27/21 12:04 AM, Cleber Rosa wrote: > On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote: >> On 2/26/21 12:21 AM, Cleber Rosa wrote: >>> The "get-vm-images" target defined in tests/Makefile.include is a >>> prerequisite for "check-acceptance", so that those files get >>> downloaded before the Avocado job even starts. >>> >>> It looks like on c401c058a1c a TARGETS variable was introduced with a >>> different content than it was previously coming from the main >>> Makefile. From that point on, the "get-vm-images" succeed without >>> doing anything because there was no matching architecture to download. >> >> Any idea about how to detect such side effects (tests silently >> disabled) automatically? >> > > It wasn't really that any tests were disabled... they all continued to > run. In this case it was a broken make rule that caused the download > of the images, ahead of time, to not be performed. > > But your question is still valid and something that could happen. The > best answer I have is that all job results could and should also be > persisted in a structured way that is succeptible to being queried. > Then on top of that, you can build queries to show stability metrics, > regressions, etc. > > To that regards, I can speak about three possibilities: > > 1) Avocado has support for Fedora's resultsdb[1][2] > > 2) Because the Acceptance tests are already communicating the test > results to GitLab (via junit), using the GitLab API that lets you > query the detailed test results > > 3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin > that will export test resutls suitable to be used on a datawarehouse > tool developed by the Continuous Kernel Integration project[3]. This > is not generally available at the moment, but should be available > soon. > > Regards, > - Cleber. > > [1] - https://taskotron.fedoraproject.org/resultsdb/results > [2] - > https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin > [3] - https://cki-project.org
Wow this is thrilling! Maybe we could use fosshost to run a resultsdb VM.
