On Mon, Feb 1, 2021, 11:45 AM Gedare Bloom <ged...@rtems.org> wrote: > > > On Mon, Feb 1, 2021 at 10:31 AM Joel Sherrill <j...@rtems.org> wrote: > >> >> >> On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom <ged...@rtems.org> wrote: >> >>> >>> >>> On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill <j...@rtems.org> wrote: >>> >>>> Hi >>>> >>>> On the aarch64 qemu testing, we are seeing some tests which seem to >>>> pass most of the time but fail intermittently. It appears to be based >>>> somewhat on host load but there may be other factors. >>>> >>>> >>> There does not appear to be a good test results state for these. Marking >>>> them expected pass or fail means they will get flagged incorrectly >>>> sometimes. >>>> >>>> >>> Are they expected to pass every time? >>> >> >> Normally yes. But there appears to be some external factors particularly >> load impacting qemu which lead to failures on target side code. >> >> > Since they can be made to pass by reducing the load (I assume), then you > should document this in the target documentation and not change the testing > expectations. The test should pass. If too much load cause qemu to fail > randomly, this should be known and avoided or else your test results are > much less useful. >
I suggested a short paragraph in the qemu section of the BSP section in the users guide and comments in the waf equivalent of a tcfg file. But there isn't any other obvious place where someone would.look. > Does anyone know this happens on other architectures with qemu? > Kinsey has noted the same on arm. > > >> >>> Intermittent failures are suspicious, and there is limited value to >>> running a test that has an "intermittent failure" state, since it will >>> always be successful if you don't care if it passes or fails. >>> >> >> Yeah. No matter what you mark them, it sucks. >> >>> >>> >>> >>>> I don't see not running them as a good option. Beyond adding a new >>>> state to reflect this oddity, any suggestions? >>>> >>>> --joel >>>> _______________________________________________ >>>> devel mailing list >>>> devel@rtems.org >>>> http://lists.rtems.org/mailman/listinfo/devel >>> >>>
_______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
