On Tue, Feb 2, 2021 at 7:40 AM Joel Sherrill <j...@rtems.org> wrote: > > > On Mon, Feb 1, 2021 at 6:50 PM Chris Johns <chr...@rtems.org> wrote: > >> >> >> On 2/2/21 9:12 am, Joel Sherrill wrote: >> > On Mon, Feb 1, 2021 at 3:50 PM Chris Johns <chr...@rtems.org >> > <mailto:chr...@rtems.org>> wrote: >> > On 2/2/21 3:42 am, Joel Sherrill wrote: >> > > Hi >> > > >> > > On the aarch64 qemu testing, we are seeing some tests which seem >> to pass >> > most of >> > > the time but fail intermittently. It appears to be based somewhat >> on host load >> > > but there may be other factors. >> > > >> > > There does not appear to be a good test results state for these. >> Marking them >> > > expected pass or fail means they will get flagged incorrectly >> sometimes. >> > >> > We have the test state 'indeterminate' ... >> > >> > >> https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states >> > < >> https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states >> > >> > >> > It is for this type of test result. >> > >> > > I don't see not running them as a good option. Beyond adding a >> new state to >> > > reflect this oddity, any suggestions? >> > >> > I prefer we used the already defined and documented state. >> > >> > +1 >> > >> > Kinsey had already marked them as indeterminate and the guys were in >> the >> > process of documenting why. I interpreted the question of what to do >> more >> > broadly than it needed to be but the discussion was good. >> >> A discussion is needed and welcome. Handling these intermittent simulator >> failures is hard. I once looked into some gdb simulator cases when I >> first put >> rtems-test together and found myself quickly heading into a deep dark >> hole. I >> have not been back since. >> > > Agreed it is ugly. > > If the BSP has a simulator variant, then using the test configuration is > appropriate. > > But for the PC and leon3, we don't have separate sim builds of the BSP so > if > there are intermittent failures there, we would have to mark them in the > set shared with hardware test runs. That's bad. > yeah, don't do that.
> > It's almost like we might need a conditional like "sp04: intermittent > sim=qemu" > or something. Which means build it but the tester ini could know the > simulator > type and adjust its expectations. May have to account for multiple > simulators > on the sim=XXX though. Just a thought. > > maybe pass a sim.tcfg file to tester that is different for the sim.cfg file than it is for the hw.cfg file? > This is a dark hole. > > >> >> Chris >> > _______________________________________________ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel
_______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
