Hello, I would like to discuss BSP Test results early in the release cycle in the hope we avoid the last minute issues we encountered with RTEMS 5 and the "expected" failure state ticket.
I would like to update this section ... https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states to state there is to be a ticket for each `expected-fail` test state. I believe this was the outcome of the discussion that took place. Please correct me if this is not correct. The purpose of the `expected-fail` is to aid the accounting of the test results to let us know if there are any regressions. We need to account for tests that fail so we can track if a recent commit results in a new failure, i.e. a regression. To do this we need to capture the state in a way `rtems-test` can indicate a regression. I think the `indeterminate` state may need further explanation as it will help in the cases a simulator passes a test but the test fails on some hardware. I am currently seeing this with spcache01 on the PC BSP. With the level of continuous building and testing we are currently doing being able to easily determine a regression will become important. Check out the example below. I would like to avoid us sitting with failures that do not have tickets and are not accounted for. I know there is a lump of work to account for the failures and after that is done I think the effort needed to maintain the failure states will drop. As a result I have been pondering how I can encourage this work be done. I am considering updating the tier-1 status to requiring there be 0 unaccounted for failures. That is the `rtems-test`'s Failure count is 0 for a hardware test run. Chris An example using Joel's recent test run (thanks Joel :)). The sparc/leon2 results show no regressions: Summary ======= Passed: 580 Failed: 0 User Input: 6 Expected Fail: 1 Indeterminate: 0 Benchmark: 3 Timeout: 0 Invalid: 0 Wrong Version: 0 Wrong Build: 0 Wrong Tools: 0 ------------------ Total: 590 [ https://lists.rtems.org/pipermail/build/2020-September/018089.html ] while the sparc/erc32 has a single failure: Summary ======= Passed: 579 Failed: 1 User Input: 6 Expected Fail: 1 Indeterminate: 0 Benchmark: 3 Timeout: 0 Invalid: 0 Wrong Version: 0 Wrong Build: 0 Wrong Tools: 0 ------------------ Total: 590 Failures: spintrcritical08.exe [ https://lists.rtems.org/pipermail/build/2020-September/018088.html ] _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
