On Mon, Sep 7, 2020 at 12:44 AM Chris Johns <chr...@rtems.org> wrote:
> On 7/9/20 2:16 pm, Joel Sherrill wrote: > > > > > > On Sun, Sep 6, 2020, 9:55 PM Chris Johns <chr...@rtems.org > > <mailto:chr...@rtems.org>> wrote: > > > > Hello, > > > > I would like to discuss BSP Test results early in the release cycle > in the hope > > we avoid the last minute issues we encountered with RTEMS 5 and the > "expected" > > failure state ticket. > > > > I would like to update this section ... > > > > > https://docs.rtems.org/branches/master/user/testing/tests.html#expected-test-states > > > > to state there is to be a ticket for each `expected-fail` test > state. I believe > > this was the outcome of the discussion that took place. Please > correct me if > > this is not correct. > > > > The purpose of the `expected-fail` is to aid the accounting of the > test results > > to let us know if there are any regressions. We need to account for > tests that > > fail so we can track if a recent commit results in a new failure, > i.e. a > > regression. To do this we need to capture the state in a way > `rtems-test` can > > indicate a regression. > > > > I think the `indeterminate` state may need further explanation as it > will help > > in the cases a simulator passes a test but the test fails on some > hardware. I am > > currently seeing this with spcache01 on the PC BSP. > > > > > > I don't mind this as long as we have a rigid format to ensure these are > easy to > > find. > > Do you mean find the location in the tcfg files? > No. I just mean sometimes there is a comment directly above the l exclude line, sometimes above a group, etc. I would want something like exclude: dl06 # #1234 dynamic loading broken on RISC-V Maybe even not as a comment but a ticket number so it could be checked that every exclude had a ticket number in the form of #NNNN[N] (four or five digits). > > > Perhaps info at the main be of the line marking this state. In the past, > > we have tended to our comments before. > > If you mean the location in the tcfg files being printed out by the test, > have > no idea how to do that. What about grep? > grep works if on the same line. Hence the above suggestion. If we want to add excludes, let's document which test and ticket in a way that theoretically we could check if the ticket is open or closed. No way to automate that it applies to this BSP's exclude I think. > > With the level of continuous building and testing we are currently > doing being > > able to easily determine a regression will become important. Check > out the > > example below. > > > > I would like to avoid us sitting with failures that do not have > tickets and are > > not accounted for. I know there is a lump of work to account for the > failures > > and after that is done I think the effort needed to maintain the > failure states > > will drop. > > > > As a result I have been pondering how I can encourage this work be > done. I am > > considering updating the tier-1 status to requiring there be 0 > unaccounted for > > failures. That is the `rtems-test`'s Failure count is 0 for a > hardware test run. > > > > Chris > > > > An example using Joel's recent test run (thanks Joel :)). > > > > > > No problem. I do want to point out that you can't tell that the tests > ran on sis > > rather than Tsim or some specific hardware. For leon3, you can add qemu > to the > > list of potential "platforms". I've mentioned this before. > > Maybe the `rtems-test` command line option name is misleading. It is > `--rtems-bsp=` but it is more like `--bsp-target=`. Some BSPs have more > than one > way to be run, for example psim and psim-run. I see in one of the tests I > linked > too the command line has `--rtems-bsp=leon2-sis`. > > > While typing this, I had the idea that maybe adding a note argument that > gets > > added to the report might be an easy solution. Whoever runs the test > could > > freeform annotate the board model, simulator and version, etc. This > would at > > least allow the logs to show which platform when we need to look for an > issue. > > If you think there should be a list of annotations a test run needs then > please > create a ticket with a list. > This would be enough to address my "HW or which simulator" question. > > > Also I am not sure but hopefully the test reports do accurately reflect > host OS. > > There is a "Host" section at the top of the results log? It is just `uname > -a`. > I think that's sufficient as long as it can distinguish Linux distributions. --joel > > Chris >
_______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
