Re: [PATCH 4/6] testsuite: Add expected-fail to psim

[Sebastian Huber]

Hello,

On 09/05/2020 03:30, Gedare Bloom wrote:

Without these tests being tagged this way the user would have no idea where the 
stand after a build and test run and that would mean we would have to make sure 
a release has no failures. I consider that as not practical or realistic.

Maybe we need another state, e.g. something-is-broken-please-fix-it.

I do not think so, it is implicit in the failure or the test is broken. The 
only change is to add unexpected-pass, that will be on master after the 5 
branch.


I disagree with this in principle, and it should be reverted after we
branch 5. It's fine for now to get the release state sync'd, but we
should find a long-term solution that distinguishes the cases:
1. we don't expect this test to pass on this bsp
2. we expect this test to pass, but know it doesn't currently

They are two very different things, and I don't like conflating them
into one "expected-fail" case
originally, I had the same point of view. What I didn't take into 
account was the perspective of the tester. Now, I think it is perfectly 
fine to flag these tests as expected failure test states. Because right 
now, due to some known bugs such as https://devel.rtems.org/ticket/3982 
and probably also some more issues, these tests fail. On this BSP and 
this RTEMS version, they will always fail. This is not some sort of 
random failure. When we change test states to expected failure I think 
we should make sure that a ticket exists, which captures that there are 
some test results which indicate issues (expected failure test state). 
The ticket system is the better place to manage this. We should not use 
the test states for this. The test states should be used to figure out 
changes between different test runs. They should enable also to quickly 
check if the outcome of a test run yields the expected results for a 
certain RTEMS version and BSP.

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Small doubt in RTEMS testsuite codes

[Richi Dubey]

Heyy Everyone,

When I am opening testsuites/sptests/sp01/init.c , I can see the
following function definition:

rtems_task Init(  rtems_task_argument argument )

and when I open testsuites/sptests/sp01/task1.c ( which is
compiled(linked) together with init.c in the makefile in build of
bsp), I can see the following function:

rtems_task Task_1_through_3(  rtems_task_argument index).

Can someone please tell me how do the sp01 testsuite run without a int
main? If Init() is the start point then where does it gets it argument
'argument' from?


Thanks,
Richi.
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Small doubt in RTEMS testsuite codes

[Sebastian Huber]

On 12/05/2020 09:48, Richi Dubey wrote:


Heyy Everyone,

When I am opening testsuites/sptests/sp01/init.c , I can see the
following function definition:

rtems_task Init(  rtems_task_argument argument )

and when I open testsuites/sptests/sp01/task1.c ( which is
compiled(linked) together with init.c in the makefile in build of
bsp), I can see the following function:

rtems_task Task_1_through_3(  rtems_task_argument index).

Can someone please tell me how do the sp01 testsuite run without a int
main? If Init() is the start point then where does it gets it argument
'argument' from?


You may have a look at:

https://docs.rtems.org/branches/master/c-user/initialization.html

https://docs.rtems.org/branches/master/c-user/config/index.html

I think your question is not really well answered by our current 
documentation set. While you read the documentation please always try to 
improve it. I you think something is missing or could be better 
explained, then just send patches.


Independent of the documentation you should set up a test environment 
with GDB support. You can set a break point to Init(). This points you 
to the code which determines the argument value.


___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH 4/6] testsuite: Add expected-fail to psim

[Chris Johns]

On 12/5/20 5:15 pm, Sebastian Huber wrote:

Hello,

On 09/05/2020 03:30, Gedare Bloom wrote:
Without these tests being tagged this way the user would have no 
idea where the stand after a build and test run and that would mean 
we would have to make sure a release has no failures. I consider 
that as not practical or realistic.

Maybe we need another state, e.g. something-is-broken-please-fix-it.
I do not think so, it is implicit in the failure or the test is 
broken. The only change is to add unexpected-pass, that will be on 
master after the 5 branch.



I disagree with this in principle, and it should be reverted after we
branch 5. It's fine for now to get the release state sync'd, but we
should find a long-term solution that distinguishes the cases:
1. we don't expect this test to pass on this bsp
2. we expect this test to pass, but know it doesn't currently

They are two very different things, and I don't like conflating them
into one "expected-fail" case
originally, I had the same point of view. What I didn't take into 
account was the perspective of the tester. Now, I think it is perfectly 
fine to flag these tests as expected failure test states. Because right 
now, due to some known bugs such as https://devel.rtems.org/ticket/3982 
and probably also some more issues, these tests fail. On this BSP and 
this RTEMS version, they will always fail. This is not some sort of 
random failure. When we change test states to expected failure I think 
we should make sure that a ticket exists, which captures that there are 
some test results which indicate issues (expected failure test state). 
The ticket system is the better place to manage this. We should not use 
the test states for this. The test states should be used to figure out 
changes between different test runs. They should enable also to quickly 
check if the outcome of a test run yields the expected results for a 
certain RTEMS version and BSP.


Thanks. It is clear to me we lack documentation on this topic and this 
is an oversight on my part which I will attempt to correct.


I have reviewed Dejagnu and considered other things like the withdrawn 
IEEE 1003.3 standard and there are states we have that need to change 
but I think the original intent is the right path.


The Dejagnu states are documented here:

https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html#A-POSIX-Conforming-Test-Framework

And the exit codes are:

https://www.gnu.org/software/dejagnu/manual/Runtest.html#Runtest

For me they define the goal and intent.

The test states are metadata for the tester so it can determine the 
result of any given set of tests in relation to the expected state of 
the test when it was built. You need to detach yourself from being a 
developer and put yourself in the position of a tester who's task is to 
give an overall pass or fail for a specific build of RTEMS without 
needing to consider the specifics of any test, bug or feature.


The primary requirement is to allow machine check of the results to 
determine regressions. A regression is a failure, pass or unresolved 
result that was not expected.


My current thinking for test states are:

PASS:
The test has succeeded and passed without a failure.

UNEXCEPTED-PASS:
The test has succeeded when it was expected to fail.

FAIL:
The test has not succeeded and has failed when it was expected to pass. 
The failure can be a failed assert, unhandled exception, resource 
constraint, or a faulty test.


EXCEPTED-FAIL:
The test has not succeeded and has failed and this is expected.

UNRESOLVED:
The test has not completed and the result cannot be determined. The 
result can be unresolved because the test did not start or end, test 
harness failure, insufficient computing resources for the test harness 
to function correctly.


EXCEPTED-UNRESOLVED:
The test has not completed and the result cannot be determined and this 
is expected.


INDETERMINATE:
The test has succeeded, has failed or in unresolved. The test is an edge 
case where the test can pass, can fail, can be unresolved and this is 
expected.


USER-INPUT:
The test has not completed and the result is unresolved because it 
requires user intervention that cannot be provided.


BENCHMARK:
The test performs a performance type test. These are currently not 
supported.


UNTESTED:
The test has not run and is a place holder for a real test that is not 
yet provided.


UNSUPPORTED:
The test is not supported for this build of RTEMS, BSP or architecture.

Note:

1. Any expected failures, unresolved, or indeterminate test results are 
considered faults and require fixing.


2. The nature of a failure cannot be inferred from the test's metadata 
state.


3. The timeout and invalid states will be merged into UNRESOLVED.

4. The excluded state will be changed to UNSUPPORTED.

5. The metadata is placed in each test because is it an effective way to 
capture the state. Tests can be run as a group, stand alone or at 
different locatio

Re: [PATCH 4/6] testsuite: Add expected-fail to psim

[Joel Sherrill]

On Tue, May 12, 2020 at 4:11 AM Chris Johns  wrote:

> On 12/5/20 5:15 pm, Sebastian Huber wrote:
> > Hello,
> >
> > On 09/05/2020 03:30, Gedare Bloom wrote:
> > Without these tests being tagged this way the user would have no
> > idea where the stand after a build and test run and that would mean
> > we would have to make sure a release has no failures. I consider
> > that as not practical or realistic.
>  Maybe we need another state, e.g. something-is-broken-please-fix-it.
> >>> I do not think so, it is implicit in the failure or the test is
> >>> broken. The only change is to add unexpected-pass, that will be on
> >>> master after the 5 branch.
> >>>
> >> I disagree with this in principle, and it should be reverted after we
> >> branch 5. It's fine for now to get the release state sync'd, but we
> >> should find a long-term solution that distinguishes the cases:
> >> 1. we don't expect this test to pass on this bsp
> >> 2. we expect this test to pass, but know it doesn't currently
> >>
> >> They are two very different things, and I don't like conflating them
> >> into one "expected-fail" case
> > originally, I had the same point of view. What I didn't take into
> > account was the perspective of the tester. Now, I think it is perfectly
> > fine to flag these tests as expected failure test states. Because right
> > now, due to some known bugs such as https://devel.rtems.org/ticket/3982
> > and probably also some more issues, these tests fail. On this BSP and
> > this RTEMS version, they will always fail. This is not some sort of
> > random failure. When we change test states to expected failure I think
> > we should make sure that a ticket exists, which captures that there are
> > some test results which indicate issues (expected failure test state).
> > The ticket system is the better place to manage this. We should not use
> > the test states for this. The test states should be used to figure out
> > changes between different test runs. They should enable also to quickly
> > check if the outcome of a test run yields the expected results for a
> > certain RTEMS version and BSP.
>
> Thanks. It is clear to me we lack documentation on this topic and this
> is an oversight on my part which I will attempt to correct.
>
> I have reviewed Dejagnu and considered other things like the withdrawn
> IEEE 1003.3 standard and there are states we have that need to change
> but I think the original intent is the right path.
>
> The Dejagnu states are documented here:
>
>
> https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html#A-POSIX-Conforming-Test-Framework
>
> And the exit codes are:
>
> https://www.gnu.org/software/dejagnu/manual/Runtest.html#Runtest
>
> For me they define the goal and intent.
>
> The test states are metadata for the tester so it can determine the
> result of any given set of tests in relation to the expected state of
> the test when it was built. You need to detach yourself from being a
> developer and put yourself in the position of a tester who's task is to
> give an overall pass or fail for a specific build of RTEMS without
> needing to consider the specifics of any test, bug or feature.
>
> The primary requirement is to allow machine check of the results to
> determine regressions. A regression is a failure, pass or unresolved
> result that was not expected.
>
> My current thinking for test states are:
>
> PASS:
> The test has succeeded and passed without a failure.
>
> UNEXCEPTED-PASS:
> The test has succeeded when it was expected to fail.
>
> FAIL:
> The test has not succeeded and has failed when it was expected to pass.
> The failure can be a failed assert, unhandled exception, resource
> constraint, or a faulty test.
>
> EXCEPTED-FAIL:
> The test has not succeeded and has failed and this is expected.
>
> UNRESOLVED:
> The test has not completed and the result cannot be determined. The
> result can be unresolved because the test did not start or end, test
> harness failure, insufficient computing resources for the test harness
> to function correctly.
>
> EXCEPTED-UNRESOLVED:
> The test has not completed and the result cannot be determined and this
> is expected.
>
> INDETERMINATE:
> The test has succeeded, has failed or in unresolved. The test is an edge
> case where the test can pass, can fail, can be unresolved and this is
> expected.
>
> USER-INPUT:
> The test has not completed and the result is unresolved because it
> requires user intervention that cannot be provided.
>
> BENCHMARK:
> The test performs a performance type test. These are currently not
> supported.
>
> UNTESTED:
> The test has not run and is a place holder for a real test that is not
> yet provided.
>
> UNSUPPORTED:
> The test is not supported for this build of RTEMS, BSP or architecture.
>
> Note:
>
> 1. Any expected failures, unresolved, or indeterminate test results are
> considered faults and require fixing.
>
> 2. The nature of a failure cannot be inferred from the t

Re: [PATCH 4/6] testsuite: Add expected-fail to psim

[Gedare Bloom]

On Tue, May 12, 2020 at 3:11 AM Chris Johns  wrote:
>
> On 12/5/20 5:15 pm, Sebastian Huber wrote:
> > Hello,
> >
> > On 09/05/2020 03:30, Gedare Bloom wrote:
> > Without these tests being tagged this way the user would have no
> > idea where the stand after a build and test run and that would mean
> > we would have to make sure a release has no failures. I consider
> > that as not practical or realistic.
>  Maybe we need another state, e.g. something-is-broken-please-fix-it.
> >>> I do not think so, it is implicit in the failure or the test is
> >>> broken. The only change is to add unexpected-pass, that will be on
> >>> master after the 5 branch.
> >>>
> >> I disagree with this in principle, and it should be reverted after we
> >> branch 5. It's fine for now to get the release state sync'd, but we
> >> should find a long-term solution that distinguishes the cases:
> >> 1. we don't expect this test to pass on this bsp
> >> 2. we expect this test to pass, but know it doesn't currently
> >>
> >> They are two very different things, and I don't like conflating them
> >> into one "expected-fail" case
> > originally, I had the same point of view. What I didn't take into
> > account was the perspective of the tester. Now, I think it is perfectly
> > fine to flag these tests as expected failure test states. Because right
> > now, due to some known bugs such as https://devel.rtems.org/ticket/3982
> > and probably also some more issues, these tests fail. On this BSP and
> > this RTEMS version, they will always fail. This is not some sort of
> > random failure. When we change test states to expected failure I think
> > we should make sure that a ticket exists, which captures that there are
> > some test results which indicate issues (expected failure test state).
> > The ticket system is the better place to manage this. We should not use
> > the test states for this. The test states should be used to figure out
> > changes between different test runs. They should enable also to quickly
> > check if the outcome of a test run yields the expected results for a
> > certain RTEMS version and BSP.
>
> Thanks. It is clear to me we lack documentation on this topic and this
> is an oversight on my part which I will attempt to correct.
>
This makes enough sense to me.

> I have reviewed Dejagnu and considered other things like the withdrawn
> IEEE 1003.3 standard and there are states we have that need to change
> but I think the original intent is the right path.
>
> The Dejagnu states are documented here:
>
> https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html#A-POSIX-Conforming-Test-Framework
>
> And the exit codes are:
>
> https://www.gnu.org/software/dejagnu/manual/Runtest.html#Runtest
>
> For me they define the goal and intent.
>
> The test states are metadata for the tester so it can determine the
> result of any given set of tests in relation to the expected state of
> the test when it was built. You need to detach yourself from being a
> developer and put yourself in the position of a tester who's task is to
> give an overall pass or fail for a specific build of RTEMS without
> needing to consider the specifics of any test, bug or feature.
>
> The primary requirement is to allow machine check of the results to
> determine regressions. A regression is a failure, pass or unresolved
> result that was not expected.
>
> My current thinking for test states are:
>
> PASS:
> The test has succeeded and passed without a failure.
>
> UNEXCEPTED-PASS:
in case you copy-paste, there are a few of these typos for EXPECTED

> The test has succeeded when it was expected to fail.
>
> FAIL:
> The test has not succeeded and has failed when it was expected to pass.
> The failure can be a failed assert, unhandled exception, resource
> constraint, or a faulty test.
>
> EXCEPTED-FAIL:
> The test has not succeeded and has failed and this is expected.
>
> UNRESOLVED:
> The test has not completed and the result cannot be determined. The
> result can be unresolved because the test did not start or end, test
> harness failure, insufficient computing resources for the test harness
> to function correctly.
>
> EXCEPTED-UNRESOLVED:
> The test has not completed and the result cannot be determined and this
> is expected.
>
> INDETERMINATE:
> The test has succeeded, has failed or in unresolved. The test is an edge
> case where the test can pass, can fail, can be unresolved and this is
> expected.
>
> USER-INPUT:
> The test has not completed and the result is unresolved because it
> requires user intervention that cannot be provided.
>
This USER-INPUT could also be EXPECTED-UNRESOLVED?

> BENCHMARK:
> The test performs a performance type test. These are currently not
> supported.
>
> UNTESTED:
> The test has not run and is a place holder for a real test that is not
> yet provided.
>
> UNSUPPORTED:
> The test is not supported for this build of RTEMS, BSP or architecture.
>
> Note:
>
> 1. Any expected fail

Re: Memory Protection project interface details (GSoC 2020)

[Utkarsh Rai]

On Tue, May 12, 2020 at 9:27 AM Gedare Bloom  wrote:

> On Thu, May 7, 2020 at 9:59 PM Hesham Almatary
>  wrote:
> >
> > Hello Utkarsh,
> >
> > I'd suggest you don't spend too much efforts on setting up BBB
> > hardware if you haven't already. Debugging on QEMU with GDB is way
> > easier, and you can consider either qemu-xilinx-zynq-a9 or rpi2 BSPs.
> > Later, you can move your code to BBB if you want, since both are based
> > on ARMv7.
> +1
>
> Past work has also used psim successfully I thought? Or am I mistaken
> there.
>
> >
> > On Thu, 7 May 2020 at 18:26, Utkarsh Rai 
> wrote:
> > >
> > > Hello,
> > > This is to ensure that all the interested parties are on the same page
> before I start my project and can give their invaluable feedback.
> Excellent, thank you for getting the initiative.
>
> I'll be taking on the primary mentorship for your project, with
> support from the co-mentors (Peter, Hesham, Sebastian). For now, I
> prefer you to continue to make your presence on the mailing list. We
> will establish other forms of communication as needed and will take on
> IRC meetings once coding begins in earnest.
>
> > > My GSoC project, providing user-configurable thread stack protection,
> requires adding architecture-specific low-level support as well as
> high-level API support. I will be starting my project with ARMv7-A (on BBB)
> based MMU since RTEMS already has quite mature support for it. As already
> mentioned in my proposal I will be focusing more on the High-level
> interface and let it drive whatever further low-level support is needed.
> > > Once the application uses MMU for thread stack address generation each
> thread will be automatically protected as the page tables other than that
> of the executing thread would be made dormant. When the user has to share
> thread stacks they will have to obtain the stack attributes of the threads
> to be shared by pthread_attr_getstack() and then get a file descriptor of
> the memory to be mapped by a call to shm_open() and finally map this to the
> stack of the other thread through
> > > mmap(), this is the POSIX compliant way I could think of. Now at the
> low level, it means mapping the page table of the thread to be shared into
> the address space of the executing thread. This is an area where the
> low-level support has to be provided. At the high-level, this means
> providing support to mmap and shared-memory interface as mmap provides
> support for a file by simply
> > > copying the memory from the file to the destination. For shared memory
> objects it can
> > > provide read/write access but cannot provide restriction of write/read
> access. One of the areas that I have to look into more detail is thread
> context-switch, as after every context switch the TLBs need to be flushed
> and reinitialized lest we get an invalid address for the executing thread.
> Since context-switch is low-level architecture-specific, this also has to
> be provided with more support.
>
> This is really dense text. Try to break apart your writing a little
> bit to help clarify your thoughts.  You should also translate some of
> your proposal into a wiki page if you haven't started that yet, and a
> blog post. Both of those will help to focus your thoughts into words.
>
> "mapping the page table" is not meaningful to me. I think you mean
> something like "mapping a page from the page table"?

Will the design
> support sharing task stacks using MPUs with 4 regions? 8?  (It seems
> challenging to me, but might be possible in some limited
> configurations. Having support for those kinds of targets might still
> be useful, with the caveat that sharing stacks is not possible.)
>

I will have to look into this in a bit more detail before I can give you a
comprehensive answer.


> The first step is to get a BSP running that has sufficient
> capabilities for you to test out memory protection with. Do a little
> bit of digging, but definitely simulation is the way to go.
>

As suggested by Hesham, I have been able to run the qemu-Xilinx-zynq-a9 BSP
on qemu and I have learned how to debug it through GDB.
The BSP supports memory protection and as pointed out once I get it done on
this, I can move my code for other BSP with ARMv7 (RPI, BBB ).


> The second step from my perspective is to determine how to introduce
> strict isolation between task stacks. Don't worry about sharing at
> this stage, but rather can you completely isolate tasks? Then you can
> start to poke holes in the isolation.
>

 My understanding of this is that to completely isolate the tasks and page
tables of the task will be placed in two separate regions (user and system
mode respectively), the page tables will be accessed only by the kernel
code and not by the application. This will prevent dormant tasks from
interfering with the current task through the application code.

As you say, you'll also need to start to understand the context switch
> code. Start looking into it to determine where you might think to

Re: Memory Protection project interface details (GSoC 2020)

[Gedare Bloom]

On Tue, May 12, 2020 at 10:58 AM Utkarsh Rai  wrote:
>
>
>
> On Tue, May 12, 2020 at 9:27 AM Gedare Bloom  wrote:
>>
>> On Thu, May 7, 2020 at 9:59 PM Hesham Almatary
>>  wrote:
>> >
>> > Hello Utkarsh,
>> >
>> > I'd suggest you don't spend too much efforts on setting up BBB
>> > hardware if you haven't already. Debugging on QEMU with GDB is way
>> > easier, and you can consider either qemu-xilinx-zynq-a9 or rpi2 BSPs.
>> > Later, you can move your code to BBB if you want, since both are based
>> > on ARMv7.
>> +1
>>
>> Past work has also used psim successfully I thought? Or am I mistaken there.
>>
>> >
>> > On Thu, 7 May 2020 at 18:26, Utkarsh Rai  wrote:
>> > >
>> > > Hello,
>> > > This is to ensure that all the interested parties are on the same page 
>> > > before I start my project and can give their invaluable feedback.
>> Excellent, thank you for getting the initiative.
>>
>> I'll be taking on the primary mentorship for your project, with
>> support from the co-mentors (Peter, Hesham, Sebastian). For now, I
>> prefer you to continue to make your presence on the mailing list. We
>> will establish other forms of communication as needed and will take on
>> IRC meetings once coding begins in earnest.
>>
>> > > My GSoC project, providing user-configurable thread stack protection, 
>> > > requires adding architecture-specific low-level support as well as 
>> > > high-level API support. I will be starting my project with ARMv7-A (on 
>> > > BBB) based MMU since RTEMS already has quite mature support for it. As 
>> > > already mentioned in my proposal I will be focusing more on the 
>> > > High-level interface and let it drive whatever further low-level support 
>> > > is needed.
>> > > Once the application uses MMU for thread stack address generation each 
>> > > thread will be automatically protected as the page tables other than 
>> > > that of the executing thread would be made dormant. When the user has to 
>> > > share thread stacks they will have to obtain the stack attributes of the 
>> > > threads to be shared by pthread_attr_getstack() and then get a file 
>> > > descriptor of the memory to be mapped by a call to shm_open() and 
>> > > finally map this to the stack of the other thread through
>> > > mmap(), this is the POSIX compliant way I could think of. Now at the low 
>> > > level, it means mapping the page table of the thread to be shared into 
>> > > the address space of the executing thread. This is an area where the 
>> > > low-level support has to be provided. At the high-level, this means 
>> > > providing support to mmap and shared-memory interface as mmap provides 
>> > > support for a file by simply
>> > > copying the memory from the file to the destination. For shared memory 
>> > > objects it can
>> > > provide read/write access but cannot provide restriction of write/read 
>> > > access. One of the areas that I have to look into more detail is thread 
>> > > context-switch, as after every context switch the TLBs need to be 
>> > > flushed and reinitialized lest we get an invalid address for the 
>> > > executing thread. Since context-switch is low-level 
>> > > architecture-specific, this also has to be provided with more support.
>>
>> This is really dense text. Try to break apart your writing a little
>> bit to help clarify your thoughts.  You should also translate some of
>> your proposal into a wiki page if you haven't started that yet, and a
>> blog post. Both of those will help to focus your thoughts into words.
>>
>> "mapping the page table" is not meaningful to me. I think you mean
>> something like "mapping a page from the page table"?
>>
>> Will the design
>> support sharing task stacks using MPUs with 4 regions? 8?  (It seems
>> challenging to me, but might be possible in some limited
>> configurations. Having support for those kinds of targets might still
>> be useful, with the caveat that sharing stacks is not possible.)
>
>
> I will have to look into this in a bit more detail before I can give you a 
> comprehensive answer.
>
>>
>> The first step is to get a BSP running that has sufficient
>> capabilities for you to test out memory protection with. Do a little
>> bit of digging, but definitely simulation is the way to go.
>
>
> As suggested by Hesham, I have been able to run the qemu-Xilinx-zynq-a9 BSP 
> on qemu and I have learned how to debug it through GDB.
> The BSP supports memory protection and as pointed out once I get it done on 
> this, I can move my code for other BSP with ARMv7 (RPI, BBB ).
>
>>
>> The second step from my perspective is to determine how to introduce
>> strict isolation between task stacks. Don't worry about sharing at
>> this stage, but rather can you completely isolate tasks? Then you can
>> start to poke holes in the isolation.
>
>
>  My understanding of this is that to completely isolate the tasks and page 
> tables of the task will be placed in two separate regions (user and system 
> mode respectively), the page tables will be accessed

Re: Adding fenv support for ARM

[Eshan Dhawan]

Hello everyone,
After adding the files and compiling. I am
getting a few errors which I am not able to rectify.
If you could take a look and tell how to sort them.

link to newlib's git repo:
https://github.com/eshandhawan51/newlib-cygwin/tree/add_fenv_support

errors :

In file included from
/home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h:15:0,
 from
../../../../../../newlib-cygwin/newlib/libm/fenv/feclearexcept.c:57:
/home/eshan/development/newlib/c-arm-rtems5-newlib/arm-rtems5/thumb/newlib/targ-include/sys/fenv.h:82:1:
error: expected '=', ',', ';', 'asm' or '__attribute__' before 'extern'
 extern const fenv_t *_fe_dfl_env;
 ^~
In file included from
../../../../../../newlib-cygwin/newlib/libm/fenv/feclearexcept.c:57:0:
/home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h:22:1:
error: expected '=', ',', ';', 'asm' or '__attribute__' before 'int'
 int feclearexcept(int excepts);
 ^~~

thanks
-Eshan
On Sat, May 9, 2020 at 2:53 AM Joel Sherrill  wrote:

>
>
> On Thu, May 7, 2020 at 1:33 PM Eshan Dhawan 
> wrote:
>
>> Hello everyone,
>> This the tread to discuss adding fenv support for ARM architecture in
>> newlib.
>>
>> Link to previous thread:
>> https://lists.rtems.org/pipermail/devel/2020-March/058473.html
>>
>> ---FreeBSD Source
>> # https://github.com/freebsd/freebsd/tree/master/lib/msun/arm
>> # https://github.com/freebsd/freebsd/blob/master/sys/arm/include/ieeefp.h
>> ---NetBSD Source
>> # https://github.com/NetBSD/src/tree/trunk/lib/libm/arch/arm
>>
>> --- Musl has no implementation for fenv support for ARM  in it
>>
>> After going through the implementations both FreeBSD and NetBSD have a
>> stub implementation.
>> But then FreeBSD implementation seems to be more portable compared to
>> that in NetBSD.
>>
>> Should I start porting it from FreeBSD?
>> And which ARM BSP should I use for testing the changes?
>> Currently, I have xilinx_zynq_a9_qemu build
>>
>
> I think the FreeBSD implementation is the one to use. It looks to support
> hard and soft floating point implementations.
>
> I think the fenv.h file there will have to be worked on so some of the
> contents go into sys or machine/fenv.h. There is a portable fenv.h.
>
>
>>
>> There would also be a requirement to discuss where to place the files in
>> Newlib.
>>
>
> Not much to discuss. It should go into
> newlib-cygwin/newlib/libm/machine/arm.
>
>>
>> Thanks
>> -Eshan
>>
>>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Adding fenv support for ARM

[Gedare Bloom]

Take the first error reported, and show what is being compiled. You've
trimmed off too much preceding this to know what is being compiled and
how this might be influencing your error.

The specific error you're getting is often due to some syntax errors
preceding the line shown.

On Tue, May 12, 2020 at 2:59 PM Eshan Dhawan  wrote:
>
> Hello everyone,
> After adding the files and compiling. I am
> getting a few errors which I am not able to rectify.
> If you could take a look and tell how to sort them.
>
> link to newlib's git repo: 
> https://github.com/eshandhawan51/newlib-cygwin/tree/add_fenv_support
>
> errors :
>
> In file included from 
> /home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h:15:0,
>  from 
> ../../../../../../newlib-cygwin/newlib/libm/fenv/feclearexcept.c:57:
> /home/eshan/development/newlib/c-arm-rtems5-newlib/arm-rtems5/thumb/newlib/targ-include/sys/fenv.h:82:1:
>  error: expected '=', ',', ';', 'asm' or '__attribute__' before 'extern'
>  extern const fenv_t *_fe_dfl_env;
>  ^~
> In file included from 
> ../../../../../../newlib-cygwin/newlib/libm/fenv/feclearexcept.c:57:0:
> /home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h:22:1: 
> error: expected '=', ',', ';', 'asm' or '__attribute__' before 'int'
>  int feclearexcept(int excepts);
>  ^~~
>
> thanks
> -Eshan
> On Sat, May 9, 2020 at 2:53 AM Joel Sherrill  wrote:
>>
>>
>>
>> On Thu, May 7, 2020 at 1:33 PM Eshan Dhawan  wrote:
>>>
>>> Hello everyone,
>>> This the tread to discuss adding fenv support for ARM architecture in 
>>> newlib.
>>>
>>> Link to previous thread: 
>>> https://lists.rtems.org/pipermail/devel/2020-March/058473.html
>>>
>>> ---FreeBSD Source
>>> # https://github.com/freebsd/freebsd/tree/master/lib/msun/arm
>>> # https://github.com/freebsd/freebsd/blob/master/sys/arm/include/ieeefp.h
>>> ---NetBSD Source
>>> # https://github.com/NetBSD/src/tree/trunk/lib/libm/arch/arm
>>>
>>> --- Musl has no implementation for fenv support for ARM  in it
>>>
>>> After going through the implementations both FreeBSD and NetBSD have a stub 
>>> implementation.
>>> But then FreeBSD implementation seems to be more portable compared to that 
>>> in NetBSD.
>>>
>>> Should I start porting it from FreeBSD?
>>> And which ARM BSP should I use for testing the changes?
>>> Currently, I have xilinx_zynq_a9_qemu build
>>
>>
>> I think the FreeBSD implementation is the one to use. It looks to support 
>> hard and soft floating point implementations.
>>
>> I think the fenv.h file there will have to be worked on so some of the 
>> contents go into sys or machine/fenv.h. There is a portable fenv.h.
>>
>>>
>>>
>>> There would also be a requirement to discuss where to place the files in 
>>> Newlib.
>>
>>
>> Not much to discuss. It should go into newlib-cygwin/newlib/libm/machine/arm.
>>>
>>>
>>> Thanks
>>> -Eshan
>>>
> ___
> devel mailing list
> devel@rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Memory Protection project interface details (GSoC 2020)

[Hesham Almatary]

On Tue, 12 May 2020 at 04:57, Gedare Bloom  wrote:
>
> On Thu, May 7, 2020 at 9:59 PM Hesham Almatary
>  wrote:
> >
> > Hello Utkarsh,
> >
> > I'd suggest you don't spend too much efforts on setting up BBB
> > hardware if you haven't already. Debugging on QEMU with GDB is way
> > easier, and you can consider either qemu-xilinx-zynq-a9 or rpi2 BSPs.
> > Later, you can move your code to BBB if you want, since both are based
> > on ARMv7.
> +1
>
> Past work has also used psim successfully I thought? Or am I mistaken there.
>
Before my 2012 project (and part of it, yes), we used psim. The use of
software TLBs wasn't very ideal/easy though, so we moved to ARM in
2013. The development/testing was mainly on a RPi board. I don't
remember there was a QEMU model for it yet.


> >
> > On Thu, 7 May 2020 at 18:26, Utkarsh Rai  wrote:
> > >
> > > Hello,
> > > This is to ensure that all the interested parties are on the same page 
> > > before I start my project and can give their invaluable feedback.
> Excellent, thank you for getting the initiative.
>
> I'll be taking on the primary mentorship for your project, with
> support from the co-mentors (Peter, Hesham, Sebastian). For now, I
> prefer you to continue to make your presence on the mailing list. We
> will establish other forms of communication as needed and will take on
> IRC meetings once coding begins in earnest.
>
> > > My GSoC project, providing user-configurable thread stack protection, 
> > > requires adding architecture-specific low-level support as well as 
> > > high-level API support. I will be starting my project with ARMv7-A (on 
> > > BBB) based MMU since RTEMS already has quite mature support for it. As 
> > > already mentioned in my proposal I will be focusing more on the 
> > > High-level interface and let it drive whatever further low-level support 
> > > is needed.
> > > Once the application uses MMU for thread stack address generation each 
> > > thread will be automatically protected as the page tables other than that 
> > > of the executing thread would be made dormant. When the user has to share 
> > > thread stacks they will have to obtain the stack attributes of the 
> > > threads to be shared by pthread_attr_getstack() and then get a file 
> > > descriptor of the memory to be mapped by a call to shm_open() and finally 
> > > map this to the stack of the other thread through
> > > mmap(), this is the POSIX compliant way I could think of. Now at the low 
> > > level, it means mapping the page table of the thread to be shared into 
> > > the address space of the executing thread. This is an area where the 
> > > low-level support has to be provided. At the high-level, this means 
> > > providing support to mmap and shared-memory interface as mmap provides 
> > > support for a file by simply
> > > copying the memory from the file to the destination. For shared memory 
> > > objects it can
> > > provide read/write access but cannot provide restriction of write/read 
> > > access. One of the areas that I have to look into more detail is thread 
> > > context-switch, as after every context switch the TLBs need to be flushed 
> > > and reinitialized lest we get an invalid address for the executing 
> > > thread. Since context-switch is low-level architecture-specific, this 
> > > also has to be provided with more support.
>
> This is really dense text. Try to break apart your writing a little
> bit to help clarify your thoughts.  You should also translate some of
> your proposal into a wiki page if you haven't started that yet, and a
> blog post. Both of those will help to focus your thoughts into words.
>
> "mapping the page table" is not meaningful to me. I think you mean
> something like "mapping a page from the page table"?  Will the design
> support sharing task stacks using MPUs with 4 regions? 8?  (It seems
> challenging to me, but might be possible in some limited
> configurations. Having support for those kinds of targets might still
> be useful, with the caveat that sharing stacks is not possible.)
>
> The first step is to get a BSP running that has sufficient
> capabilities for you to test out memory protection with. Do a little
> bit of digging, but definitely simulation is the way to go.
>
> The second step from my perspective is to determine how to introduce
> strict isolation between task stacks. Don't worry about sharing at
> this stage, but rather can you completely isolate tasks? Then you can
> start to poke holes in the isolation.
>
> As you say, you'll also need to start to understand the context switch
> code. Start looking into it to determine where you might think to
> implement changing the address space of the executing thread. Another
> challenge is that RTEMS can dispatch to a new task from the interrupt
> handler, which may cause some problems for you as well to handle.
>
> Have you figured out where in the code thread stacks are allocated?
> How do you plan to make the thread stacks known to other threads?
>
> TLB sho

Re: Adding fenv support for ARM

[Joel Sherrill]

Gedare is right but one more piece of information is needed. When newlib is
being built, it lays target dependent files on top of generic ones. So the
default sys/fenv.h is replaced with something like
libc/machine/arm/sys/fenv.h. Make sure that
/home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h is
in fact the file you think it should be.

Also look at what is supposed to be in the generic fenv.h and .
You will probably have to tweak it some.

Check out what is in x86_64/fenv.h and riscv/fenv.h with respect to the
default environment variable. It looks to be declared differently in the
other ports so you may have to make it match.

But more output does help.

--joel



On Tue, May 12, 2020 at 4:53 PM Gedare Bloom  wrote:

> Take the first error reported, and show what is being compiled. You've
> trimmed off too much preceding this to know what is being compiled and
> how this might be influencing your error.
>
> The specific error you're getting is often due to some syntax errors
> preceding the line shown.
>
> On Tue, May 12, 2020 at 2:59 PM Eshan Dhawan 
> wrote:
> >
> > Hello everyone,
> > After adding the files and compiling. I am
> > getting a few errors which I am not able to rectify.
> > If you could take a look and tell how to sort them.
> >
> > link to newlib's git repo:
> https://github.com/eshandhawan51/newlib-cygwin/tree/add_fenv_support
> >
> > errors :
> >
> > In file included from
> /home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h:15:0,
> >  from
> ../../../../../../newlib-cygwin/newlib/libm/fenv/feclearexcept.c:57:
> >
> /home/eshan/development/newlib/c-arm-rtems5-newlib/arm-rtems5/thumb/newlib/targ-include/sys/fenv.h:82:1:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before 'extern'
> >  extern const fenv_t *_fe_dfl_env;
> >  ^~
> > In file included from
> ../../../../../../newlib-cygwin/newlib/libm/fenv/feclearexcept.c:57:0:
> >
> /home/eshan/development/newlib/newlib-cygwin/newlib/libc/include/fenv.h:22:1:
> error: expected '=', ',', ';', 'asm' or '__attribute__' before 'int'
> >  int feclearexcept(int excepts);
> >  ^~~
> >
> > thanks
> > -Eshan
> > On Sat, May 9, 2020 at 2:53 AM Joel Sherrill  wrote:
> >>
> >>
> >>
> >> On Thu, May 7, 2020 at 1:33 PM Eshan Dhawan 
> wrote:
> >>>
> >>> Hello everyone,
> >>> This the tread to discuss adding fenv support for ARM architecture in
> newlib.
> >>>
> >>> Link to previous thread:
> https://lists.rtems.org/pipermail/devel/2020-March/058473.html
> >>>
> >>> ---FreeBSD Source
> >>> # https://github.com/freebsd/freebsd/tree/master/lib/msun/arm
> >>> #
> https://github.com/freebsd/freebsd/blob/master/sys/arm/include/ieeefp.h
> >>> ---NetBSD Source
> >>> # https://github.com/NetBSD/src/tree/trunk/lib/libm/arch/arm
> >>>
> >>> --- Musl has no implementation for fenv support for ARM  in it
> >>>
> >>> After going through the implementations both FreeBSD and NetBSD have a
> stub implementation.
> >>> But then FreeBSD implementation seems to be more portable compared to
> that in NetBSD.
> >>>
> >>> Should I start porting it from FreeBSD?
> >>> And which ARM BSP should I use for testing the changes?
> >>> Currently, I have xilinx_zynq_a9_qemu build
> >>
> >>
> >> I think the FreeBSD implementation is the one to use. It looks to
> support hard and soft floating point implementations.
> >>
> >> I think the fenv.h file there will have to be worked on so some of the
> contents go into sys or machine/fenv.h. There is a portable fenv.h.
> >>
> >>>
> >>>
> >>> There would also be a requirement to discuss where to place the files
> in Newlib.
> >>
> >>
> >> Not much to discuss. It should go into
> newlib-cygwin/newlib/libm/machine/arm.
> >>>
> >>>
> >>> Thanks
> >>> -Eshan
> >>>
> > ___
> > devel mailing list
> > devel@rtems.org
> > http://lists.rtems.org/mailman/listinfo/devel
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Memory Protection project interface details (GSoC 2020)

[Joel Sherrill]

I hate to top post but I'm not sure where to insert this.

As a first step, I would ensure that paging catches stack overflow errors.
Then move on to an optional capability where threads cannot access the
stacks of other threads. POSIX does not say anything about whether that
should work or not but there are cases (especially in RTEMS) where if a
blocking thread has a message queue buffer on a stack and another thread
does a send, then it will write to another thread's stack. This is accepted
programming practice.

I'm not opposed to an option where per-thread stack protection is
available. But making it mandatory is a bad step. Using the MMU to detect
stack overflow is good.

As Hesham mentioned, this is hard on some architectures if you don't have a
nice page management system. Can you make sure the minimum processor
architectural requirements are documented. Not just in an email. This will
ultimately be information in the CPU Supplement.

--joel

On Tue, May 12, 2020 at 5:02 PM Hesham Almatary <
hesham.almat...@cl.cam.ac.uk> wrote:

> On Tue, 12 May 2020 at 04:57, Gedare Bloom  wrote:
> >
> > On Thu, May 7, 2020 at 9:59 PM Hesham Almatary
> >  wrote:
> > >
> > > Hello Utkarsh,
> > >
> > > I'd suggest you don't spend too much efforts on setting up BBB
> > > hardware if you haven't already. Debugging on QEMU with GDB is way
> > > easier, and you can consider either qemu-xilinx-zynq-a9 or rpi2 BSPs.
> > > Later, you can move your code to BBB if you want, since both are based
> > > on ARMv7.
> > +1
> >
> > Past work has also used psim successfully I thought? Or am I mistaken
> there.
> >
> Before my 2012 project (and part of it, yes), we used psim. The use of
> software TLBs wasn't very ideal/easy though, so we moved to ARM in
> 2013. The development/testing was mainly on a RPi board. I don't
> remember there was a QEMU model for it yet.
>
>
> > >
> > > On Thu, 7 May 2020 at 18:26, Utkarsh Rai 
> wrote:
> > > >
> > > > Hello,
> > > > This is to ensure that all the interested parties are on the same
> page before I start my project and can give their invaluable feedback.
> > Excellent, thank you for getting the initiative.
> >
> > I'll be taking on the primary mentorship for your project, with
> > support from the co-mentors (Peter, Hesham, Sebastian). For now, I
> > prefer you to continue to make your presence on the mailing list. We
> > will establish other forms of communication as needed and will take on
> > IRC meetings once coding begins in earnest.
> >
> > > > My GSoC project, providing user-configurable thread stack
> protection, requires adding architecture-specific low-level support as well
> as high-level API support. I will be starting my project with ARMv7-A (on
> BBB) based MMU since RTEMS already has quite mature support for it. As
> already mentioned in my proposal I will be focusing more on the High-level
> interface and let it drive whatever further low-level support is needed.
> > > > Once the application uses MMU for thread stack address generation
> each thread will be automatically protected as the page tables other than
> that of the executing thread would be made dormant. When the user has to
> share thread stacks they will have to obtain the stack attributes of the
> threads to be shared by pthread_attr_getstack() and then get a file
> descriptor of the memory to be mapped by a call to shm_open() and finally
> map this to the stack of the other thread through
> > > > mmap(), this is the POSIX compliant way I could think of. Now at the
> low level, it means mapping the page table of the thread to be shared into
> the address space of the executing thread. This is an area where the
> low-level support has to be provided. At the high-level, this means
> providing support to mmap and shared-memory interface as mmap provides
> support for a file by simply
> > > > copying the memory from the file to the destination. For shared
> memory objects it can
> > > > provide read/write access but cannot provide restriction of
> write/read access. One of the areas that I have to look into more detail is
> thread context-switch, as after every context switch the TLBs need to be
> flushed and reinitialized lest we get an invalid address for the executing
> thread. Since context-switch is low-level architecture-specific, this also
> has to be provided with more support.
> >
> > This is really dense text. Try to break apart your writing a little
> > bit to help clarify your thoughts.  You should also translate some of
> > your proposal into a wiki page if you haven't started that yet, and a
> > blog post. Both of those will help to focus your thoughts into words.
> >
> > "mapping the page table" is not meaningful to me. I think you mean
> > something like "mapping a page from the page table"?  Will the design
> > support sharing task stacks using MPUs with 4 regions? 8?  (It seems
> > challenging to me, but might be possible in some limited
> > configurations. Having support for those kinds of targets m

Re: [PATCH 4/6] testsuite: Add expected-fail to psim

[Chris Johns]

On 13/5/20 12:11 am, Gedare Bloom wrote:

On Tue, May 12, 2020 at 3:11 AM Chris Johns  wrote:


On 12/5/20 5:15 pm, Sebastian Huber wrote:

Hello,

On 09/05/2020 03:30, Gedare Bloom wrote:

Without these tests being tagged this way the user would have no
idea where the stand after a build and test run and that would mean
we would have to make sure a release has no failures. I consider
that as not practical or realistic.

Maybe we need another state, e.g. something-is-broken-please-fix-it.

I do not think so, it is implicit in the failure or the test is
broken. The only change is to add unexpected-pass, that will be on
master after the 5 branch.


I disagree with this in principle, and it should be reverted after we
branch 5. It's fine for now to get the release state sync'd, but we
should find a long-term solution that distinguishes the cases:
1. we don't expect this test to pass on this bsp
2. we expect this test to pass, but know it doesn't currently

They are two very different things, and I don't like conflating them
into one "expected-fail" case

originally, I had the same point of view. What I didn't take into
account was the perspective of the tester. Now, I think it is perfectly
fine to flag these tests as expected failure test states. Because right
now, due to some known bugs such as https://devel.rtems.org/ticket/3982
and probably also some more issues, these tests fail. On this BSP and
this RTEMS version, they will always fail. This is not some sort of
random failure. When we change test states to expected failure I think
we should make sure that a ticket exists, which captures that there are
some test results which indicate issues (expected failure test state).
The ticket system is the better place to manage this. We should not use
the test states for this. The test states should be used to figure out
changes between different test runs. They should enable also to quickly
check if the outcome of a test run yields the expected results for a
certain RTEMS version and BSP.


Thanks. It is clear to me we lack documentation on this topic and this
is an oversight on my part which I will attempt to correct.


This makes enough sense to me.


I have reviewed Dejagnu and considered other things like the withdrawn
IEEE 1003.3 standard and there are states we have that need to change
but I think the original intent is the right path.

The Dejagnu states are documented here:

https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html#A-POSIX-Conforming-Test-Framework

And the exit codes are:

https://www.gnu.org/software/dejagnu/manual/Runtest.html#Runtest

For me they define the goal and intent.

The test states are metadata for the tester so it can determine the
result of any given set of tests in relation to the expected state of
the test when it was built. You need to detach yourself from being a
developer and put yourself in the position of a tester who's task is to
give an overall pass or fail for a specific build of RTEMS without
needing to consider the specifics of any test, bug or feature.

The primary requirement is to allow machine check of the results to
determine regressions. A regression is a failure, pass or unresolved
result that was not expected.

My current thinking for test states are:

PASS:
The test has succeeded and passed without a failure.

UNEXCEPTED-PASS:

in case you copy-paste, there are a few of these typos for EXPECTED


Brain to fingers wiring issue ... I will sort that out.




The test has succeeded when it was expected to fail.

FAIL:
The test has not succeeded and has failed when it was expected to pass.
The failure can be a failed assert, unhandled exception, resource
constraint, or a faulty test.

EXCEPTED-FAIL:
The test has not succeeded and has failed and this is expected.

UNRESOLVED:
The test has not completed and the result cannot be determined. The
result can be unresolved because the test did not start or end, test
harness failure, insufficient computing resources for the test harness
to function correctly.

EXCEPTED-UNRESOLVED:
The test has not completed and the result cannot be determined and this
is expected.

INDETERMINATE:
The test has succeeded, has failed or in unresolved. The test is an edge
case where the test can pass, can fail, can be unresolved and this is
expected.

USER-INPUT:
The test has not completed and the result is unresolved because it
requires user intervention that cannot be provided.


This USER-INPUT could also be EXPECTED-UNRESOLVED?


Yes. These are states and what we get rtems-test to report is something 
we can work on. For example I think we present the passes, fails and 
unresolved as gross values. The means user-input is counted in the gross 
unresolved count because it is unresolved. Under unresolved we can 
detail its composition and the net unsolved value should be 0 for tier 1 
or specific BSPs.



BENCHMARK:
The test performs a performance type test. These are currently not
supported.

UNTESTED:
The te

Re: [PATCH 4/6] testsuite: Add expected-fail to psim

[Chris Johns]

On 12/5/20 11:22 pm, Joel Sherrill wrote:
On Tue, May 12, 2020 at 4:11 AM Chris Johns > wrote:


On 12/5/20 5:15 pm, Sebastian Huber wrote:
 > Hello,
 >
 > On 09/05/2020 03:30, Gedare Bloom wrote:
 > Without these tests being tagged this way the user would have no
 > idea where the stand after a build and test run and that
would mean
 > we would have to make sure a release has no failures. I consider
 > that as not practical or realistic.
  Maybe we need another state, e.g.
something-is-broken-please-fix-it.
 >>> I do not think so, it is implicit in the failure or the test is
 >>> broken. The only change is to add unexpected-pass, that will be on
 >>> master after the 5 branch.
 >>>
 >> I disagree with this in principle, and it should be reverted
after we
 >> branch 5. It's fine for now to get the release state sync'd, but we
 >> should find a long-term solution that distinguishes the cases:
 >> 1. we don't expect this test to pass on this bsp
 >> 2. we expect this test to pass, but know it doesn't currently
 >>
 >> They are two very different things, and I don't like conflating them
 >> into one "expected-fail" case
 > originally, I had the same point of view. What I didn't take into
 > account was the perspective of the tester. Now, I think it is
perfectly
 > fine to flag these tests as expected failure test states. Because
right
 > now, due to some known bugs such as
https://devel.rtems.org/ticket/3982
 > and probably also some more issues, these tests fail. On this BSP
and
 > this RTEMS version, they will always fail. This is not some sort of
 > random failure. When we change test states to expected failure I
think
 > we should make sure that a ticket exists, which captures that
there are
 > some test results which indicate issues (expected failure test
state).
 > The ticket system is the better place to manage this. We should
not use
 > the test states for this. The test states should be used to
figure out
 > changes between different test runs. They should enable also to
quickly
 > check if the outcome of a test run yields the expected results for a
 > certain RTEMS version and BSP.

Thanks. It is clear to me we lack documentation on this topic and this
is an oversight on my part which I will attempt to correct.

I have reviewed Dejagnu and considered other things like the withdrawn
IEEE 1003.3 standard and there are states we have that need to change
but I think the original intent is the right path.

The Dejagnu states are documented here:


https://www.gnu.org/software/dejagnu/manual/A-POSIX-Conforming-Test-Framework.html#A-POSIX-Conforming-Test-Framework

And the exit codes are:

https://www.gnu.org/software/dejagnu/manual/Runtest.html#Runtest

For me they define the goal and intent.

The test states are metadata for the tester so it can determine the
result of any given set of tests in relation to the expected state of
the test when it was built. You need to detach yourself from being a
developer and put yourself in the position of a tester who's task is to
give an overall pass or fail for a specific build of RTEMS without
needing to consider the specifics of any test, bug or feature.

The primary requirement is to allow machine check of the results to
determine regressions. A regression is a failure, pass or unresolved
result that was not expected.

My current thinking for test states are:

PASS:
The test has succeeded and passed without a failure.

UNEXCEPTED-PASS:
The test has succeeded when it was expected to fail.

FAIL:
The test has not succeeded and has failed when it was expected to pass.
The failure can be a failed assert, unhandled exception, resource
constraint, or a faulty test.

EXCEPTED-FAIL:
The test has not succeeded and has failed and this is expected.

UNRESOLVED:
The test has not completed and the result cannot be determined. The
result can be unresolved because the test did not start or end, test
harness failure, insufficient computing resources for the test harness
to function correctly.

EXCEPTED-UNRESOLVED:
The test has not completed and the result cannot be determined and this
is expected.

INDETERMINATE:
The test has succeeded, has failed or in unresolved. The test is an
edge
case where the test can pass, can fail, can be unresolved and this is
expected.

USER-INPUT:
The test has not completed and the result is unresolved because it
requires user intervention that cannot be provided.

BENCHMARK:
The test performs a performance type test. These are currently not
supported.

UNTESTED:
The test has not run and is a place holder for a real t