On 15.06.23 15:51, Joel Sherrill wrote:
On Mon, Jun 12, 2023 at 10:58 AM Gedare Bloom <ged...@rtems.org
<mailto:ged...@rtems.org>> wrote:
On Mon, Jun 12, 2023 at 12:56 AM Sebastian Huber
<sebastian.hu...@embedded-brains.de
<mailto:sebastian.hu...@embedded-brains.de>> wrote:
>
>
>
> On 01.06.23 22:06, Gedare Bloom wrote:
> > On Thu, Jun 1, 2023 at 2:00 PM Gedare Bloom<ged...@rtems.org
<mailto:ged...@rtems.org>> wrote:
> >> ---------- Forwarded message ---------
> >> From: Sebastian Huber<sebastian.hu...@embedded-brains.de
<mailto:sebastian.hu...@embedded-brains.de>>
> >> Date: Wed, May 31, 2023 at 10:31 AM
> >>
> >> The existing tests in the RTEMS test suite are basically BSP
> >> independent. This patch set introduces BSP-specific validation
tests.
> >> These tests are disabled for other BSPs through the build
system, for
> >> example:
> >>
> >> spec/build/testsuites/validation/bsp-sparc-leon3-gr712rc.yml
> >> [...]
> >> cxxflags: []
> >> enabled-by: sparc/gr712rc
> >> features: c cprogram
> >> [...]
> > The use of the enabled-by: field to control the BSP-specific tests
> > looks reasonable. However, we should decide where/how any
BSP-specific
> > tests should reside. It looks to me like the current approach is to
> > dump all test files in a single monolithic 'validation'
directory, and
> > let the user/script sort it out. This results in a mix of tests
> > intended for all targets, and some for specific targets. This is
> > pretty much non-maintainable from my point-of-view without some
> > additional tool support. Correct me if I'm wrong.
>
> Yes, this is the current approach. There is no strict one-to-one
> relationship of test cases and test suites. The file names are
somewhat
> descriptive, for example:
>
> ls -l *leon3*
> -rw-r--r-- 1 sebastian_h domain users 3739 31. Mai 10:44
> tc-bsp-sparc-leon3-gr712rc.c
> -rw-r--r-- 1 sebastian_h domain users 6138 31. Mai 10:44
> tc-fatal-bsp-sparc-leon3-shutdown.c
> -rw-r--r-- 1 sebastian_h domain users 5135 31. Mai 10:44
> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> -rw-r--r-- 1 sebastian_h domain users 2723 31. Mai 10:44
> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.h
> -rw-r--r-- 1 sebastian_h domain users 5406 31. Mai 10:44
> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> -rw-r--r-- 1 sebastian_h domain users 2753 31. Mai 10:44
> tr-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.h
> -rw-r--r-- 1 sebastian_h domain users 5270 31. Mai 10:44
> tr-fatal-bsp-sparc-leon3-clock-initialization.c
> -rw-r--r-- 1 sebastian_h domain users 2681 31. Mai 10:44
> tr-fatal-bsp-sparc-leon3-clock-initialization.h
> -rw-r--r-- 1 sebastian_h domain users 2488 31. Mai 10:44
> ts-bsp-sparc-leon3-gr712rc.c
> -rw-r--r-- 1 sebastian_h domain users 2845 31. Mai 10:44
> ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-boot.c
> -rw-r--r-- 1 sebastian_h domain users 2919 31. Mai 10:44
> ts-fatal-bsp-sparc-leon3-cache-snooping-disabled-secondary.c
> -rw-r--r-- 1 sebastian_h domain users 2797 31. Mai 10:44
> ts-fatal-bsp-sparc-leon3-clock-initialization.c
> -rw-r--r-- 1 sebastian_h domain users 3148 31. Mai 10:45
> ts-fatal-bsp-sparc-leon3-shutdown-response.c
> -rw-r--r-- 1 sebastian_h domain users 4909 31. Mai 10:44
> ts-fatal-bsp-sparc-leon3-shutdown.c
>
> We could also introduce subdirectories to organize things. The test
> framework prints out the file name in messages, so it would be
nice if
> they remain unique. With subdirectories this would lead to longer
path
> names, for example
>
> testsuites/validation/sparc/leon3/tc-bsp-sparc-leon3-gr712rc.c
>
I see. Maybe it makes sense to have all BSP-specific tests in a bsps
subdirectory, with the unique names encoded to ensure the arch/bsp
combination appears in the filename, such as:
testsuites/validation/bsps/tc-bsp-sparc-leon3-gr712rc.c
If so, 'bsp-' can probably be omitted from the filename. This way,
architecture-specific testing may also be easily possible, like
tc-sparc-something.c
We also have to account for device driver/class type tests which
are not BSP specific but only work if a BSP supports a specific feature.
For example, Alex wrote some GPIO tests for the Microblaze work. Assuming
these were based on a common GPIO API, they would work on any BSP
which supported that API.
Similarly for other optional devices where the drivers are in rtems.git
The enabled-by expression is quite flexible. For the placement, if it is
BSP-specific just place it in testsuites/validation/bsps or
testsuites/unit/bsps.
> >
> > I would like to discuss possible ways to manage the integration of
> > tests that are conditionally-built based on the arch/bsp tuple. We
> > should have clear guidance for others who want to add such tests in
> > the future, or who would modify existing tests.
>
> Yes, this makes sense. We could add a new section for
BSP-specific tests to:
>
> https://docs.rtems.org/branches/master/eng/req/howto.html
<https://docs.rtems.org/branches/master/eng/req/howto.html>
>
> For a pre-qualified BSP you have to specify the fatal errors and
write
> validation tests for it. Other BSP-specific specification and
validation
> may be necessary for the kernel IO device driver, cache controller
> support, memory management unit initialization, memory protection
unit
> initialization, etc.
>
Good, that would definitely be important to document.
I think we should have some of the BSP-specific tests under some other
location in testsuites, while others would be under validation where
they are used for pre-qualification?
Maybe it is sensible to introduce testsuites/bsps/ also.
I'm not entirely clear about the difference between 'validation' and
any other kind of tests, such as where specific tests should be
located.
> Even for normal BSPs, some unit tests would be helpful. For
example, I
> had to fix _AArch32_PMSA_Initialize() twice:
>
>
https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c
<https://github.com/RTEMS/rtems/commits/master/cpukit/score/cpu/arm/aarch32-psma-init.c>
>
> Unit tests would have probably found the errors before
applications hit
> the special cases.
>
Something like testsuites/bsps/arm-aarch32-psma-init.c (or whatever
file naming convention we like to decide on. I forget the meaning of
tc/tr/ts/tx but recall they are about specific kinds of testing for
validation.)
I was wondering it we more of less mirrored the bsps/ structure
shared/
ARCH/shared
ARCH/BSP
We have solved this problem elsewhere. We don't have to encode it in the
file names.
The test framework prints out the location of events using the file name
and the line number. So, it would be good if the file names are unique
and descriptive. This is why I like Gedare's proposal to just use "bsps".
--
embedded brains GmbH
Herr Sebastian HUBER
Dornierstr. 4
82178 Puchheim
Germany
email: sebastian.hu...@embedded-brains.de
phone: +49-89-18 94 741 - 16
fax: +49-89-18 94 741 - 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
