Hello Chris,
thanks for your comments. I converted now all tests except the Ada
tests, tests which use the pax tool, and the dl* tests. For an approach
to address the test states please see below.
On 11/11/2019 23:25, Chris Johns wrote:
On 11/11/19 7:13 pm, Sebastian Huber wrote:
what is the purpose of the *.tcfg test configuration files?
The tcfg files provide a way to implement the "test controls" ...
https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls
.. as well as excluding a test. A test executable that is not excluded must
build and be loadable on a BSP target and executed.
There are some *norun* tests.
Tests can be excluded due to
an architecture issue (lacking an architecture feature), BSP issue (not enough
memory) or an RTEMS or tools support related issue (missing feature). A build
test run on a BSP target uses the "test banners" to publish it's state ...
https://docs.rtems.org/branches/master/user/testing/tests.html#
The existing implementation is a mix of build system controls to exclude a test
and command line defines to control the build in a standard and controlled way.
The rtems-test tool understands the test banners and adapts it's control based
on what the test reports. For example a `user-input` test is terminated and the
next test is run.
Ok, it seems a test program has exactly one state for a given BSP.
You can disable compilation of test programs with them, e.g.
exclude: flashdisk01
Yes. There is also `rexclude` a regex filter, for example `rexclude:
dl[0-9][0-9]`.
The "testsuites/rtems-test-check.py" contains a bit more stuff. What is really
needed?
I suspect you will end up needing to match the functionality provided. What
rtems-test-check.py supports has grown since I introduced this support. It
started off as a simple script however it did not do what we needed and adding
features cause the performance to blow out so I switch to python and since then
more complexity has been added. The script is currently run once for each test,
that can be avoided with waf.
I don't want to use this script in the new build system. I want to map
the *.tcfg functionality to the build specification and use the means
that are already there as far as possible. I would like to avoid too
many special cases.
I have to map this to the new build system.
Yes you will. The mapping of the tcfg files is more complex than it may appear.
The complexity is in what we need to represent and not in the implementation. In
a waf build system the configure phase could be a single pass where the tcfg
files are loaded in a single pass and then held in memory while the build
instance for each test is created.
A quick summary:
1. includes
2. excludes and include with regular expression filtering, ie exclude a bunch
then include something specific
3. test controls
Include support allows a single definition of a test profile and reuse across a
number of BSPs. The use of regex filters allow for subtle variations of how this
works.
In theory, we could support regular expressions in the "enabled-by"
attribute. Is this really necessary? It shouldn't be an issue to
enumerate all tests that belong to a certain group.
$ grep --include='*.tcfg' rexclude -r .
./testsuites/testdata/disable-libdl-tests.tcfg:rexclude: dl[0-9][0-9]
It is only used for the dl* tests and it would be simple to just give a
list of tests.
One approach is to use the "enabled-by" attribute to disable test programs on
demand, e.g.:
$ cat spec/build/testsuites/samples/RTEMS-BUILD-TEST-SAMPLE-TICKER.yml
Hmm, I had not noticed .yml is being used. By default emacs does not know this
is a YAML format file. Why has this been selected over .yaml? Having to teach
editors about a new extension is painful.
Doorstop uses this extension. We would have to adjust doorstop to use
another extension.
active: true
build-type: test-program
cflags: []
cppflags: []
cxxflags: []
derived: false
enabled-by:
- not: DISABLE_TEST_SAMPLE_TICKER
features: c cprogram
header: ''
includes: []
level: 1.15
links: []
normative: true
order: 0
ref: ''
reviewed: null
source:
- testsuites/samples/ticker/init.c
- testsuites/samples/ticker/tasks.c
stlib: []
target: testsuites/samples/ticker.exe
text: ''
type: build
use: []
A BSP can use a "build-type: option" file to disable tests, e.g.
actions:
- disable-tests:
- SAMPLE/TICKER
- LIB/FLASHDISK01
I would be careful before heading down this path. Lets assume 150 BSPs and 570
tests, that is 85,500 tests and test states. Adding a modest number of test
cases could effect the readability of these files. I think you will need to
structure this rather than having it expressed as a flat list.
You will have a flat list per BSP (with includes via item "links") just
as you have it now with the *.tcfg files.
There are 77 tcfg files. There are top level definitions that are used by BSPs:
testsuites/testdata/disable-intrcritical-tests.tcfg
testsuites/testdata/disable-libdl-tests.tcfg
testsuites/testdata/disable-mrfs-tests.tcfg
testsuites/testdata/rtems.tcfg
testsuites/testdata/small-memory-testsuite.tcfg
testsuites/testdata/require-tick-isr.tcfg
testsuites/testdata/disable-jffs2-tests.tcfg
testsuites/testdata/disable-iconv-tests.tcfg
You can do includes through the "links" attribute. The number of test
state specification items will be roughly the same as the *.tcfg file count.
The rtems.tcfg defines the state of tests common to all BSPs, for example
`user-input` and `benchmark`. These are invariant. Others like
small-memory-testsuite.tcfg are more complex including other test profiles like
disable-iconv-tests.tcfg etc.
I can see us having test profiles based on an architecture or BSP family before
you reach the BSP. I have not checked to see if this is already happening.
If you could structure the dependency in away that reflected what there is it
would help. For example tms570ls3137_hdk_with_loader-testsuite.tcfg includes
small-memory-testsuite.tcfg plus it excludes linpack. If that BSPs spec was just
`small-memory` and `exclude: linpack` it would be readable and maintainable.
I understand wanting to use the spec support however I do not know enough about
it to offer a specific solution.
The "links" attribute is a very important thing in the specification
item business. I guess I have to add some dot images to the software
requirements engineering chapter.
An open issue is how and if we should set other states, e.g. expected to fail.
We could generate a "bsp/teststate.h" file for each BSP which optionally defines
for each test a state other than "pass", e.g.
#define TEST_SAMPLE_TICKER_STATE RTEMS_TEST_STATE_FAIL
Why not stay with defines for this specific case? I understand and agree the
issue of figuring out what can happen with a build because you need to
understand the command line but in this case the parameters are bounded and
known. You need a build instance for each test a BSP has and you can add a
specific set of defines to each build instance. Waf will track the flags in a
build and any changes will rebuild the dependent targets.
This could be also realized with a special option action:
actions:
- set-test-states:
- SAMPLE/TICKER: disable
- LIB/FLASHDISK01: xfail
I have avoided abbreviations so far and used `expected-fail`. I would like it to
stay that way if that is OK.
The "set-test-states" action can validate if the state exists. I can use the
"enable-by" for "disable" states and the "bsp/teststate.h" file for other
states.
My concerns with a single test state header are:
1. size, we have a lot of tests
It doesn't matter much where you place the information. With the
CPPFLAGS approach the information just moves to the configuration set
environment.
2. slipping into adding things which should not be there but could be, ie adding
C code to control something
3. potential cross-talk between tests, ie I borrow a piece of code from another
test and do not change a dependent define.
I think 2. and 3. are theoretical issues since the content of the header
file would be determined by the build option item actions.
I also like the idea of a single place to inspect how tests are configured.
Currently you have an optional *.tcfg per base BSP variant with an
ability to include files. You also have a magic rtems.tcfg which is
added by default. This is not a single place to inspect from my point of
view. However, I don't think a good alternative exists.
I re-implemented now practically the *.tcfg files and the existing
mechanism (module the regular expressions). You can find an example here:
https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-TSTFIXME.yml?h=build
This is more or less the old rtems.tcfg.
It is pulled in by:
https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/RTEMS-BUILD-BSP-BSPOPTS.yml?h=build
Which in turn is included by every BSP, e.g.
https://git.rtems.org/sebh/rtems.git/tree/spec/build/bsps/sparc/erc32/RTEMS-BUILD-BSP-SPARC-ERC32-ERC32.yml?h=build
For the "set-test-state" action see:
https://git.rtems.org/sebh/rtems.git/tree/wscript?h=build#n593
Each test has a TEST_EXCLUDE_<NAME> enable and a "cppflags" attribute
set to "- ${TEST_CPPFLAGS_<NAME>}".
The enable and cppflags are set by the "set-test-state" action and are
stored in the environment:
$ grep -r TEST_ build/c4che/sparc/erc32_cache.py
ENABLE = ['sparc', 'erc32', 'RTEMS_NETWORKING', 'RTEMS_NEWLIB',
'RTEMS_POSIX_API', 'RTEMS_SMP', 'BUILD_BENCHMARKS', 'BUILD_FSTESTS',
'BUILD_LIBTESTS', 'BUILD_MPTESTS', 'BUILD_PSXTESTS', 'BUILD_PSXTMTESTS',
'BUILD_RHEALSTONE', 'BUILD_SAMPLES', 'BUILD_SMPTESTS', 'BUILD_SPTESTS',
'BUILD_TMTESTS', 'TEST_EXCLUDE_DL04', 'TEST_EXCLUDE_DL05',
'TEST_EXCLUDE_DL06', 'TEST_EXCLUDE_DL07', 'TEST_EXCLUDE_DL01',
'TEST_EXCLUDE_DL02', 'TEST_EXCLUDE_DL03', 'TEST_EXCLUDE_TAR01',
'TEST_EXCLUDE_TAR02', 'TEST_EXCLUDE_TAR03', 'TEST_EXCLUDE_DL08',
'TEST_EXCLUDE_DL09', 'TEST_EXCLUDE_DL10', 'TEST_EXCLUDE_MGHTTPD01']
TEST_CPPFLAGS_CAPTURE = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_DHRYSTONE = ['-DTEST_STATE_BENCHMARK=1']
TEST_CPPFLAGS_FILEIO = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_LINPACK = ['-DTEST_STATE_BENCHMARK=1']
TEST_CPPFLAGS_MONITOR = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_PSXFENV01 = ['-DTEST_STATE_EXPECTED_FAIL=1']
TEST_CPPFLAGS_TERMIOS = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_TOP = ['-DTEST_STATE_USER_INPUT=1']
TEST_CPPFLAGS_WHETSTONE = ['-DTEST_STATE_BENCHMARK=1']
--
Sebastian Huber, embedded brains GmbH
Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.
Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
