On 16/03/2017 08:44, Tanu Hari Dixit wrote:
> Hello Chris, Gedare, Joel,
> Thank you for answering my questions. Sorry for the late reply, it
> took me time to go through the tasks suggested.
>>
>> What platforms have you run tester on?
>
> I was searching for a way to compile and execute a single test and the
> community suggetsed rtems-tester. So I ran the tester on sparc/erc32
> (and no simulator).
Great, this is what the tool is for.
> I am unable to find a relevant doc on how to run rtems-tester on qemu.
> Though there is this rtems-tools.git/tester/rtems/testing/qemu.cfg so
> I know that it is supported.
The `--list-bsps` gives some hints however not all BSPs that use qemu have this
in the test BSP name. After a waf configure and build try:
ruru rtems-tools.git $ pwd
/opt/work/chris/rtems/tools/rtems-tools.git
ruru rtems-tools.git $ ./tester/rtems-test --list-bsps | grep qemu
realview_pbx_a9_qemu
xilinx_zynq_a9_qemu
xilinx_zynq_a9_qemu_smp
xilinx_zynq_zc706_qemu
The low level and simplest way to currently see is:
ruru rtems-tools.git $ grep -r qemu.cfg tester/rtems
tester/rtems/testing/bsps/xilinx_zynq_a9_qemu.mc:xilinx_zynq_a9_qemu:
none, none, '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/generic_or1k.mc:generic_or1k: none, none,
'%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/xilinx_zynq_zc706_qemu.mc:xilinx_zynq_zc706:
none, none, '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/xilinx_zynq_a9_qemu_smp.mc:xilinx_zynq_a9_qemu:
none, none, '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/realview_pbx_a9_qemu.mc:realview_pbx_a9_qemu:
none, none, '%{_rtscripts}/qemu.cfg'
It is not the best way but it does work :)
> Also I couldn't find any other hint in
> the rtems-tools repository as to which simulators (except for qemu)
> are supported. rtems-testing/sim-scripts support gdbsim, skyeye and
> tsim (other than qemu). So we can add support for these and gem5 (as
> Gedare suggested). Please correct me if I am wrong.
You will need to review each .cfg file we have and see what there is. I see
tsim is there.
> Furthermore, if you could please point me to a doc that explains using
> rtems-tester on qemu? I have tested qemu for running an application,
> the steps for which I have documented here
> (http://tokencolour.github.io/2017/03/16/how-to-run-applications-on-qemu-rtems.html).
I am not sure there is any doco. If you have built the RSB qemu which runs the
zynq and you have built the xilinx_zynq_a9_qemu BSP you should be able to do:
ruru xilinx_zynq_a9_qemu $ pwd
/opt/work/chris/rtems/kernel/bsps/xilinx_zynq_a9_qemu
ruru xilinx_zynq_a9_qemu $ rtems-test --report-mode=all
--rtems-bsp=xilinx_zynq_a9_qemu `find . -name hello.exe`
RTEMS Testing - Tester, 4.12.not_released
Command Line: /opt/work/rtems/4.12/bin/rtems-test --report-mode=all
--rtems-bsp=xilinx_zynq_a9_qemu
./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
Python: 2.7.13 (default, Jan 12 2017, 01:19:30) [GCC 4.2.1 Compatible FreeBSD
Clang 3.8.0 (tags/RELEASE_380/final 262564)]
[1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
[1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
> run: qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
> -net none -M xilinx-zynq-a9 -m 256M -kernel
> ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
] Warning: nic cadence_gem.0 has no peer
] Warning: nic cadence_gem.1 has no peer
]
]
] *** BEGIN OF TEST HELLO WORLD ***
] Hello World
] *** END OF TEST HELLO WORLD ***
Result: passed Time: 0:00:01.069137 hello.exe
Passed: 1
Failed: 0
Timeouts: 0
Invalid: 0
-----------
Total: 1
Average test time: 0:00:01.526334
Testing time : 0:00:01.526334
> I can play with visualizing aggregated testing results with plotting
> techniques (like matplotlib or maybe something that the devs
> recommend) :).
I would prefer this not be part of rtems-test. I am happy to have rtems-test
generate report data in XML that can be used by other tools to generate visual
data. I also happy to see these other tools being developed as part of this
project if there is time and added to the RTEMS Tools Project if suitable, just
not part of the rtems-test tool.
A couple of points ...
The separation is done to avoid extra dependencies being added to the
rtems-test tool. Separating this type of functionality this way means the way
this data is used is up to the user and not the rtems-test tool. It also brings
the tool inline with the RSB which also exports XML reports.
Second, any tool to do this and added to the RTEMS Tool Project has to be fully
cross platform, which means FreeBSD, OSX, Windows and even Linux. If the tool
can be based on just standard Python libraries it will be cross platform and
preferred.
> Also, where can I find more about "expected fail" state? Is it for
> those tests that fail nevertheless and need to be skipped? Can you
> please point me to an example?
I do not think there currently are examples. I suspect a few tests which are
expected fails maybe currently configured as exclude because that was all we
had.
A test is tagged as 'expected-fail' in a .tcfg per BSP file using the newer tag
format. To list the test config files try `find . -name \*.tcfg` from the top
of the RTEMS source tree and take a look at a few of these files.
Setting a test as expected-fail causes the test to be built with compile
command line directive of `-DTEST_STATE_EXPECTED_FAIL=1` and this results in
the message:
*** TEST STATE: EXPECTED-FAIL
being printed in the test. I have also added the test state `indeterminate`
which prints:
*** TEST STATE: INDETERMINATE
See testsuites/support/include/buffer_test_io.h for the details.
rtems-test needs to scan for this text like it currently scans for the begin
and end messages and it needs to record the result for the test and have the
result stats updated.
> Also, what kind of regression will one need to perform?
None, it is a report. If there are per BSP expected test results that are
accessible to rtems-test you can easily report to the user if they have
regressed or improved. The expected test results could simply be fail,
expected-fail and indeterminate counts the remainder being pass. This could be
enhanced to specify specific tests which would give a more accurate picture in
case failures move around and maintain the same test result counts.
> Do the final states of the test need to maintain an
> overall state of the system so as to recommend/predict the state of
> affairs next time the tests are run (i.e. a predictor (which has
> memory) kind of thing)?
No it is just relative to the current BSP expected test results.
> I also think that converting macro files to yaml won't be very
> difficult. I might be very myopic here and it would be great if I get
> a heads up. What will be the challenge here?
Determining what we need to describe and creating and documenting the format.
These types of task are easy to visualize but time consuming to get right and
working. Also making the changes then verifying all the converted
configurations are working is a lot of work and may require interactions with
others who have the various simulators or hardware being used. For example tsim
and ARM JTAG.
> For the generic serial console, will pySerial do the trick, on
> Windows? Do you have some other package in mind?
pySerial is the package (https://github.com/pyserial/pyserial). It has a
suitable license. We would need to being the specific pieces of code into the
rtemstoolkit, wrap and make available for the rtems-test tool.
> I have a problem here, though. I don't own a Zedboard and can't afford
> one. I can check with my department at college once I reach there, but
> I don't have my hopes too high. Will this pose a problem in the course
> of the project?
I do not think so. I have one and I know others do as well, Joel?. We can see
if some form of remote access can be made available. Please indicate this in
your proposal.
> I am yet to build for xilinx_zynq_zc706, so I'll ask more questions on
> the configuration control once I am done with it. Also, I'm trying out
> the old and new approach for Coverage reporting. I'll ask questions on
> it, once I try it.
I suggest you concentrate on the xilinx_zynq_a9_qemu BSP and then the RSB qemu
under the bare config tree.
Chris
_______________________________________________
devel mailing list
[email protected]
http://lists.rtems.org/mailman/listinfo/devel
