On 16/03/2017 08:44, Tanu Hari Dixit wrote:
> Hello Chris, Gedare, Joel,
> Thank you for answering my questions. Sorry for the late reply, it
> took me time to go through the tasks suggested.
>>
>> What platforms have you run tester on?
> 
> I was searching for a way to compile and execute a single test and the
> community suggetsed rtems-tester. So I ran the tester on sparc/erc32
> (and no simulator).
Great, this is what the tool is for.

> I am unable to find a relevant doc on how to run rtems-tester on qemu.
> Though there is this rtems-tools.git/tester/rtems/testing/qemu.cfg so
> I know that it is supported.

The `--list-bsps` gives some hints however not all BSPs that use qemu have this 
in the test BSP name. After a waf configure and build try:

ruru rtems-tools.git $ pwd
/opt/work/chris/rtems/tools/rtems-tools.git
ruru rtems-tools.git $ ./tester/rtems-test --list-bsps | grep qemu
  realview_pbx_a9_qemu
  xilinx_zynq_a9_qemu
  xilinx_zynq_a9_qemu_smp
  xilinx_zynq_zc706_qemu

The low level and simplest way to currently see is:

ruru rtems-tools.git $ grep -r qemu.cfg tester/rtems
tester/rtems/testing/bsps/xilinx_zynq_a9_qemu.mc:xilinx_zynq_a9_qemu:      
none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/generic_or1k.mc:generic_or1k:      none,    none,  
'%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/xilinx_zynq_zc706_qemu.mc:xilinx_zynq_zc706:      
none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/xilinx_zynq_a9_qemu_smp.mc:xilinx_zynq_a9_qemu:      
none,    none,  '%{_rtscripts}/qemu.cfg'
tester/rtems/testing/bsps/realview_pbx_a9_qemu.mc:realview_pbx_a9_qemu:      
none,    none,  '%{_rtscripts}/qemu.cfg'

It is not the best way but it does work :)

> Also I couldn't find any other hint in
> the rtems-tools repository as to which simulators (except for qemu)
> are supported. rtems-testing/sim-scripts support gdbsim, skyeye and
> tsim (other than qemu). So we can add support for these and gem5 (as
> Gedare suggested). Please correct me if I am wrong.

You will need to review each .cfg file we have and see what there is. I see 
tsim is there.

> Furthermore, if you could please point me to a doc that explains using
> rtems-tester on qemu? I have tested qemu for running an application,
> the steps for which I have documented here
> (http://tokencolour.github.io/2017/03/16/how-to-run-applications-on-qemu-rtems.html).

I am not sure there is any doco. If you have built the RSB qemu which runs the 
zynq and you have built the xilinx_zynq_a9_qemu BSP you should be able to do:

ruru xilinx_zynq_a9_qemu $ pwd
/opt/work/chris/rtems/kernel/bsps/xilinx_zynq_a9_qemu
ruru xilinx_zynq_a9_qemu $ rtems-test --report-mode=all 
--rtems-bsp=xilinx_zynq_a9_qemu `find . -name hello.exe`
RTEMS Testing - Tester, 4.12.not_released
 Command Line: /opt/work/rtems/4.12/bin/rtems-test --report-mode=all 
--rtems-bsp=xilinx_zynq_a9_qemu 
./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
 Python: 2.7.13 (default, Jan 12 2017, 01:19:30) [GCC 4.2.1 Compatible FreeBSD 
Clang 3.8.0 (tags/RELEASE_380/final 262564)]
[1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
[1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
> run: qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic 
> -net none -M xilinx-zynq-a9 -m 256M -kernel 
> ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
] Warning: nic cadence_gem.0 has no peer
] Warning: nic cadence_gem.1 has no peer
]
]
] *** BEGIN OF TEST HELLO WORLD ***
] Hello World
] *** END OF TEST HELLO WORLD ***
Result: passed     Time: 0:00:01.069137 hello.exe

Passed:   1
Failed:   0
Timeouts: 0
Invalid:  0
-----------
Total:    1

Average test time: 0:00:01.526334
Testing time     : 0:00:01.526334

> I can play with visualizing aggregated testing results with plotting
> techniques (like matplotlib or maybe something that the devs
> recommend) :).

I would prefer this not be part of rtems-test. I am happy to have rtems-test 
generate report data in XML that can be used by other tools to generate visual 
data. I also happy to see these other tools being developed as part of this 
project if there is time and added to the RTEMS Tools Project if suitable, just 
not part of the rtems-test tool.

A couple of points ...

The separation is done to avoid extra dependencies being added to the 
rtems-test tool. Separating this type of functionality this way means the way 
this data is used is up to the user and not the rtems-test tool. It also brings 
the tool inline with the RSB which also exports XML reports.

Second, any tool to do this and added to the RTEMS Tool Project has to be fully 
cross platform, which means FreeBSD, OSX, Windows and even Linux. If the tool 
can be based on just standard Python libraries it will be cross platform and 
preferred.

> Also, where can I find more about "expected fail" state? Is it for
> those tests that fail nevertheless and need to be skipped? Can you
> please point me to an example? 

I do not think there currently are examples. I suspect a few tests which are 
expected fails maybe currently configured as exclude because that was all we 
had.

A test is tagged as 'expected-fail' in a .tcfg per BSP file using the newer tag 
format. To list the test config files try `find . -name \*.tcfg` from the top 
of the RTEMS source tree and take a look at a few of these files.

Setting a test as expected-fail causes the test to be built with compile 
command line directive of `-DTEST_STATE_EXPECTED_FAIL=1` and this results in 
the message:

 *** TEST STATE: EXPECTED-FAIL

being printed in the test. I have also added the test state `indeterminate` 
which prints:

 *** TEST STATE: INDETERMINATE

See testsuites/support/include/buffer_test_io.h for the details.

rtems-test needs to scan for this text like it currently scans for the begin 
and end messages and it needs to record the result for the test and have the 
result stats updated.

> Also, what kind of regression will one need to perform?

None, it is a report. If there are per BSP expected test results that are 
accessible to rtems-test you can easily report to the user if they have 
regressed or improved. The expected test results could simply be fail, 
expected-fail and indeterminate counts the remainder being pass. This could be 
enhanced to specify specific tests which would give a more accurate picture in 
case failures move around and maintain the same test result counts.

> Do the final states of the test need to maintain an
> overall state of the system so as to recommend/predict the state of
> affairs next time the tests are run (i.e. a predictor (which has
> memory) kind of thing)?

No it is just relative to the current BSP expected test results.

> I also think that converting macro files to yaml won't be very
> difficult. I might be very myopic here and it would be great if I get
> a heads up. What will be the challenge here?

Determining what we need to describe and creating and documenting the format. 
These types of task are easy to visualize but time consuming to get right and 
working. Also making the changes then verifying all the converted 
configurations are working is a lot of work and may require interactions with 
others who have the various simulators or hardware being used. For example tsim 
and ARM JTAG.

> For the generic serial console, will pySerial do the trick, on
> Windows? Do you have some other package in mind?

pySerial is the package (https://github.com/pyserial/pyserial). It has a 
suitable license. We would need to being the specific pieces of code into the 
rtemstoolkit, wrap and make available for the rtems-test tool.

> I have a problem here, though. I don't own a Zedboard and can't afford
> one. I can check with my department at college once I reach there, but
> I don't have my hopes too high. Will this pose a problem in the course
> of the project?

I do not think so. I have one and I know others do as well, Joel?. We can see 
if some form of remote access can be made available. Please indicate this in 
your proposal.

> I am yet to build for xilinx_zynq_zc706, so I'll ask more questions on
> the configuration control once I am done with it. Also, I'm trying out
> the old and new approach for Coverage reporting. I'll ask questions on
> it, once I try it.

I suggest you concentrate on the xilinx_zynq_a9_qemu BSP and then the RSB qemu 
under the bare config tree.

Chris
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Reply via email to