On Wed, Mar 15, 2017 at 6:57 PM, Chris Johns <chr...@rtems.org> wrote:
> On 16/03/2017 08:44, Tanu Hari Dixit wrote:
>> Hello Chris, Gedare, Joel,
>> Thank you for answering my questions. Sorry for the late reply, it
>> took me time to go through the tasks suggested.
>>>
>>> What platforms have you run tester on?
>>
>> I was searching for a way to compile and execute a single test and the
>> community suggetsed rtems-tester. So I ran the tester on sparc/erc32
>> (and no simulator).
>
> Great, this is what the tool is for.
>
>> I am unable to find a relevant doc on how to run rtems-tester on qemu.
>> Though there is this rtems-tools.git/tester/rtems/testing/qemu.cfg so
>> I know that it is supported.
>
> The `--list-bsps` gives some hints however not all BSPs that use qemu have
> this in the test BSP name. After a waf configure and build try:
>
> ruru rtems-tools.git $ pwd
> /opt/work/chris/rtems/tools/rtems-tools.git
> ruru rtems-tools.git $ ./tester/rtems-test --list-bsps | grep qemu
> realview_pbx_a9_qemu
> xilinx_zynq_a9_qemu
> xilinx_zynq_a9_qemu_smp
> xilinx_zynq_zc706_qemu
>
> The low level and simplest way to currently see is:
>
> ruru rtems-tools.git $ grep -r qemu.cfg tester/rtems
> tester/rtems/testing/bsps/xilinx_zynq_a9_qemu.mc:xilinx_zynq_a9_qemu:
> none, none, '%{_rtscripts}/qemu.cfg'
> tester/rtems/testing/bsps/generic_or1k.mc:generic_or1k: none, none,
> '%{_rtscripts}/qemu.cfg'
> tester/rtems/testing/bsps/xilinx_zynq_zc706_qemu.mc:xilinx_zynq_zc706:
> none, none, '%{_rtscripts}/qemu.cfg'
> tester/rtems/testing/bsps/xilinx_zynq_a9_qemu_smp.mc:xilinx_zynq_a9_qemu:
> none, none, '%{_rtscripts}/qemu.cfg'
> tester/rtems/testing/bsps/realview_pbx_a9_qemu.mc:realview_pbx_a9_qemu:
> none, none, '%{_rtscripts}/qemu.cfg'
>
> It is not the best way but it does work :)
>
>> Also I couldn't find any other hint in
>> the rtems-tools repository as to which simulators (except for qemu)
>> are supported. rtems-testing/sim-scripts support gdbsim, skyeye and
>> tsim (other than qemu). So we can add support for these and gem5 (as
>> Gedare suggested). Please correct me if I am wrong.
>
> You will need to review each .cfg file we have and see what there is. I see
> tsim is there.
>
>> Furthermore, if you could please point me to a doc that explains using
>> rtems-tester on qemu? I have tested qemu for running an application,
>> the steps for which I have documented here
>> (http://tokencolour.github.io/2017/03/16/how-to-run-applications-on-qemu-rtems.html).
>
> I am not sure there is any doco. If you have built the RSB qemu which runs
> the zynq and you have built the xilinx_zynq_a9_qemu BSP you should be able to
> do:
>
> ruru xilinx_zynq_a9_qemu $ pwd
> /opt/work/chris/rtems/kernel/bsps/xilinx_zynq_a9_qemu
> ruru xilinx_zynq_a9_qemu $ rtems-test --report-mode=all
> --rtems-bsp=xilinx_zynq_a9_qemu `find . -name hello.exe`
> RTEMS Testing - Tester, 4.12.not_released
> Command Line: /opt/work/rtems/4.12/bin/rtems-test --report-mode=all
> --rtems-bsp=xilinx_zynq_a9_qemu
> ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
> Python: 2.7.13 (default, Jan 12 2017, 01:19:30) [GCC 4.2.1 Compatible
> FreeBSD Clang 3.8.0 (tags/RELEASE_380/final 262564)]
> [1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
> [1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe
>> run: qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic
>> -net none -M xilinx-zynq-a9 -m 256M -kernel
>> ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe
> ] Warning: nic cadence_gem.0 has no peer
> ] Warning: nic cadence_gem.1 has no peer
> ]
> ]
> ] *** BEGIN OF TEST HELLO WORLD ***
> ] Hello World
> ] *** END OF TEST HELLO WORLD ***
> Result: passed Time: 0:00:01.069137 hello.exe
>
> Passed: 1
> Failed: 0
> Timeouts: 0
> Invalid: 0
> -----------
> Total: 1
>
> Average test time: 0:00:01.526334
> Testing time : 0:00:01.526334
>
Thanks Chris for the excellent answers. I want to just pile-on with my
typical command line for running tester as another point in the
conversation:
Command Line: ./rtems-test --log=log_zynq_run
--rtems-bsp=xilinx_zynq_a9_qemu
--rtems-tools=/home/gedare/install/4.12
/home/gedare/work/rtems/builds/b-xilinx_zynq_a9_qemu/arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites
The last argument points to the testsuites in my build tree, which
results in running all of the testsuites I built.
I almost always just do...
$ head log_zynq_run
and then copy-paste the command line into my terminal to run tester.
It keeps me from having to remember/type anything more than once. :)
(I do the same thing with RSB. It is a great feature of the log file!)
>> I can play with visualizing aggregated testing results with plotting
>> techniques (like matplotlib or maybe something that the devs
>> recommend) :).
>
> I would prefer this not be part of rtems-test. I am happy to have rtems-test
> generate report data in XML that can be used by other tools to generate
> visual data. I also happy to see these other tools being developed as part of
> this project if there is time and added to the RTEMS Tools Project if
> suitable, just not part of the rtems-test tool.
>
> A couple of points ...
>
> The separation is done to avoid extra dependencies being added to the
> rtems-test tool. Separating this type of functionality this way means the way
> this data is used is up to the user and not the rtems-test tool. It also
> brings the tool inline with the RSB which also exports XML reports.
>
> Second, any tool to do this and added to the RTEMS Tool Project has to be
> fully cross platform, which means FreeBSD, OSX, Windows and even Linux. If
> the tool can be based on just standard Python libraries it will be cross
> platform and preferred.
>
>> Also, where can I find more about "expected fail" state? Is it for
>> those tests that fail nevertheless and need to be skipped? Can you
>> please point me to an example?
>
> I do not think there currently are examples. I suspect a few tests which are
> expected fails maybe currently configured as exclude because that was all we
> had.
>
> A test is tagged as 'expected-fail' in a .tcfg per BSP file using the newer
> tag format. To list the test config files try `find . -name \*.tcfg` from the
> top of the RTEMS source tree and take a look at a few of these files.
>
> Setting a test as expected-fail causes the test to be built with compile
> command line directive of `-DTEST_STATE_EXPECTED_FAIL=1` and this results in
> the message:
>
> *** TEST STATE: EXPECTED-FAIL
>
> being printed in the test. I have also added the test state `indeterminate`
> which prints:
>
> *** TEST STATE: INDETERMINATE
>
> See testsuites/support/include/buffer_test_io.h for the details.
>
> rtems-test needs to scan for this text like it currently scans for the begin
> and end messages and it needs to record the result for the test and have the
> result stats updated.
>
>> Also, what kind of regression will one need to perform?
>
> None, it is a report. If there are per BSP expected test results that are
> accessible to rtems-test you can easily report to the user if they have
> regressed or improved. The expected test results could simply be fail,
> expected-fail and indeterminate counts the remainder being pass. This could
> be enhanced to specify specific tests which would give a more accurate
> picture in case failures move around and maintain the same test result counts.
>
>> Do the final states of the test need to maintain an
>> overall state of the system so as to recommend/predict the state of
>> affairs next time the tests are run (i.e. a predictor (which has
>> memory) kind of thing)?
>
> No it is just relative to the current BSP expected test results.
>
>> I also think that converting macro files to yaml won't be very
>> difficult. I might be very myopic here and it would be great if I get
>> a heads up. What will be the challenge here?
>
> Determining what we need to describe and creating and documenting the format.
> These types of task are easy to visualize but time consuming to get right and
> working. Also making the changes then verifying all the converted
> configurations are working is a lot of work and may require interactions with
> others who have the various simulators or hardware being used. For example
> tsim and ARM JTAG.
>
>> For the generic serial console, will pySerial do the trick, on
>> Windows? Do you have some other package in mind?
>
> pySerial is the package (https://github.com/pyserial/pyserial). It has a
> suitable license. We would need to being the specific pieces of code into the
> rtemstoolkit, wrap and make available for the rtems-test tool.
>
>> I have a problem here, though. I don't own a Zedboard and can't afford
>> one. I can check with my department at college once I reach there, but
>> I don't have my hopes too high. Will this pose a problem in the course
>> of the project?
>
> I do not think so. I have one and I know others do as well, Joel?. We can see
> if some form of remote access can be made available. Please indicate this in
> your proposal.
>
>> I am yet to build for xilinx_zynq_zc706, so I'll ask more questions on
>> the configuration control once I am done with it. Also, I'm trying out
>> the old and new approach for Coverage reporting. I'll ask questions on
>> it, once I try it.
>
> I suggest you concentrate on the xilinx_zynq_a9_qemu BSP and then the RSB
> qemu under the bare config tree.
>
> Chris
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
