Thank you so much Chris for the elaborate answers. I have a clearer idea of the project now. Thank you Gedare for the tip, it definitely comes in handy. I'm sorry for the late reply. I have more questions and I will be grateful if the devs can help me with these.
How does one use the console? I see extensive use of termios in rtems-tools.git/tester/rt/stty.py which is in turn used in tester/rt/console.py Can you please demonstrate one example so that I get an idea? I believe PySerial will need to be configured in a similar way i.e. making objects of class tty and assigning them attributes with Pyserial instead of termios (like ignbrk and the rest). Please correct me if I am wrong. Moreover, about the telnet console, I went through the ser2net package. It serves as a good way to convert (or maybe the right word is "use") serial connections via telnet. I can make a simple bash script to install ser2net and then change its conf file to fix the appropriate port numbers and banners(which I can take as entry from user). Then I shall use them via telnetlib to communicate to the hosts and then I can run tests via the console on the target. Is this the right direction or should I be thinking differently? Also I am really confused as to how rtemstoolkit comes into the picture. Can you please provide me with a rough workflow of the console? I also had questions regarding extending support for "expected-fail" and "user-input" state as mentioned here (https://devel.rtems.org/ticket/2946). What follows is my understanding of what I need to do to extend support for "user-input"(and "expected-fail"), please correct me if I am wrong. I need to make a directive like DTEST_STATE_USER_INPUT and declare it to be 1 in /rtems/src/rtems/tools/build/rtems-test-check and choose a suitable TEST_STATE_STRING in buffer_test_io.h. Consequently, parse for it in tester/rt/report.py. (probably this has to be done for "expected-fail" state also). Also, I would need to extend the attributes of class report that already has self.passed = 0 self.failed = 0 self.timeouts = 0 self.invalids = 0 to also include self.expected_fails and self.user_inputs. Please correct me if I am wrong. Can you please point me to the commit with which you added support for "extended-fail"? Furthermore, I was able to run the hello world example on arm/xilinx_zync_a9_qemu with rtems-test as you demonstrated. Thank you. Also, I understand why you emphasize on keeping a plotting tool different from rtems-tester. An aside question: Do you use winpdb to debug the python scripts in rtems-tester? Do you recommend some other tool? Thanking you, Tanu Hari Dixit. On Sat, Mar 18, 2017 at 1:53 AM, Gedare Bloom <ged...@rtems.org> wrote: > On Wed, Mar 15, 2017 at 6:57 PM, Chris Johns <chr...@rtems.org> wrote: >> On 16/03/2017 08:44, Tanu Hari Dixit wrote: >>> Hello Chris, Gedare, Joel, >>> Thank you for answering my questions. Sorry for the late reply, it >>> took me time to go through the tasks suggested. >>>> >>>> What platforms have you run tester on? >>> >>> I was searching for a way to compile and execute a single test and the >>> community suggetsed rtems-tester. So I ran the tester on sparc/erc32 >>> (and no simulator). >> >> Great, this is what the tool is for. >> >>> I am unable to find a relevant doc on how to run rtems-tester on qemu. >>> Though there is this rtems-tools.git/tester/rtems/testing/qemu.cfg so >>> I know that it is supported. >> >> The `--list-bsps` gives some hints however not all BSPs that use qemu have >> this in the test BSP name. After a waf configure and build try: >> >> ruru rtems-tools.git $ pwd >> /opt/work/chris/rtems/tools/rtems-tools.git >> ruru rtems-tools.git $ ./tester/rtems-test --list-bsps | grep qemu >> realview_pbx_a9_qemu >> xilinx_zynq_a9_qemu >> xilinx_zynq_a9_qemu_smp >> xilinx_zynq_zc706_qemu >> >> The low level and simplest way to currently see is: >> >> ruru rtems-tools.git $ grep -r qemu.cfg tester/rtems >> tester/rtems/testing/bsps/xilinx_zynq_a9_qemu.mc:xilinx_zynq_a9_qemu: >> none, none, '%{_rtscripts}/qemu.cfg' >> tester/rtems/testing/bsps/generic_or1k.mc:generic_or1k: none, none, >> '%{_rtscripts}/qemu.cfg' >> tester/rtems/testing/bsps/xilinx_zynq_zc706_qemu.mc:xilinx_zynq_zc706: >> none, none, '%{_rtscripts}/qemu.cfg' >> tester/rtems/testing/bsps/xilinx_zynq_a9_qemu_smp.mc:xilinx_zynq_a9_qemu: >> none, none, '%{_rtscripts}/qemu.cfg' >> tester/rtems/testing/bsps/realview_pbx_a9_qemu.mc:realview_pbx_a9_qemu: >> none, none, '%{_rtscripts}/qemu.cfg' >> >> It is not the best way but it does work :) >> >>> Also I couldn't find any other hint in >>> the rtems-tools repository as to which simulators (except for qemu) >>> are supported. rtems-testing/sim-scripts support gdbsim, skyeye and >>> tsim (other than qemu). So we can add support for these and gem5 (as >>> Gedare suggested). Please correct me if I am wrong. >> >> You will need to review each .cfg file we have and see what there is. I see >> tsim is there. >> >>> Furthermore, if you could please point me to a doc that explains using >>> rtems-tester on qemu? I have tested qemu for running an application, >>> the steps for which I have documented here >>> (http://tokencolour.github.io/2017/03/16/how-to-run-applications-on-qemu-rtems.html). >> >> I am not sure there is any doco. If you have built the RSB qemu which runs >> the zynq and you have built the xilinx_zynq_a9_qemu BSP you should be able >> to do: >> >> ruru xilinx_zynq_a9_qemu $ pwd >> /opt/work/chris/rtems/kernel/bsps/xilinx_zynq_a9_qemu >> ruru xilinx_zynq_a9_qemu $ rtems-test --report-mode=all >> --rtems-bsp=xilinx_zynq_a9_qemu `find . -name hello.exe` >> RTEMS Testing - Tester, 4.12.not_released >> Command Line: /opt/work/rtems/4.12/bin/rtems-test --report-mode=all >> --rtems-bsp=xilinx_zynq_a9_qemu >> ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe >> Python: 2.7.13 (default, Jan 12 2017, 01:19:30) [GCC 4.2.1 Compatible >> FreeBSD Clang 3.8.0 (tags/RELEASE_380/final 262564)] >> [1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe >> [1/1] p:0 f:0 t:0 i:0 | arm/xilinx_zynq_a9_qemu: hello.exe >>> run: qemu-system-arm -no-reboot -serial null -serial mon:stdio -nographic >>> -net none -M xilinx-zynq-a9 -m 256M -kernel >>> ./arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites/samples/hello/hello.exe >> ] Warning: nic cadence_gem.0 has no peer >> ] Warning: nic cadence_gem.1 has no peer >> ] >> ] >> ] *** BEGIN OF TEST HELLO WORLD *** >> ] Hello World >> ] *** END OF TEST HELLO WORLD *** >> Result: passed Time: 0:00:01.069137 hello.exe >> >> Passed: 1 >> Failed: 0 >> Timeouts: 0 >> Invalid: 0 >> ----------- >> Total: 1 >> >> Average test time: 0:00:01.526334 >> Testing time : 0:00:01.526334 >> > Thanks Chris for the excellent answers. I want to just pile-on with my > typical command line for running tester as another point in the > conversation: > > Command Line: ./rtems-test --log=log_zynq_run > --rtems-bsp=xilinx_zynq_a9_qemu > --rtems-tools=/home/gedare/install/4.12 > /home/gedare/work/rtems/builds/b-xilinx_zynq_a9_qemu/arm-rtems4.12/c/xilinx_zynq_a9_qemu/testsuites > > The last argument points to the testsuites in my build tree, which > results in running all of the testsuites I built. > > I almost always just do... > $ head log_zynq_run > and then copy-paste the command line into my terminal to run tester. > It keeps me from having to remember/type anything more than once. :) > > (I do the same thing with RSB. It is a great feature of the log file!) > >>> I can play with visualizing aggregated testing results with plotting >>> techniques (like matplotlib or maybe something that the devs >>> recommend) :). >> >> I would prefer this not be part of rtems-test. I am happy to have rtems-test >> generate report data in XML that can be used by other tools to generate >> visual data. I also happy to see these other tools being developed as part >> of this project if there is time and added to the RTEMS Tools Project if >> suitable, just not part of the rtems-test tool. >> >> A couple of points ... >> >> The separation is done to avoid extra dependencies being added to the >> rtems-test tool. Separating this type of functionality this way means the >> way this data is used is up to the user and not the rtems-test tool. It also >> brings the tool inline with the RSB which also exports XML reports. >> >> Second, any tool to do this and added to the RTEMS Tool Project has to be >> fully cross platform, which means FreeBSD, OSX, Windows and even Linux. If >> the tool can be based on just standard Python libraries it will be cross >> platform and preferred. >> >>> Also, where can I find more about "expected fail" state? Is it for >>> those tests that fail nevertheless and need to be skipped? Can you >>> please point me to an example? >> >> I do not think there currently are examples. I suspect a few tests which are >> expected fails maybe currently configured as exclude because that was all we >> had. >> >> A test is tagged as 'expected-fail' in a .tcfg per BSP file using the newer >> tag format. To list the test config files try `find . -name \*.tcfg` from >> the top of the RTEMS source tree and take a look at a few of these files. >> >> Setting a test as expected-fail causes the test to be built with compile >> command line directive of `-DTEST_STATE_EXPECTED_FAIL=1` and this results in >> the message: >> >> *** TEST STATE: EXPECTED-FAIL >> >> being printed in the test. I have also added the test state `indeterminate` >> which prints: >> >> *** TEST STATE: INDETERMINATE >> >> See testsuites/support/include/buffer_test_io.h for the details. >> >> rtems-test needs to scan for this text like it currently scans for the begin >> and end messages and it needs to record the result for the test and have the >> result stats updated. >> >>> Also, what kind of regression will one need to perform? >> >> None, it is a report. If there are per BSP expected test results that are >> accessible to rtems-test you can easily report to the user if they have >> regressed or improved. The expected test results could simply be fail, >> expected-fail and indeterminate counts the remainder being pass. This could >> be enhanced to specify specific tests which would give a more accurate >> picture in case failures move around and maintain the same test result >> counts. >> >>> Do the final states of the test need to maintain an >>> overall state of the system so as to recommend/predict the state of >>> affairs next time the tests are run (i.e. a predictor (which has >>> memory) kind of thing)? >> >> No it is just relative to the current BSP expected test results. >> >>> I also think that converting macro files to yaml won't be very >>> difficult. I might be very myopic here and it would be great if I get >>> a heads up. What will be the challenge here? >> >> Determining what we need to describe and creating and documenting the >> format. These types of task are easy to visualize but time consuming to get >> right and working. Also making the changes then verifying all the converted >> configurations are working is a lot of work and may require interactions >> with others who have the various simulators or hardware being used. For >> example tsim and ARM JTAG. >> >>> For the generic serial console, will pySerial do the trick, on >>> Windows? Do you have some other package in mind? >> >> pySerial is the package (https://github.com/pyserial/pyserial). It has a >> suitable license. We would need to being the specific pieces of code into >> the rtemstoolkit, wrap and make available for the rtems-test tool. >> >>> I have a problem here, though. I don't own a Zedboard and can't afford >>> one. I can check with my department at college once I reach there, but >>> I don't have my hopes too high. Will this pose a problem in the course >>> of the project? >> >> I do not think so. I have one and I know others do as well, Joel?. We can >> see if some form of remote access can be made available. Please indicate >> this in your proposal. >> >>> I am yet to build for xilinx_zynq_zc706, so I'll ask more questions on >>> the configuration control once I am done with it. Also, I'm trying out >>> the old and new approach for Coverage reporting. I'll ask questions on >>> it, once I try it. >> >> I suggest you concentrate on the xilinx_zynq_a9_qemu BSP and then the RSB >> qemu under the bare config tree. >> >> Chris _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
