On 22/7/20 1:04 am, Sebastian Huber wrote: > This patch set adds a couple of improvements to the test framework: > > * The header file changes from <t.h> to <rtems/test.h>. > > * Support for a stack of test fixtures. This helps to write test > building blocks. > > * The test check messages are now optional. > > * The support for interrupt tests and change all the interrupt critical > tests to use this new test support. This should address the sporadic > failures and timeouts. > > For the documentation please have a look at: > > https://ftp.rtems.org/pub/rtems/people/sebh/eng2.pdf >
I assume you asking us to look only at section 8? The doco looks good. The following are some review comment: 8.1 a) Supported languages? 8.1.1 a) The wording ".. outcome all right, ..." seems non-specific for someting I would assume needs to be specific. What about " ... is not the required outcome for the test, ..."? b) As previously discussed there are a few more states a test result can be other than pass and fail and I thikn the wording here may need tightening. I am not sure what is needed. Also how do resource leaks effect the result? 8.1.2 a) It test case naming to be specified or can be use anything we like, ie lower case with `_`, Camelcase, ...? b) Change "static constructors" to "static C constructors". 8.1.3 a) Change "test case as well as to give thescope for log messags." to " test case as well as the scope for log messages." b) Change "function. It can be set within the scope of one .." to "function and can be set within the scope of one .." c) Does the framework provide a standard way to export a dynamic test environment so the test and the results can be reviewed as a complete set of data? 8.1.4 a) Should "Each non-quiet test check fetches.." be "A non-quiet test check fetches.."? 8.1.5 a) Change "..if various resources are leaked .." to ".. if various resources have leaked ..". 8.1.7 a) Change "You can add test case destructors with T_add_destructor(). They are.." to "A test case destructor can be added with T_add_destructor(). The destructors are..". b) Maybe add it is best to use statically allocated memory only? 8.1.8 a) "Meet its expecation" ... I am not sure what this means, who expecation and what does expecation mean? Do you mean " .. test check meets the check's exepected outcome."? b) I am not sure what "The actual value ..." is referring to. The value the test expects to be the input from the test to the check? Ah coming back to here, I see this is in the next bit. Would adding a little bit in the text before using the special words like actual and expected be helpful? c) I think the T_assert_ etc variants of the test checks should be explained somewhere. I think it is important to know tests can continue even after a fail or they can be made to stop. I had noted I had not seen this documented and I have now found it buried in here. d) I think the word `available` can be removed from the initial sentence on all sub-sectons? 8.1.8.10 a) I think type variant piece of test should be moved to the top and add to the initial sentence so you have " are available where the type variant `xyz` must be ...". 8.1.8.12 a) What about 0 or EINTR as OK? :) 8.1.6 a) Are the printf variants check for type correctness by the compiler? If so I suggest this is mentioned. 8.1.11 a) Can "You can .." be removed so it is not person specific? For example "You can convert time into ticks with the" ciuld be "The time can be converted into ticks with the"? This comment covers all the "You can"s. :) 8.1.13 a) What is the picture drawn in? I will comment on the patches as well. Chris > Sebastian Huber (33): > libtest: <rtems/test.h> to <rtems/simple-test.h> > libtest: Move <t.h> to <rtems/test.h> > libtest: Add T_busy() > libtest: Add T_get_one_clock_tick_busy() > libtest: Add T_make_runner() > libtest: Support custom scope messages via fixture > libtest: Add push/pop fixture support > libtest: Add T_get_scope() > libtest: Split POSIX Keys support > libtest: Add T_stop() > libtest: Add T_CHECK_FMT > libtest: Make check message optional > libtest: Add T_unreachable() > libtest: Add quiet assert NULL pointer checks > libtest: Add rtems_test_run() > libtest: Add T_interrupt_test() > psxintrcritical01: Use T_interrupt_test() > spintrcritical01/2/3/4/5: Use T_interrupt_test() > spintrcritical06/spintrcritical07: Remove tests > spintrcritical08: Use T_interrupt_test() > spintrcritical09: Use T_interrupt_test() > spintrcritical10: Use T_interrupt_test() > spintrcritical11/12: Use T_interrupt_test() > spintrcritical13/14: Use T_interrupt_test() > spintrcritical15: Use T_interrupt_test() > spintrcritical16: Use T_interrupt_test() > spintrcritical18: Use T_interrupt_test() > spintrcritical20: Use T_interrupt_test() > spintrcritical21: Use T_interrupt_test() > spintrcritical22: Use T_interrupt_test() > spintrcritical23: Use T_interrupt_test() > spintrcritical24: Use T_interrupt_test() > spintrcritical_support: Remove > > cpukit/Makefile.am | 5 + > cpukit/headers.am | 3 +- > cpukit/include/rtems/doxygen-appl-config.h | 3736 +++++++++++++++++ > cpukit/include/rtems/simple-test.h | 327 ++ > cpukit/include/rtems/test.h | 2653 ++++++++++-- > cpukit/include/t.h | 2379 ----------- > cpukit/libtest/t-test-busy-tick.c | 121 + > cpukit/libtest/t-test-busy.c | 58 + > cpukit/libtest/t-test-checks-eno.c | 10 +- > cpukit/libtest/t-test-checks-psx.c | 10 +- > cpukit/libtest/t-test-checks.c | 194 +- > cpukit/libtest/t-test-hash-sha256.c | 2 +- > cpukit/libtest/t-test-interrupt.c | 420 ++ > cpukit/libtest/t-test-malloc.c | 2 +- > cpukit/libtest/t-test-rtems-context.c | 8 +- > cpukit/libtest/t-test-rtems-fds.c | 5 +- > cpukit/libtest/t-test-rtems-heap.c | 6 +- > cpukit/libtest/t-test-rtems-measure.c | 2 +- > cpukit/libtest/t-test-rtems-objs.c | 100 +- > cpukit/libtest/t-test-rtems-posix-keys.c | 116 + > cpukit/libtest/t-test-rtems.c | 10 +- > cpukit/libtest/t-test-rtems.h | 62 + > cpukit/libtest/t-test-time.c | 2 +- > cpukit/libtest/t-test.c | 279 +- > cpukit/libtest/testbeginend.c | 2 +- > cpukit/libtest/testbusy.c | 89 +- > cpukit/libtest/testextension.c | 2 +- > cpukit/libtest/testparallel.c | 2 +- > cpukit/libtest/testrun.c | 87 + > cpukit/libtest/testwrappers.c | 2 +- > spec/build/cpukit/librtemstest.yml | 1 + > testsuites/ada/support/initimpl.h | 2 +- > testsuites/libtests/Makefile.am | 9 + > testsuites/libtests/block08/system.h | 2 +- > testsuites/libtests/block09/init.c | 2 +- > testsuites/libtests/block10/init.c | 2 +- > testsuites/libtests/capture01/system.h | 2 +- > testsuites/libtests/configure.ac | 1 + > testsuites/libtests/dl01/dl01-o1.c | 2 +- > testsuites/libtests/dl02/dl02-o1.c | 2 +- > testsuites/libtests/dl02/dl02-o2.c | 2 +- > testsuites/libtests/dl05/dl05-o5.cc | 2 +- > testsuites/libtests/dl06/dl06-o1.c | 2 +- > testsuites/libtests/dl06/dl06-o2.c | 2 +- > testsuites/libtests/dl07/dl07-o1.c | 2 +- > testsuites/libtests/dl07/dl07-o2.c | 2 +- > testsuites/libtests/dl07/dl07-o3.c | 2 +- > testsuites/libtests/dl07/dl07-o4.c | 2 +- > testsuites/libtests/dl07/dl07-o5.c | 2 +- > testsuites/libtests/dl08/dl08-o1.c | 2 +- > testsuites/libtests/dl08/dl08-o2.c | 2 +- > testsuites/libtests/dl08/dl08-o3.c | 2 +- > testsuites/libtests/dl08/dl08-o4.c | 2 +- > testsuites/libtests/dl08/dl08-o5.c | 2 +- > .../dl08/dl08-o6-123456789-123456789.c | 2 +- > testsuites/libtests/dl09/dl09-o1.c | 2 +- > testsuites/libtests/dl09/dl09-o2.c | 2 +- > testsuites/libtests/dl09/dl09-o3.c | 2 +- > testsuites/libtests/dl09/dl09-o4.c | 2 +- > testsuites/libtests/dl09/dl09-o5.c | 2 +- > testsuites/libtests/dl10/dl10-o1.c | 2 +- > testsuites/libtests/dl10/dl10-o2.c | 2 +- > testsuites/libtests/dl10/dl10-o3.c | 2 +- > testsuites/libtests/dl10/dl10-o4.c | 2 +- > testsuites/libtests/dl10/dl10-o5.c | 2 +- > testsuites/libtests/dl10/dl10-o6.c | 2 +- > testsuites/libtests/mathl/init.c | 2 +- > testsuites/libtests/ttest01/init.c | 6 +- > testsuites/libtests/ttest01/test-assert.c | 2 +- > testsuites/libtests/ttest01/test-checks.c | 25 +- > testsuites/libtests/ttest01/test-destructor.c | 2 +- > testsuites/libtests/ttest01/test-eno.c | 2 +- > testsuites/libtests/ttest01/test-example.c | 2 +- > testsuites/libtests/ttest01/test-fixture.c | 121 +- > testsuites/libtests/ttest01/test-leak.c | 2 +- > testsuites/libtests/ttest01/test-log.c | 2 +- > testsuites/libtests/ttest01/test-malloc.c | 2 +- > testsuites/libtests/ttest01/test-plan.c | 2 +- > testsuites/libtests/ttest01/test-psx.c | 2 +- > testsuites/libtests/ttest01/test-rtems.c | 2 +- > testsuites/libtests/ttest01/test-simple.c | 2 +- > testsuites/libtests/ttest01/test-step.c | 2 +- > .../libtests/ttest01/test-task-context.c | 2 +- > testsuites/libtests/ttest01/test-time.c | 2 +- > testsuites/libtests/ttest01/test-verbosity.c | 2 +- > testsuites/libtests/ttest02/init.c | 176 + > testsuites/libtests/ttest02/ttest02.doc | 11 + > testsuites/libtests/ttest02/ttest02.scn | 37 + > testsuites/psxtests/Makefile.am | 3 +- > testsuites/psxtests/psxbarrier01/main.c | 2 +- > testsuites/psxtests/psxconfig01/init.c | 2 +- > testsuites/psxtests/psxfenv01/init.c | 2 +- > testsuites/psxtests/psxintrcritical01/init.c | 135 +- > .../psxintrcritical01/psxintrcritical01.scn | 33 +- > testsuites/psxtests/psxinttypes01/init.c | 2 +- > testsuites/psxtests/psxndbm01/init.c | 2 +- > testsuites/psxtests/psxrwlock01/main.c | 2 +- > testsuites/samples/base_mp/system.h | 2 +- > testsuites/samples/base_sp/system.h | 2 +- > testsuites/samples/capture/system.h | 2 +- > testsuites/samples/cdtest/main.cc | 2 +- > testsuites/samples/cdtest/system.h | 2 +- > testsuites/samples/fileio/system.h | 2 +- > testsuites/samples/iostream/system.h | 2 +- > testsuites/samples/nsecs/system.h | 2 +- > testsuites/samples/paranoia/system.h | 2 +- > testsuites/samples/pppd/system.h | 2 +- > testsuites/samples/ticker/system.h | 2 +- > testsuites/smptests/smpatomic01/init.c | 2 +- > testsuites/smptests/smplock01/init.c | 2 +- > testsuites/smptests/smpmulticast01/init.c | 2 +- > testsuites/smptests/smpscheduler03/init.c | 2 +- > testsuites/smptests/smpscheduler05/init.c | 2 +- > testsuites/smptests/smpscheduler06/init.c | 2 +- > testsuites/smptests/smpscheduler07/init.c | 2 +- > testsuites/sptests/Makefile.am | 95 +- > testsuites/sptests/configure.ac | 2 - > testsuites/sptests/spheapprot/init.c | 2 +- > .../spintrcritical01/spintrcritical01.scn | 30 +- > .../spintrcritical01/spintrcritical01impl.h | 121 +- > .../spintrcritical02/spintrcritical02.scn | 30 +- > .../spintrcritical03/spintrcritical03.scn | 30 +- > .../spintrcritical04/spintrcritical04.scn | 30 +- > .../spintrcritical05/spintrcritical05.scn | 30 +- > testsuites/sptests/spintrcritical06/init.c | 2 - > .../spintrcritical06/spintrcritical06.doc | 28 - > .../spintrcritical06/spintrcritical06.scn | 6 - > .../spintrcritical06/spintrcritical06impl.h | 145 - > testsuites/sptests/spintrcritical07/init.c | 2 - > .../spintrcritical07/spintrcritical07.doc | 28 - > .../spintrcritical07/spintrcritical07.scn | 6 - > testsuites/sptests/spintrcritical08/init.c | 165 +- > .../spintrcritical08/spintrcritical08.scn | 30 +- > testsuites/sptests/spintrcritical09/init.c | 121 +- > .../spintrcritical09/spintrcritical09.scn | 28 +- > testsuites/sptests/spintrcritical10/init.c | 198 +- > .../spintrcritical10/spintrcritical10.scn | 33 +- > .../spintrcritical11/spintrcritical11.scn | 30 +- > .../spintrcritical11/spintrcritical11impl.h | 110 +- > .../spintrcritical12/spintrcritical12.scn | 30 +- > .../spintrcritical13/spintrcritical13.scn | 31 +- > .../spintrcritical13/spintrcritical13impl.h | 150 +- > .../spintrcritical14/spintrcritical14.scn | 31 +- > testsuites/sptests/spintrcritical15/init.c | 133 +- > .../spintrcritical15/spintrcritical15.scn | 34 +- > testsuites/sptests/spintrcritical16/init.c | 135 +- > .../spintrcritical16/spintrcritical16.scn | 31 +- > testsuites/sptests/spintrcritical18/init.c | 146 +- > .../spintrcritical18/spintrcritical18.scn | 30 +- > testsuites/sptests/spintrcritical20/init.c | 145 +- > .../spintrcritical20/spintrcritical20.scn | 32 +- > testsuites/sptests/spintrcritical21/init.c | 220 +- > .../spintrcritical21/spintrcritical21.scn | 27 +- > testsuites/sptests/spintrcritical21/system.h | 39 - > testsuites/sptests/spintrcritical22/init.c | 103 +- > .../spintrcritical22/spintrcritical22.scn | 20 + > testsuites/sptests/spintrcritical23/init.c | 117 +- > .../spintrcritical23/spintrcritical23.scn | 20 + > testsuites/sptests/spintrcritical24/init.c | 128 +- > .../spintrcritical24/spintrcritical24.scn | 32 + > .../spintrcritical_support/intrcritical.c | 171 - > .../spintrcritical_support/intrcritical.h | 52 - > testsuites/sptests/sptimecounter02/init.c | 2 +- > testsuites/sptests/sptimecounter03/init.c | 2 +- > testsuites/sptests/sptimecounter04/init.c | 2 +- > testsuites/support/include/buffer_test_io.h | 2 +- > testsuites/support/include/tmacros.h | 2 +- > testsuites/tmtests/tmfine01/init.c | 2 +- > testsuites/tmtests/tmonetoone/init.c | 2 +- > 169 files changed, 10028 insertions(+), 4543 deletions(-) > create mode 100644 cpukit/include/rtems/doxygen-appl-config.h > create mode 100644 cpukit/include/rtems/simple-test.h > delete mode 100644 cpukit/include/t.h > create mode 100644 cpukit/libtest/t-test-busy-tick.c > create mode 100644 cpukit/libtest/t-test-busy.c > create mode 100644 cpukit/libtest/t-test-interrupt.c > create mode 100644 cpukit/libtest/t-test-rtems-posix-keys.c > create mode 100644 cpukit/libtest/t-test-rtems.h > create mode 100644 cpukit/libtest/testrun.c > create mode 100644 testsuites/libtests/ttest02/init.c > create mode 100644 testsuites/libtests/ttest02/ttest02.doc > create mode 100644 testsuites/libtests/ttest02/ttest02.scn > delete mode 100644 testsuites/sptests/spintrcritical06/init.c > delete mode 100644 testsuites/sptests/spintrcritical06/spintrcritical06.doc > delete mode 100644 testsuites/sptests/spintrcritical06/spintrcritical06.scn > delete mode 100644 testsuites/sptests/spintrcritical06/spintrcritical06impl.h > delete mode 100644 testsuites/sptests/spintrcritical07/init.c > delete mode 100644 testsuites/sptests/spintrcritical07/spintrcritical07.doc > delete mode 100644 testsuites/sptests/spintrcritical07/spintrcritical07.scn > delete mode 100644 testsuites/sptests/spintrcritical21/system.h > delete mode 100644 testsuites/sptests/spintrcritical_support/intrcritical.c > delete mode 100644 testsuites/sptests/spintrcritical_support/intrcritical.h > _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
