Hello Chris,
On 19/03/2019 06:17, Chris Johns wrote:
Hi,
I am not sure how this fits in with what we have so before I can support any
changes of this scale and direction I need to understand:
1. Are all existing tests to be converted to this new framework?
the main purpose for the new framework is to write unit, integration and
validation tests for the RTEMS SMP pre-qualification activity. I am not
sure what we do with the existing test suite. One option is to pick up
existing tests and covert them. How this is organized should be
discussed in a separate thread.
2. How does this effect the existing eco-system support such as the `rtems-test`
command and the documentation around that command?
The rtems-test command just looks at the begin and end of test messages.
You can still use it with the new framework, see the test output of the
example test:
*** BEGIN OF TEST TTEST 1 ***
*** TEST VERSION: 5.0.0.286e9354e008b08983e6390a68f8ecc5071de069
*** TEST STATE: EXPECTED-PASS
*** TEST BUILD: RTEMS_DEBUG RTEMS_NETWORKING RTEMS_POSIX_API RTEMS_SMP
*** TEST TOOLS: 7.4.0 20181206 (RTEMS 5, RSB
e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f)
A:ttest01
S:Platform:RTEMS
S:Compiler:7.4.0 20181206 (RTEMS 5, RSB
e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f)
S:Version:5.0.0.286e9354e008b08983e6390a68f8ecc5071de069
S:BSP:erc32
S:RTEMS_DEBUG:1
S:RTEMS_MULTIPROCESSING:0
S:RTEMS_POSIX_API:1
S:RTEMS_PROFILING:0
S:RTEMS_SMP:1
B:example
P:0:0:UI1:test-example.c:5
F:1:0:UI1:test-example.c:6:test fails
F:*:0:UI1:test-example.c:8:quiet test fails
P:2:0:UI1:test-example.c:9
F:3:0:UI1:test-example.c:10:step test fails
F:4:0:UI1:test-example.c:11:this is a format string
E:example:N:5:F:4:D:0.001000
Z:ttest01:C:1:N:5:F:4:D:0.003000
*** END OF TEST TTEST 1 ***
What you get in addition is a structured output in the middle. This can
be used to generated more detailed reports. This is another topic. The
test framework should just enable you to easily parse the test output
and do something with it.
3. What does 'T' in THE_T_TEST_FRAMEWORK_H stand for? I prefer we prefix RTEMS_
where it makes sense.
The 'T' is just a random name which associates with testing. I searched
a bit for <t.h> and a T_ prefix and didn't found an existing project.
So, there should be no name conflicts. It is short, so this is good for
typing. The T Test Framework is portable. It runs also on Linux,
FreeBSD, and MSYS2. I will probably add it also as a stand a lone
project to github.
4. I see in another email you post a Sphinx generated report. What are those
tests and what is used to capture and create that report and will this in time
include all existing tests?
I wrote a very simple and stupid Python script to extract this
information from the test output just to evaluate if the output format
makes sense. The output is from some example tests I used to test the
framework. For the pre-qualification we need test plans, tests, test
reports, traceability to requirements and test verification. The
framework enables you to efficiently write test code and generation of
easy to parse output.
Chris
On 14/3/19 9:24 pm, Sebastian Huber wrote:
Update #3199.
---
cpukit/Makefile.am | 11 +
cpukit/headers.am | 1 +
cpukit/include/t.h | 2128 +++++++++++++++++++++++++++++++++
cpukit/libtest/t-test-checks-eno.c | 145 +++
cpukit/libtest/t-test-checks-psx.c | 17 +
--
Sebastian Huber, embedded brains GmbH
Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.
Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
#!/usr/bin/env python3
# SPDX-License-Identifier: BSD-2-Clause
#
# Copyright (C) 2018 embedded brains GmbH
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
import re
import sys
import matplotlib.pyplot as plt
import astropy.stats
class Context:
def __init__(self):
self.testSuiteName = "?"
self.testCaseName = "?"
self.measureName = "?"
self.measureVariant = "?"
self.measureLoad = 0
self.measureSamples = []
self.log = ""
self.failures = ""
def removeLastLogLine(self):
ctx.log = ctx.log[: ctx.log.rindex("\n", 0, ctx.log.rindex("\n")) + 1]
def keepOnlyLastLogLine(self):
ctx.log = ctx.log[ctx.log.rindex("\n", 0, ctx.log.rindex("\n")) + 1 :]
class LineProcessor:
def __init__(self, r):
self.r = re.compile(r)
def match(self, line):
return self.r.match(line)
class TestSuiteBegin(LineProcessor):
def __init__(self):
super(TestSuiteBegin, self).__init__(r"^A:(.*)$")
def process(self, ctx, m):
ctx.testSuiteName = m.group(1)
name = "Test Suite - " + ctx.testSuiteName
print("");
print(name);
print("=" * len(name));
class TestSuiteEnd(LineProcessor):
def __init__(self):
super(TestSuiteEnd, self).__init__(r"^Z:(.*)$")
def process(self, ctx, m):
return
class TestCaseBegin(LineProcessor):
def __init__(self):
super(TestCaseBegin, self).__init__(r"^B:(.*)$")
def process(self, ctx, m):
ctx.testCaseName = m.group(1)
ctx.keepOnlyLastLogLine()
ctx.failures = ""
name = "Test Case - " + ctx.testCaseName
print("");
print(name);
print("-" * len(name));
class TestCaseEnd(LineProcessor):
def __init__(self):
super(TestCaseEnd, self).__init__(r"^E:(.*):N:(.*):F:(.*):D:(.*)$")
def process(self, ctx, m):
print("")
if int(m.group(2)) > 0:
print("In this test case " + m.group(2) + " actual test steps were executed.")
if int(m.group(3)) > 0:
print("A total of " + m.group(3) + " test steps failed.")
else:
print("No test step failed.")
else:
print("This test case executed no test steps.")
print("The test case execution time was " + m.group(4) + "s.")
if len(ctx.failures) > 0:
print("")
print(".. csv-table:: Test Failures")
print(" :header: \"Step\", \"File\", \"Line\", \"Failure Message\"")
print(" :widths: 8, 30, 8, 54")
print("")
print(ctx.failures.strip("\n"))
print("")
print(".. code-block:: none")
print(" :caption: Test Log")
print("")
print(ctx.log.strip("\n"))
class TestStep(LineProcessor):
def __init__(self):
super(TestStep, self).__init__(r"^([PF]):([0-9*]+):([0-9]+):([^:]+):([^:]+):([0-9*]+):?(.*)$")
def process(self, ctx, m):
if m.group(1) == "F":
failure = " \"" + m.group(2) + "\", \"" + m.group(5) + "\", \"" + m.group(6) + "\""
if m.group(7):
failure += ", \"``" + m.group(7) + "``\""
else:
failure += ", \"\""
failure += "\n"
ctx.failures += failure.replace("*", "\\*")
class MeasureBegin(LineProcessor):
def __init__(self):
super(MeasureBegin, self).__init__(r"^M:B:(.*)$")
def process(self, ctx, m):
ctx.measureName = m.group(1)
ctx.measureVariant = "?"
ctx.measureLoad = 0
ctx.measureSamples = []
class MeasureEnd(LineProcessor):
def __init__(self):
super(MeasureEnd, self).__init__(r"^M:E:(.*)$")
def process(self, ctx, m):
var = ctx.measureVariant
base = ctx.testSuiteName + "-" + ctx.testCaseName + "-" + ctx.measureName + "-" + ctx.measureVariant
if ctx.measureLoad > 0:
load = str(ctx.measureLoad)
base += "-" + load
var += " " + load
name = "Runtime Measurement - " + ctx.measureName + " (" + var + ")"
print("");
print(name);
print("~" * len(name));
if len(ctx.measureSamples) == 0:
return
print("");
print(".. image:: ../images/contributor/" + base + ".*")
print(" :width: 400")
print(" :align: center")
mx = ctx.measureSamples[-1]
if mx < 1e-6:
unit = "ns"
ctx.measureSamples = [s * 1e9 for s in ctx.measureSamples]
elif mx < 1e-3:
unit = "μs"
ctx.measureSamples = [s * 1e6 for s in ctx.measureSamples]
else:
unit = "s"
mi = ctx.measureSamples[0]
mx = ctx.measureSamples[-1]
fig, (ax1, ax2) = plt.subplots(2, 1)
ax1.set_title(ctx.measureName + " (" + var + ")")
ax1.hist(ctx.measureSamples, 100, cumulative = True)
ax1.set_xlim(mi, mx)
ax1.set_ylabel("Sample Count")
# needs more than 32GiB of RAM: dx, bins = astropy.stats.knuth_bin_width(ctx.measureSamples, True)
dx, bins = astropy.stats.freedman_bin_width(ctx.measureSamples, True)
ax2.hist(ctx.measureSamples, bins)
ax2.set_xlim(mi, mx)
ax2.set_xlabel("Runtime [%s]" % (unit))
ax2.set_ylabel("Sample Count")
plt.savefig("../../images/contributor/" + base + ".png")
plt.savefig("../../images/contributor/" + base + ".pdf")
#plt.show()
class MeasureVariant(LineProcessor):
def __init__(self):
super(MeasureVariant, self).__init__(r"^M:V:(.*)")
def process(self, ctx, m):
ctx.measureVariant = m.group(1)
class MeasureLoad(LineProcessor):
def __init__(self):
super(MeasureLoad, self).__init__(r"^M:L:(.*)")
def process(self, ctx, m):
ctx.measureLoad = int(m.group(1))
class MeasureSample(LineProcessor):
def __init__(self):
super(MeasureSample, self).__init__(r"^M:S:(.*):(.*)$")
def process(self, ctx, m):
ctx.measureSamples.extend([float(m.group(2))] * int(m.group(1)))
ctx.removeLastLogLine()
class CatchAll(LineProcessor):
def __init__(self):
super(CatchAll, self).__init__(r"^(.*)$")
def process(self, ctx, m):
return
print(".. SPDX-License-Identifier: CC-BY-SA-4.0")
print(".. Copyright (c) 2018 embedded brains GmbH")
print("")
print("Test Reports")
print("************")
lineProcessors = [TestSuiteBegin(), TestSuiteEnd(), TestCaseBegin(), TestCaseEnd(), TestStep(), MeasureBegin(), MeasureEnd(), MeasureVariant(), MeasureLoad(), MeasureSample(), CatchAll()]
ctx = Context()
for line in sys.stdin:
line = line.strip("\n")
if len(line) == 0:
continue
ctx.log += " " + line + "\n"
for lp in lineProcessors:
m = lp.match(line)
if m:
lp.process(ctx, m)
break
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
