On 30/6/2022 4:34 pm, Sebastian Huber wrote:
> On 30/06/2022 07:58, Sebastian Huber wrote:
>> On 29/06/2022 17:54, Kinsey Moore wrote:
>>> On 6/29/2022 04:34, Sebastian Huber wrote:
>>>> On 29/06/2022 11:20, Chris Johns wrote:
>>>>>
>>>>>> On 29 Jun 2022, at 4:42 pm, Sebastian Huber
>>>>>> <sebastian.hu...@embedded-brains.de> wrote:
>>>>>>
>>>>>> On 29/06/2022 08:40, Sebastian Huber wrote:
>>>>>>> Report the same data in JSON and YAML reports. Do not report redundant
>>>>>>> information.
>>>>>>> Update 4671.
>>>>>>
>>>>>> This patch changes the JSON reports. Are there already consumers for the
>>>>>> JSON reports so that we have to be backward compatible?
>>>>>
>>>>> Could compatibility be added back in if this proves to be an issue?
>>>>>
>>>>> I am wondering if that could be considered if comparability becomes an
>>>>> issue.
>>>>
>>>> The JSON report was added by:
>>>>
>>>> commit 5fcaf843bdb491e2da14dc1af61806ac6d20bae5
>>>> Author: Kinsey Moore <kinsey.mo...@oarcorp.com>
>>>> Date: Wed Aug 21 16:34:12 2019 +0000
>>>>
>>>> Add JSON log generation
>>>>
>>>> Add log formatter hooks and JSON log formatter to the test
>>>> infrastructure
>>>> for consumption by automated processes or report generators.
>>>>
>>>>
>>>> I am not sure if these automated processes or report generators already
>>>> exist.
>>>>
>>>> The existing attribute names are quite inconsistent, for example "Command
>>>> Line", "passed_count", "wrong-version_count". I would use lower case only
>>>> with "-" as a separator. The JSON report should contain all information of
>>>> a
>>>> test run.
>>>>
>>>> The new report looks like this:
>>>>
>>>> {
>>>> "command-line": [
>>>> "/opt/rtems/6/bin/rtems-test",
>>>> "--rtems-bsp=xilinx_zynq_a9_qemu",
>>>> "--report-format=json",
>>>> "--report-path=report",
>>>> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/"
>>>> ],
>>>> "end-time": "2022-06-28T14:08:47.595131",
>>>> "host": "Linux-5.3.18-150300.59.76-default-x86_64-with-glibc2.3.4
>>>> (Linux
>>>> lupus 5.3.18-150300.59.76-default #1 SMP Thu Jun 16 04:23:47 UTC 2022
>>>> (2cc2ade) x86_64 x86_64)",
>>>> "python": "3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]",
>>>> "reports": [
>>>> {
>>>> "arch": "arm",
>>>> "bsp": "xilinx_zynq_a9_qemu",
>>>> "command-line": "qemu-system-arm -no-reboot -nographic -net
>>>> none
>>>> -serial null -serial mon:stdio -M xilinx-zynq-a9 -m 256M -kernel
>>>> build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>> "end-time": "2022-06-28T12:08:48.161691+00:00",
>>>> "executable":
>>>> "build/arm/xilinx_zynq_a9_qemu/testsuites/samples/cdtest.exe",
>>>> "executable-sha512":
>>>> "413bc8a506053c95a0e1d93942cce9fd0d653f7b3cde42623085ce147aea324f4baeff7ce0102008322fb1341bc2b38defd3925a64144e47e13cc5f32185bf78",
>>>>
>>>> "output": [
>>>> "qemu-system-arm: warning: nic cadence_gem.0 has no peer",
>>>> "qemu-system-arm: warning: nic cadence_gem.1 has no peer",
>>>> "",
>>>> "",
>>>> "*** BEGIN OF TEST CONSTRUCTOR/DESTRUCTOR ***",
>>>> "*** TEST VERSION:
>>>> 6.0.0.3302b72754df5f37214e86dd68522189857772c7",
>>>> "*** TEST STATE: EXPECTED_PASS",
>>>> "*** TEST BUILD: RTEMS_DEBUG RTEMS_POSIX_API",
>>>> "*** TEST TOOLS: 12.1.1 20220622 (RTEMS 6, RSB
>>>> bf3f66c29a046ec1c5029e4f919b525947df6245, Newlib e7ffbdb)",
>>>> "GLOBAL: Hey I'm in base class constructor number 1 for
>>>> 0x214474.",
>>>> "GLOBAL: Hey I'm in base class constructor number 2 for
>>>> 0x214480.",
>>>> "GLOBAL: Hey I'm in derived class constructor number 3 for
>>>> 0x214480.",
>>>> "LOCAL: Hey I'm in base class constructor number 4 for
>>>> 0x228864.",
>>>> ...
>>>>
>>>>
>>>> "WRoZqtoO3A8AAAABDAAAAJkUSmRGNOBaaHW1UwAAoQGQ////AAAAAQwAAABck8Bx+K/znDWWTEcA",
>>>>
>>>> "AKEB2P///wAAAAEMAAAAVP6TRqQZY+4+srvAAAChAfD///8AAAABDAAAAMXOxS0Rhzqx6Old2wAA",
>>>>
>>>> "oQH4////AAAAAQwAAADJyx1ZCnZ4l+m/XkMAAKEB0P///wAAAAEMAAAAN3+9YAwMW8gTHIoPAACh",
>>>>
>>>> "Adj///8AAAABDAAAABtxsj3zXZF/+UqzfAAAoQHw////AAAAAQwAAAAwMDMeE7mphT6yu8AAAKEB",
>>>> "8P///wAAAAEMAAAAr2rLCcwzVnf5SrN8AAChAfD///8AAAAA",
>>>> "*** END OF GCOV INFO BASE64 ***",
>>>> ""
>>>> ],
>>>> "result": "passed",
>>>> "start-time": "2022-06-28T12:08:47.721822+00:00"
>>>> }
>>>> ],
>>>> "start-time": "2022-06-28T14:08:47.595131"
>>>> }
>>>>
>>> There are no publicly available report generators for the existing format,
>>> but I do have one I use to generate STR documents for delivery using
>>> mustache
>>> templates and further processing. The existing format was designed to be
>>> easily consumable by that and other simple templating mechanisms by
>>> providing
>>> structure (test subsets) and precalculating values that would otherwise be
>>> implicit in the data. Changes to the names of various fields should be easy
>>> to accommodate, but a more significant restructure is going to present
>>> problems continuing with any simple templating mechanism and would instead
>>> require a more comprehensive document generator for all cases.
>>>
>>> I'm not necessarily opposed to this change as it significantly simplifies
>>> the
>>> report generation code, but it will definitely require some rework in my
>>> processes. I can't speak for any users of the YAML report format, but it was
>>> committed over a year and a half ago.
>>
>> Ok, I will bring back the redundant data and use a consistent naming.
>>
>> The YAML format was added for the RTEMS pre-qualification project, however,
>> it
>> is pretty useless since the YAML processing is very slow in Python. Also the
>> test report generators need to be rewritten. This patch is a part of this
>> work.
>
> Do you really need this "test_groups" stuff?
>
> "test_groups": [
> {
> "index": 1,
> "name": "benchmarks",
> "results": [
> {
> "name": "dhrystone",
> "result": "benchmark"
> },
>
> From my point of view the JSON/YAML reports should not satisfy every possible
> consumer directly.
How do you determine the list you are satisfying? I can see this argument
working in two directions.
If there a technical issue with providing this data?
Does the output provide format metadata to determine the version of the output?
Chris
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
