Sorry for the even longer delay between replies. I've been on vacation.
I think clarifying a few things will be helpful.
barectf takes a yaml file describing a bunch of data types that can be
logged. The python tool will generate a c file that has function like
log_my_data_type(MyDataType* data, Stream * stream);
and a CTF descriptor
Calling that function will serialize data, and send it down stream.
Stream is conceptual trace output buffer. Barectf interacts with the
stream through a a set of functions, like write_bytes, and
is_stream_full? ( I made up these names for clarity).
The bytes written to stream will be in CTF format. Therefore, putting
them in a file with the generated descriptor will allow you to read the
stream with babeltrace or another tool that supports CTF files.
The README on the GitHub page is actually quite good at getting you up
to speed quickly for areas where my explanation is lacking.
So I'm thinking barectuf could be stuck into RTEMS and used as the
serialization mechanism for all trace facilities. For the trace linker,
we could define a set of data types for function entry, function exit,
etc. Rather than using printk to send out the trace, we would use barectf.
I've also spent some time looking at the lttng-live protocol. If RTEMS
did adopt barectf, or another CTF serialization solution, it would open
up the possibility of connecting to an running RTEMS application
directly via TCP/IP using babeltrace or tracecompass and see traces in
real time.
The lttng-live protocol is not overly complicated, and probably could be
implemented in a few weeks. I've almost done with a rough proof of
concept server that has a single hard coded stream and send pre-captured
trace data.
On 01/03/16 18:49, Chris Johns wrote:
> On 12/22/15 02:38, Isaac Gutekunst wrote:
>>
>>
>> On 12/20/2015 08:02 PM, Chris Johns wrote:
>>> On 19/12/2015 6:36 AM, isaac.gutekunst wrote:
>>>>
>>>> We're interested in trace some application data in our RTEMS project.
>>>> Here's a bunch of background and thoughts:
>>>>
>>>> I've read the following pages:
>>>> * https://devel.rtems.org/wiki/Developer/Tracing
>>>> * https://devel.rtems.org/wiki/Developer/Tracing/Trace_Linker
>>>>
>>>> I think I have an understanding of the current system as documented.
>>>> Tracing user defined data types seems to be missing. I'm looking for
>>>> something more like barectf. Is there any such support available,
or any
>>>> plans to support it? I don't want to reinvent the wheel.
>>>
>>> User defined data is not directly support but can be added if you add a
>>> custom generator. There is not a great deal of documentation about how
>>> to do this. An example generator is:
>>>
>>> https://git.rtems.org/rtems-tools/tree/linkers/rtld-trace-buffer.ini
>>>
>> Cool. I'll take a closer look at this.
>>> You could add anything specific you want. It is not a great
solution but
>>> tracing into data being passed by pointers is difficult.
>>>
>> Definitely. You quickly come to the great serialization problem, and are
>> bound to reinvent the wheel a few times.
>
> Yes and SMP makes this even more complicated. We need to ensure we
have a low overhead solution. Work in this area has been done by Joel
and Jennifer.
>
I haven't figured out what the Capture Engine does exactly. Does it
simply buffer the printk output?
Apologies for not looking into the code myself. It looks like the
Capture Engine outputs data in some format that babletrace can
understand and convert into CTF.
I'm proposing that we integrate barectf into the Capture Engine (or
elsewhere), so the Capture Engine directly outputs CTF. However, I may
be confused about what's going on.
I'm presuming that the instrumented application has calls to functions
inserted for entry and exit to the functions a user wants to trace.
These functions generate some data, and then the Capture Engine stores
this and makes it available later.
I'm unclear about what this data is, and how it gets to the capture
engine. Where is the code for this located?
>>> You would need to use the various options to get a copy of the trace
>>> wrapper file and then get the generator to see the headers and various
>>> data you want.
>>>
>>>> If there isn't support already, maybe some parts of the trace
>>>> infrastructure can be reused, notably the backed for storing the data,
>>>> and/or sending it out over the network. Are there plans to make the
>>>> Trace Linker natively output CTF data?
>>>
>>> We would like to support CTF output. Getting this done would need
>>> developer funding.
>>
>> We would like tracing for at least user defined data, and will be
>> developing that functionality. If we could do so in a way that helps the
>> existing RTEMS trace, we'd love to.
>>
>>>
>>>> Perhaps barectf could be used for this purpose.
>>>
>>> I do not know what this. Do you have a reference?
>>>
>> https://github.com/efficios/barectf
>>
>
> Thanks
>
>> It takes in a YAML file, and generates a C file with functions that
>> allow tracing user defined structs. It will output those to a custom
>> "backend" in serialized form. The documentation is pretty good. I've
>> used in briefly in a bare metal PIC32 project with success.
>>
>> A potential plan of action:
>> 1) Make a barectf backend for RTEMS. (This could re-use existing
>> features). Probably package/make it available via RSB.
>
> Is this just for data or does it handle instrumenting functions? I
have not looked at barectf.
>
It's really just for serializing arbitrary structs into CTF data.
>> 2) Update the trace linker to use it (I don't really know what this
>> means just yet).
>
> Are 1) and 2) has the same thing or part of the same piece of work?
>
> The capture engine Joel and Jennifer updated supports SMP logging.
Can this be used to do this?
>
Again, maybe once I have a better understanding of the Capture Engine,
I'll know what I'm actually proposing
> My goal is to see all support integrated into the same trace
framework so our users have a single place for trace support. The trace
framework can have multiple parts, they just need to coexist in a single
framework.
>
>> 3) Make a mechanism/ document a standard way to integrate trace with a
>> user application. I'd imagine this could involve a RSB package, although
>> it's rather simple (yaml file becomes a C and H file that needs to be
>> linked against a 'backend').
>
> How does this relate to the RTEMS Trace linker and the function
wrapping that it currently implements? Can the RTEMS Trace Linker be the
"user interface" to bring all this together? This would then provide a
single user interface for creating a traceable RTEMS executable.
>
I believe so. This would mostly be pushing CTF serialization into RTEMS
and updating the Trace Linker to insert calls to barectf generated
functions to log data, instead of the existing printk calls.
User level tracing would just be generation of additional barectf trace
functions, outputting data to streams managed by the Capture Engine (I
think).
>>
>> 4) For the ARM architecture, using the ITM to output data could be very
>> useful. This could fit somewhere into this picture.
>>
>
> Transports should be a separate part. The capture engine should be
creating buffers and a separate module manages the user interfacing and
data extraction. Currently thee is just a cli. I am not across the ITM
hardware and how you manager the user interactions.
>
Using the ITM is potentially a little painful. It can essentially be
thought of as a UART.
If the Capture Engine just creates buffers, it shouldn't be too hard to
consume them in delayed realtime and send them out the ITM, or anything
really.
>> I would be doing most or all of the work here, unless of course anyone
>> wants to help.
>>
>> I'm looking for ideas and suggestions. If there is some functionality of
>> direction you'd like us to pursue, let me know.
>>
>
> Sorry about the delayed response, I have been away for a few weeks.
>
> Chris
On 01/03/2016 06:49 PM, Chris Johns wrote:
On 12/22/15 02:38, Isaac Gutekunst wrote:
On 12/20/2015 08:02 PM, Chris Johns wrote:
On 19/12/2015 6:36 AM, isaac.gutekunst wrote:
We're interested in trace some application data in our RTEMS project.
Here's a bunch of background and thoughts:
I've read the following pages:
* https://devel.rtems.org/wiki/Developer/Tracing
* https://devel.rtems.org/wiki/Developer/Tracing/Trace_Linker
I think I have an understanding of the current system as documented.
Tracing user defined data types seems to be missing. I'm looking for
something more like barectf. Is there any such support available, or
any
plans to support it? I don't want to reinvent the wheel.
User defined data is not directly support but can be added if you add a
custom generator. There is not a great deal of documentation about how
to do this. An example generator is:
https://git.rtems.org/rtems-tools/tree/linkers/rtld-trace-buffer.ini
Cool. I'll take a closer look at this.
You could add anything specific you want. It is not a great solution but
tracing into data being passed by pointers is difficult.
Definitely. You quickly come to the great serialization problem, and are
bound to reinvent the wheel a few times.
Yes and SMP makes this even more complicated. We need to ensure we have
a low overhead solution. Work in this area has been done by Joel and
Jennifer.
You would need to use the various options to get a copy of the trace
wrapper file and then get the generator to see the headers and various
data you want.
If there isn't support already, maybe some parts of the trace
infrastructure can be reused, notably the backed for storing the data,
and/or sending it out over the network. Are there plans to make the
Trace Linker natively output CTF data?
We would like to support CTF output. Getting this done would need
developer funding.
We would like tracing for at least user defined data, and will be
developing that functionality. If we could do so in a way that helps the
existing RTEMS trace, we'd love to.
Perhaps barectf could be used for this purpose.
I do not know what this. Do you have a reference?
https://github.com/efficios/barectf
Thanks
It takes in a YAML file, and generates a C file with functions that
allow tracing user defined structs. It will output those to a custom
"backend" in serialized form. The documentation is pretty good. I've
used in briefly in a bare metal PIC32 project with success.
A potential plan of action:
1) Make a barectf backend for RTEMS. (This could re-use existing
features). Probably package/make it available via RSB.
Is this just for data or does it handle instrumenting functions? I have
not looked at barectf.
2) Update the trace linker to use it (I don't really know what this
means just yet).
Are 1) and 2) has the same thing or part of the same piece of work?
The capture engine Joel and Jennifer updated supports SMP logging. Can
this be used to do this?
My goal is to see all support integrated into the same trace framework
so our users have a single place for trace support. The trace framework
can have multiple parts, they just need to coexist in a single framework.
3) Make a mechanism/ document a standard way to integrate trace with a
user application. I'd imagine this could involve a RSB package, although
it's rather simple (yaml file becomes a C and H file that needs to be
linked against a 'backend').
How does this relate to the RTEMS Trace linker and the function wrapping
that it currently implements? Can the RTEMS Trace Linker be the "user
interface" to bring all this together? This would then provide a single
user interface for creating a traceable RTEMS executable.
4) For the ARM architecture, using the ITM to output data could be very
useful. This could fit somewhere into this picture.
Transports should be a separate part. The capture engine should be
creating buffers and a separate module manages the user interfacing and
data extraction. Currently thee is just a cli. I am not across the ITM
hardware and how you manager the user interactions.
I would be doing most or all of the work here, unless of course anyone
wants to help.
I'm looking for ideas and suggestions. If there is some functionality of
direction you'd like us to pursue, let me know.
Sorry about the delayed response, I have been away for a few weeks.
Chris
_______________________________________________
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
