Stefan,
I'm sorry to be so long in responding. Your response was very helpful,
but I had to suddenly change my focus to other projects. I've been
looking at it again.
I took the easy way out for now and attempted to name the codeobj
objects deterministically, as you suggested at the end of your message:
https://github.com/SyamGadde/cython/commit/b56135154f546eec34d88342dc1698cf1803492e
The assumption is that only one codeobj is ever generated for each named
function within a module. I'm afraid I may have taken a hammer to the
code instead of a chisel, so your comments are welcome.
This is actually sufficient for using the kernprof/line_profiler
decorator @profile as-is on Cython-generated non-cdef functions/methods
as long as you are using the following cython directives:
--directive profile=true
--directive linetrace=true
--directive binding=true
The binding=true is necessary so that the function objects get the
func_code attribute set to the codeobj above. Actually line_profiler
just uses func_code as a unique identifier for the function and doesn't
really look at the contents of func_code except to determine if a
function is a generator. If there is another attribute available for
uniquely identifying function objects that don't have func_code, then
line_profiler could potentially use that too and it could work on cdef-
and cpdef-declared functions...
...except for the fact that cdef and cpdef functions don't accept
arbitrary decorators, which is the typical way to use
kernprof/line_profiler. But I'm pretty happy to be able to use
kernprof/line_profiler on compiled pure-Python functions for now.
Thanks for your help!
-syam
On 02/27/2014 04:06 PM, Stefan Behnel wrote:
> Hi!
>
> Syam Gadde, 27.02.2014 16:22:
>> I tried using line tracing in conjunction with line_profiler/kernprof to
>> attempt to see if line-based profiling works in Cython. I think it's
>> pretty close. But I think the problem is that line_profiler is getting
>> a different PyCodeObject when it wraps the function and when it actually
>> gets the trace call. It adds the initial code object to a map, and
>> later when it gets the trace call, decides that the trace call is not
>> something it needs to pay attention to because the new code object that
>> it gets is not the same as the original one.
>>
>> The first code object is created at function declaration by
>> __Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to
>> a variable __pyx_k_codeobj_NNN. The second code object is created,
>> essentially, during the first entry into the function (in
>> __Pyx_TraceCall, via __Pyx_TraceSetupAndCall). It seems that setting
>> __pyx_frame_code to the initial code object before calling TraceCall()
>> would fix this.
> That's a part of it, yes. Here's another bit in the puzzle:
>
> https://github.com/cython/cython/pull/93
>
> The problem is that the code objects that we currently create along the way
> have a fixed Python code line number (because that was the simplest way to
> get it working). The two changes above will get us pretty far. What remains
> to be done then is to enable line number calculation at runtime by
> emulating byte code offsets in the frame object instead of using absolute
> line numbers. The pull request refers to CPython's
> Objects/lnotab_notes.txt, which has some details on the inner workings.
>
> Properly calculating line numbers at runtime would also have other nice
> side effects, because it would remove the need to create multiple code
> objects for a single function in the first place. So it's very desirable.
>
>
>> Is this easy to do? I'd do it myself, but I'd need help figuring out
>> how to get the name of the appropriate __pyx_k_codeobj_NNN variable from
>> within FuncDefNode.generate_function_definitions(), which calls
>> put_trace_call().
> The best way is usually to run it through a debugger and see what you can
> get your hands on. :) The constant names (__pyx_k_...) are generated on the
> fly at C code generation time, so you can't get them before that. Two
> possible solutions: either change the way how the C names of code objects
> are being generated (e.g. by making their C name depend on the mangled C
> name of the function so that they can be deterministically named at
> analysis time), or make the code object node remember its generated
> constant name and reuse it in other places.
>
> As you can see, it's not entirely trivial overall, but we've already been
> piling up the bits and pieces so that the only remaining effort is to put
> everything together. If you could give it a try, I'm sure you'd make a lot
> of people happy with it.
>
> Stefan
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinf
