Dear David,
On 14 September 2010 18:39, David Malcolm <[email protected]> wrote:
> On Tue, 2010-09-14 at 11:20 +0200, Dag Sverre Seljebotn wrote:
>
> (various comments inline)
>
>> mark florisson wrote:
>> > Hello,
>> >
>> > There currently is no proper debugger available to debug Cython
>> code.
>> > The current way we debug Cython code is:
>> > - the print statement
>> > - a C debugger
>> > - pdb
>> >
>> > None of the aforementioned methods is very powerful and time
>> effective
>> > at the same time. For instance, 'print' requires recompilation and
>> is
>> > not so powerful, pdb doesn't provide access to anything C-level and
>> > gdb often requires a lot of digging through generated C code. This
>> is
>> > why I feel that there is a need to a proper Cython debugger.
>> >
>> > This debugger should have, for starters, the following features:
>> > - Cython variable/type/function name correspondence (the ability to
>> > set breakpoints for Cython function/method names, the ability to
>> > inspect variables by their Cython name (be it C or Python
>> variables),
>> > etc)
>> > - A line number and source code correspondence with the Cython code
>> > (ability to view the code at the C and Cython abstraction levels)
>> > - Be able to inspect Python code, print python backtraces, etc
>> > (there's already Misc/gdbinit in the Python source distribution and
>> > there is the EasierPythonDebugging project)
>
> I wrote the gdb7 hooks that I think you're referring to:
> http://fedoraproject.org/wiki/Features/EasierPythonDebugging
>
> This is in Python 2.7, Python 3.2a1, Fedora 13's builds of 2.6/3.1, and
> in some other distros.
>
> The code in question can be seen here:
> http://svn.python.org/view/python/trunk/Tools/gdb/libpython.py
> along with a test suite here:
> http://svn.python.org/view/python/trunk/Lib/test/test_gdb.py
Thanks for pointing us to the source code, I didn't realize this was
included in the CPython source code distribution.
>> > Such a debugger would need some kind of information, namely it would need:
>> > - a mapping from Cython line numbers to a range of C line numbers
>> > - a mapping from Cython names to mangled C names
>> > - naming information with regard to scope
>> >
>> > My plan is to extend GDB with Python and introduce new Cython commands
>> > that would deal with these issues. It would use the information
>> > exported by the Cython compiler to realize this (the easiest and most
>> > portable way would probably be to write it to a separate file in some
>> > format (xml or json come to mind)).
>> >
>
> [snip]
>
>> >
>> I think it is very likely that we would love to ship the necesarry
>> modifications (once stable) as part of main Cython. If your Cython
>> patches can become part of mainline GDB then there's no question about
>> it at all, but even if you have to maintain a forked GDB I'm all for it
>> (and in that case we could host the forked GDB on cython.org as well and
>> so on).
>
> You shouldn't need to fork GDB; gdb 7 embeds python scripting.
>
> Unfortunately the exact methods you'll have in the gdb python classes in
> any given Linux distribution's build of gdb will vary (at risk of
> further blowing my own trumpet, much of this python integration was done
> by colleagues of mine at RH, so the Fedora build of GDB tends to be
> slightly ahead of FSF GDB in this regard).
>
> You may run into issues with optimization: under the wrong conditions,
> GCC can optimize away vital variables in such a way that GDB can't read
> them. I run into this with optimization on 64-bit with the hooks I
> wrote for python: it can't read the PyFrameObject *f var in
> PyEval_EvalFrameEx, arguably the most important local within the CPython
> runtime :( (again, my RH colleagues are working on this, but it will
> involve adding new functionality to DWARF).
>
> Having said all that, I implemented various CPython hooks using gdb7:
> py-list
> py-up/py-down
> py-print
> and these work with an unoptimized build of python.
>
> It ought to be possible to do something similar with cython code. It
> may not even be necessary to modify cython: perhaps some searching for
> locals named "__pyx_*" iirc would get you 70% of the way there?
Although that sounds like a wonderful idea, I think there are also
issues with that. One issue is that a user must be able to set Cython
breakpoints before the Cython module would be loaded, and for that the
symbol name would be needed beforehand. Also, I don't know if these
mangled names are consistent now and in the future and if you would be
able to unambiguously associate a Cython variable name with a mangled
name.
> I can attest that having the prettyprinters enabled does make it much
> easier to debug cython code: all of the PyObject* get prettyprinted.
I've been looking at the code and this is pretty neat. I did encounter
some issues, for instance if you load the script before loading the
python interpreter you get this traceback because these types are not
defined at that time:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File ".libpython.py", line 49, in <module>
_type_size_t = gdb.lookup_type('size_t')
RuntimeError: No type named size_t.
So I think it would be a good idea to not make that code module-level.
>
> One other thought: if it's possibly to expose the cython structures in
> some meaningful way, perhaps we could change upstream python's gdb hooks
> to simply integrate them into the py-* commands I mentioned above? (so
> e.g. cython c functions get somehow treated as python frames; currently
> I have a test predicate:
> Frame.is_evalframeex()
> which perhaps could be generalized?)
>
> (Not sure; it would complicate the selftests within python itself)
>
I think it would be hard to make them actual Python frames because
creating frames in the inferior process from gdb is probably quite
dangerous, and the alternative would be to modify Cython so that it
creates Python stack frames (this sounds feasible but I think it might
be a little bit of work). However, if this could be done (it would
only do so if this 'debug' flag is active), then tracebacks and locals
inspection etc wouldn't need special attention and the code would
appear as normal Python code (apart from the Code objects obviously).
However, this would form a problem for non-primitive C-type Cython
variables.
So at the very least we could have a 'py-locals' or some such command
that would show the value of all the locals (the Python locals would
be printed by py-print and C locals by gdb print). For regular python
code it would show the locals from the current stack frame. For the
Cython part to work we would need information from the Cython compiler
because we wouldn't want to list any temporary or irrelevant
variables.
So I think we should be able to integrate these two projects into one
fruitful project, and with proper documentation it could help both
regular Python users and Cython users.
>> Let us know if you would like a more "official" branch on cython.org,
>> although just keeping one on bitbucket.org is usually just as easy. Then
>> we can pull the changes into mainline when things work well.
>>
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> [email protected]
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> [email protected]
> http://codespeak.net/mailman/listinfo/cython-dev
>
Cheers,
Mark
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
