[Cython] Core dump when Python is built without debug symbols
Hello, First, apologies if this is not exactly the right place to post this, but I have tried already cython-users@ and my post is still pending somewhere for approval, so I've decided to give it a shot here. I'm working on Cython wrappers for a C library and everything was going smooth on my dev machine, until I've decided to install the wrappers on another machine. The only difference between these machines is that my dev machine has Python built with debug symbols and the other machine does not have debug symbols in Python. The issue I'm having and was trying to solve for the past few days is that if I run my Cython wrappers on a machine where Python does not have debug symbols it core dumps ugly. If I run the same code on another machine with Python + debug symbols then everything is running perfect. I've done some debugging and tracing as well, and noticed that the core dump happens at PyTuple_Size() when I have Python *without* debug symbols. Running Python + debug symbols shows that the interpreter takes a different path and does not core dump (does not even call PyTuple_Size() for some reason) Anyway, I've uploaded the backtrace and some gdb tracing in here: * http://users.unix-heaven.org/~dnaeon/cython-wrappers/wrappers-trace.txt The code that I have you can find in here: * http://users.unix-heaven.org/~dnaeon/cython-wrappers/src/ What really puzzles me is why it core dumps when Python does not have debug symbols and why it takes a different path (and not core dumping) when Python is built with debug symbols. I'd say that if it core dumps because of a NULL pointer or something similar, then having Python with or without debug symbols should not make a difference, but it does for some reason. I've been working on this for the past few days and I'm out of ideas now. Any help/hints/feedback is much appreciated. Thanks and regards, Marin -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/ ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Core dump when Python is built without debug symbols
On Thu, May 30, 2013 at 3:08 PM, Nikita Nemkin wrote: > > > In pkg-db.pxi, this line: > > return Pkg(pkg) > > and in pkg-pkg.pxi this line: > > def __cinit__(self, pkg): > self._pkg = pkg > > are the problem. > > Thanks, Nikita! That was the problem indeed :) The funny part is that I was looking for a way to actually pass a C pointer to __cinit__ and doing it that way the compiler didn't complain at all. In fact it used to work, until I've removed Python debug symbols and then I started seeing surprises :) > You are trying to pass a C pointer to __cinit__. What actually > happens is that, because of the cast, Cython treats pkg as > as Python object _itself_ and tries to do some object operations on it. > > Two ways to solve this (really common) problem: > > 1) Pass your pointer as a real python object containing the integer value >of the pointer: > > def __cinit__(self, pkg): > self._pkg = pkg # or Py_intptr_t > > return Pkg(pkg) > > Note the cast. When used on a C pointer, it makes a C > integer that Cython convert to python int object. And when > is used on a python int object, Cyton extracts its integer value > as a C-level value, so it is castable to a C-level pointer. > > Using Py_ssize_t (or Py_intptr_t or your platform intptr_t) guarantees > that the pointer will fit. > > 2) Don't use __cinit__. Declare your own cdef init method and call that. >Since it's cdef method, it can take any C-level arguments, pointerw, >whatever without the need to convert to python objects and back. > >@cython.final # small optimization >cdef _init(self, c_pkg.pkg * pkg): > self._pkg = pkg > ># cdef __cinit__ not needed. Maybe use it to set default ># attribute values. > >cdef Pkg pkg = Pkg.__new__(Pkg) # fast object creation. Pkg() also > works. >pkg._init(pkg) > >(When you use the __new__ trick, __cnit__ is called without >arguments and __init__ is not called at all. The benefit to >this ugly syntax is that __new__ is faster than creating the object >with traditional Pyhton syntax.) > > Also, if all your initialization is pointer assigment, you can > just do it directly: > >cdef Pkg pkg = Pkg.__new__(Pkg) # fast object creation >pkg._pkg = pkg > > > I've chosen to stick with 2) -- I guess I like it better and having an additional _init() is fine with me, makes things clearer. Thanks again! > There is a problem related to cython-dev. I think the code you > wrote should have never compiled in the first place. > cast must be required if you want to reinterpret a C pointer > as object or vice versa. > > Actually the code compiles just fine and also works as expected :) The only condition is to have Python with debug symbols installed. Any idea why having Python with debug symbols actually overcomes this issue? Thanks again Nikita! Best regards, Marin > > Best regards, > Nikita Nemkin > __**_ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/**mailman/listinfo/cython-devel<http://mail.python.org/mailman/listinfo/cython-devel> > -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/ ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Core dump when Python is built without debug symbols
On Thu, May 30, 2013 at 5:21 PM, Nikita Nemkin wrote: > On Thu, 30 May 2013 19:51:16 +0600, Marin Atanasov Nikolov < > dna...@gmail.com> wrote: > > Any idea why having Python with debug symbols actually overcomes this >> issue? >> > > In debug builds, Python object structure is prepended with a couple of > extra debug fields, which means a different piece of memory was wrongly > interpreted and manipulated as a Python object. > > It is pure luck that whatever values were there didn't cause a crash. > > > Thanks for the insight, Nikita! Your assistance is much appreciated! Best regards, Marin > > Best regards, > Nikita Nemkin > __**_ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/**mailman/listinfo/cython-devel<http://mail.python.org/mailman/listinfo/cython-devel> > -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/ ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] C data type and a C function() sharing the same name
Hello, Working on creating Cython wrappers for a C library I came across a strange problem. I have in the C library a struct like this: struct my_jobs; I also have this function, which returns the next job in the queue: int my_jobs(struct my_jobs *jobs); Translating this into Cython and putting this in the .pxd file it looks like this: cdef struct my_jobs int my_jobs(my_jobs *jobs) During build I'm having issues because it seems that the function my_jobs() is translated in a way that it should return a "int struct my_jobs". The real problem I see is that I cannot have a data type and a function sharing the same name. How can I overcome this issue? Suppose that I wrote the C API I could change that, but how would you really solve this if you cannot touch what's in upstream? Any ways to solve this? Thanks and regards, Marin -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/ ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] C data type and a C function() sharing the same name
Thanks, Nikita! I've been looking at the Cython documentation, but was not able to find it previously, thanks! I'm still waiting for my previous posts to show up in the cython-users@mailing list (although I am subscribed there), but they don't seem to show up. Thanks for your help! Best regards, Marin //offtopic Why does it take so long for a post to be approved/published on cython-users@ ? On Sat, Jun 1, 2013 at 5:12 PM, Nikita Nemkin wrote: > On Sat, 01 Jun 2013 19:29:17 +0600, Marin Atanasov Nikolov < > dna...@gmail.com> wrote: > > Hello, >> >> Working on creating Cython wrappers for a C library I came across a >> strange >> problem. >> >> I have in the C library a struct like this: >> >> struct my_jobs; >> >> I also have this function, which returns the next job in the queue: >> >> int my_jobs(struct my_jobs *jobs); >> >> Translating this into Cython and putting this in the .pxd file it looks >> like this: >> >> cdef struct my_jobs >> int my_jobs(my_jobs *jobs) >> >> During build I'm having issues because it seems that the function >> my_jobs() >> is translated in a way that it should return a "int struct my_jobs". >> >> The real problem I see is that I cannot have a data type and a function >> sharing the same name. >> >> How can I overcome this issue? Suppose that I wrote the C API I could >> change that, but how would you really solve this if you cannot touch >> what's in upstream? >> >> Any ways to solve this? >> > > This question would be more appropriate on the cython-users mailing list. > > Use renaming: > http://docs.cython.org/src/**userguide/external_C_code.** > html#resolving-naming-**conflicts-c-name-**specifications<http://docs.cython.org/src/userguide/external_C_code.html#resolving-naming-conflicts-c-name-specifications> > > For example, rename the function: > > int my_jobs_func "my_jobs" (my_jobs *jobs) > > or the struct: > > cdef struct my_jobs_t "my_jobs" > > or both. > > Best regards, > Nikita Nemkin > __**_ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/**mailman/listinfo/cython-devel<http://mail.python.org/mailman/listinfo/cython-devel> > -- Marin Atanasov Nikolov dnaeon AT gmail DOT com http://www.unix-heaven.org/ ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
