[Cython] Compiler crash on unmatched kwarg

2013-05-30 Thread Emmanuel Gil Peyrot 
Hi,

When the name of a kwarg in the definition and the declaration of a
function doesn’t match, the compiler crash instead of returning a
meaningful error.

I have joined a testcase, it is reproductible at least on 0.19.1 and
latest git.

-- 
Emmanuel Gil Peyrot
XMPP: 
OpenPGP: 24B1D609


testcase.tar.xz
Description: Binary data


pgpDMlz63gHBQ.pgp
Description: PGP signature
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Core dump when Python is built without debug symbols

2013-05-30 Thread Marin Atanasov Nikolov 
Hello,

First, apologies if this is not exactly the right place to post this, but I
have  tried already cython-users@ and my post is still pending somewhere
for approval, so I've decided to give it a shot here.

I'm working on Cython wrappers for a C library and everything was going
smooth on my dev machine, until I've decided to install the wrappers on
another machine.

The only difference between these machines is that my dev machine has
Python built with debug symbols and the other machine does not have debug
symbols in Python.

The issue I'm having and was trying to solve for the past few days is that
if I run my Cython wrappers on a machine where Python does not have debug
symbols it core dumps ugly. If I run the same code on another machine with
Python + debug symbols then everything is running perfect.

I've done some debugging and tracing as well, and noticed that the core
dump happens at PyTuple_Size() when I have Python *without* debug symbols.
Running Python + debug symbols shows that the interpreter takes a
 different path and does not core dump (does not even call PyTuple_Size()
for some reason)

Anyway, I've uploaded the backtrace and some gdb tracing in here:

* http://users.unix-heaven.org/~dnaeon/cython-wrappers/wrappers-trace.txt

The code that I have you can find in here:

* http://users.unix-heaven.org/~dnaeon/cython-wrappers/src/

What really puzzles me is why it core dumps when Python does not have debug
symbols and why it takes a different path (and not core dumping) when
Python is built with debug symbols.

I'd say that if it core dumps because of a NULL pointer or something
similar, then having Python with or without debug symbols should not make a
difference, but it does for some reason.

I've been working on this for the past few days and I'm out of ideas now.

Any help/hints/feedback is much appreciated.

Thanks and regards,
Marin

-- 
Marin Atanasov Nikolov

dnaeon AT gmail DOT com
http://www.unix-heaven.org/
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Core dump when Python is built without debug symbols

2013-05-30 Thread Nikita Nemkin 
On Thu, 30 May 2013 17:23:39 +0600, Marin Atanasov Nikolov  
 wrote:



Hello,

First, apologies if this is not exactly the right place to post this,  
but I

have  tried already cython-users@ and my post is still pending somewhere
for approval, so I've decided to give it a shot here.

I'm working on Cython wrappers for a C library and everything was going
smooth on my dev machine, until I've decided to install the wrappers on
another machine.

The only difference between these machines is that my dev machine has
Python built with debug symbols and the other machine does not have debug
symbols in Python.

The issue I'm having and was trying to solve for the past few days is  
that

if I run my Cython wrappers on a machine where Python does not have debug
symbols it core dumps ugly. If I run the same code on another machine  
with

Python + debug symbols then everything is running perfect.

I've done some debugging and tracing as well, and noticed that the core
dump happens at PyTuple_Size() when I have Python *without* debug  
symbols.

Running Python + debug symbols shows that the interpreter takes a
 different path and does not core dump (does not even call PyTuple_Size()
for some reason)

Anyway, I've uploaded the backtrace and some gdb tracing in here:

* http://users.unix-heaven.org/~dnaeon/cython-wrappers/wrappers-trace.txt

The code that I have you can find in here:

* http://users.unix-heaven.org/~dnaeon/cython-wrappers/src/

What really puzzles me is why it core dumps when Python does not have  
debug

symbols and why it takes a different path (and not core dumping) when
Python is built with debug symbols.

I'd say that if it core dumps because of a NULL pointer or something
similar, then having Python with or without debug symbols should not  
make a

difference, but it does for some reason.

I've been working on this for the past few days and I'm out of ideas now.

Any help/hints/feedback is much appreciated.


In pkg-db.pxi, this line:

return Pkg(pkg)

and in pkg-pkg.pxi this line:

def __cinit__(self, pkg):
self._pkg = pkg

are the problem.

You are trying to pass a C pointer to __cinit__. What actually
happens is that, because of the  cast, Cython treats pkg as
as Python object _itself_ and tries to do some object operations on it.

Two ways to solve this (really common) problem:

1) Pass your pointer as a real python object containing the integer value
   of the pointer:

def __cinit__(self, pkg):
self._pkg = pkg  # or Py_intptr_t

return Pkg(pkg)

Note the  cast. When used on a C pointer, it makes a C
integer that Cython convert to python int object. And when 
is used on a python int object, Cyton extracts its integer value
as a C-level value, so it is castable to a C-level pointer.

Using Py_ssize_t (or Py_intptr_t or your platform intptr_t) guarantees
that the pointer will fit.

2) Don't use __cinit__. Declare your own cdef init method and call that.
   Since it's cdef method, it can take any C-level arguments, pointerw,
   whatever without the need to convert to python objects and back.

   @cython.final  # small optimization
   cdef _init(self, c_pkg.pkg * pkg):
self._pkg = pkg

   # cdef __cinit__ not needed. Maybe use it to set default
   # attribute values.

   cdef Pkg pkg = Pkg.__new__(Pkg)  # fast object creation. Pkg() also  
works.

   pkg._init(pkg)

   (When you use the __new__ trick, __cnit__ is called without
   arguments and __init__ is not called at all. The benefit to
   this ugly syntax is that __new__ is faster than creating the object
   with traditional Pyhton syntax.)

Also, if all your initialization is pointer assigment, you can
just do it directly:

   cdef Pkg pkg = Pkg.__new__(Pkg)  # fast object creation
   pkg._pkg = pkg


There is a problem related to cython-dev. I think the code you
wrote should have never compiled in the first place. 
cast must be required if you want to reinterpret a C pointer
as object or vice versa.


Best regards,
Nikita Nemkin
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Exception check optimization

2013-05-30 Thread Nikita Nemkin 
On Thu, 30 May 2013 08:25:55 +0600, Robert Bradshaw   
wrote:



On Sun, May 26, 2013 at 10:04 AM, Nikita Nemkin  wrote:

Hi,

I wonder why is __pyx_filename (in exception check blocks)
tracked dynamically? AFAIK it's impossible to split function
body between multiple files (include only works at the top level),
which makes filename a compile time constant for any given function.

If the above is correct, __pyx_filename variable can be eliminated,
saving at least 5 bytes per exception check (more on x64).


Probably because exception checks are heavy-weight enough that we
haven't bothered optimizing them at this level yet.


By "exception checks" I mean these things:

if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 15;  
__pyx_clineno = __LINE__; goto __pyx_L1_error;}


They are fast, but the problem is the extra code: neither
__pyx_filename nor __pyx_clineno assignments are necessary.
Removing them reduces binary size by 2-3%.

If you're OK with it, I'll make a patch that removes __pyx_filename
and disables __pyx_clineno by default.


Best regards,
Nikita Nemkin
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Core dump when Python is built without debug symbols

2013-05-30 Thread Marin Atanasov Nikolov 
On Thu, May 30, 2013 at 3:08 PM, Nikita Nemkin  wrote:
>
>
> In pkg-db.pxi, this line:
>
> return Pkg(pkg)
>
> and in pkg-pkg.pxi this line:
>
> def __cinit__(self, pkg):
> self._pkg = pkg
>
> are the problem.
>
>
Thanks, Nikita! That was the problem indeed :)

The funny part is that I was looking for a way to actually pass a C pointer
to __cinit__ and doing it that way the compiler didn't complain at all. In
fact it used to work, until I've removed Python debug symbols and then I
started seeing surprises :)


> You are trying to pass a C pointer to __cinit__. What actually
> happens is that, because of the  cast, Cython treats pkg as
> as Python object _itself_ and tries to do some object operations on it.
>
> Two ways to solve this (really common) problem:
>
> 1) Pass your pointer as a real python object containing the integer value
>of the pointer:
>
> def __cinit__(self, pkg):
> self._pkg = pkg  # or Py_intptr_t
>
> return Pkg(pkg)
>
> Note the  cast. When used on a C pointer, it makes a C
> integer that Cython convert to python int object. And when 
> is used on a python int object, Cyton extracts its integer value
> as a C-level value, so it is castable to a C-level pointer.
>
> Using Py_ssize_t (or Py_intptr_t or your platform intptr_t) guarantees
> that the pointer will fit.
>
> 2) Don't use __cinit__. Declare your own cdef init method and call that.
>Since it's cdef method, it can take any C-level arguments, pointerw,
>whatever without the need to convert to python objects and back.
>
>@cython.final  # small optimization
>cdef _init(self, c_pkg.pkg * pkg):
> self._pkg = pkg
>
># cdef __cinit__ not needed. Maybe use it to set default
># attribute values.
>
>cdef Pkg pkg = Pkg.__new__(Pkg)  # fast object creation. Pkg() also
> works.
>pkg._init(pkg)
>
>(When you use the __new__ trick, __cnit__ is called without
>arguments and __init__ is not called at all. The benefit to
>this ugly syntax is that __new__ is faster than creating the object
>with traditional Pyhton syntax.)
>
> Also, if all your initialization is pointer assigment, you can
> just do it directly:
>
>cdef Pkg pkg = Pkg.__new__(Pkg)  # fast object creation
>pkg._pkg = pkg
>
>
>
I've chosen to stick with 2) -- I guess I like it better and having an
additional _init() is fine with me, makes things clearer. Thanks again!


> There is a problem related to cython-dev. I think the code you
> wrote should have never compiled in the first place. 
> cast must be required if you want to reinterpret a C pointer
> as object or vice versa.
>
>
Actually the code compiles just fine and also works as expected :) The only
condition is to have Python with debug symbols installed.

Any idea why having Python with debug symbols actually overcomes this issue?

Thanks again Nikita!

Best regards,
Marin


>
> Best regards,
> Nikita Nemkin
> __**_
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/**mailman/listinfo/cython-devel
>



-- 
Marin Atanasov Nikolov

dnaeon AT gmail DOT com
http://www.unix-heaven.org/
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Core dump when Python is built without debug symbols

2013-05-30 Thread Nikita Nemkin 
On Thu, 30 May 2013 19:51:16 +0600, Marin Atanasov Nikolov  
 wrote:


Any idea why having Python with debug symbols actually overcomes this  
issue?


In debug builds, Python object structure is prepended with a couple of
extra debug fields, which means a different piece of memory was wrongly
interpreted and manipulated as a Python object.

It is pure luck that whatever values were there didn't cause a crash.


Best regards,
Nikita Nemkin
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Exception check optimization

2013-05-30 Thread Stefan Behnel 
Am 30.05.2013 14:38, schrieb Nikita Nemkin:
> On Thu, 30 May 2013 08:25:55 +0600, Robert Bradshaw wrote:
>> On Sun, May 26, 2013 at 10:04 AM, Nikita Nemkin wrote:
>>> I wonder why is __pyx_filename (in exception check blocks)
>>> tracked dynamically? AFAIK it's impossible to split function
>>> body between multiple files (include only works at the top level),
>>> which makes filename a compile time constant for any given function.

What about includes? Are they allowed inside of functions? If not, then
this sounds like a safe assumption.

If they are allowed, then it should still be possible to special case
the file name setting, i.e. set it to the default at function entry and
only set it to a different value if it happens to change inside of a
function. The C compiler should often (normally?) be smart enough to
eliminate the constant assignment.


>>> If the above is correct, __pyx_filename variable can be eliminated,
>>> saving at least 5 bytes per exception check (more on x64).
>>
>> Probably because exception checks are heavy-weight enough that we
>> haven't bothered optimizing them at this level yet.
> 
> By "exception checks" I mean these things:
> 
> if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno =
> 15; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
> 
> They are fast, but the problem is the extra code: neither
> __pyx_filename nor __pyx_clineno assignments are necessary.
> Removing them reduces binary size by 2-3%.
> 
> If you're OK with it, I'll make a patch that removes __pyx_filename
> and disables __pyx_clineno by default.

I'm fine with both. I consider seeing the C line a debug feature that
people shouldn't need normally.

Stefan

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Core dump when Python is built without debug symbols

2013-05-30 Thread Marin Atanasov Nikolov 
On Thu, May 30, 2013 at 5:21 PM, Nikita Nemkin  wrote:

> On Thu, 30 May 2013 19:51:16 +0600, Marin Atanasov Nikolov <
> dna...@gmail.com> wrote:
>
>  Any idea why having Python with debug symbols actually overcomes this
>> issue?
>>
>
> In debug builds, Python object structure is prepended with a couple of
> extra debug fields, which means a different piece of memory was wrongly
> interpreted and manipulated as a Python object.
>
> It is pure luck that whatever values were there didn't cause a crash.
>
>
>
Thanks for the insight, Nikita! Your assistance is much appreciated!

Best regards,
Marin


>
> Best regards,
> Nikita Nemkin
> __**_
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/**mailman/listinfo/cython-devel
>



-- 
Marin Atanasov Nikolov

dnaeon AT gmail DOT com
http://www.unix-heaven.org/
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel