Re: [Cython] Use of long type for intermediate integral variables

[Robert McGibbon]

> "libc.stdint.int64_t" is hand-wavingly declared as "long"

There are some deeper issues in the rest of your message, but as a
preliminary matter, isn't this a clear error for linux-32 and windows?

-Robert


On Wed, Jul 1, 2015 at 11:30 PM, Stefan Behnel  wrote:

> Robert McGibbon schrieb am 01.07.2015 um 11:12:
> > I noticed an issue on Windows when debugging an issue in scipy
> > , but I think it might be a
> > little more general.  In some places in the generated code, it looks like
> > intermediate integral variables are declared with type long, even when
> long
> > is too small to hold necessary value. For example, with the code pasted
> > below, the value n+1 is stored in a variable of type long (using Cython
> > 0.22.1) before being supplied to F.__getitem__.
> >
> > This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit
> > linux, where longs are 32-bits, so you get an overflow for a program like
> > the example below. The result is that it prints 1 instead of the expected
> > value, 2**53+1 = 9007199254740993. But this same issue comes up basically
> > whenever you do arithmetic on an array index in 64-bit Windows, for
> indices
> > larger than 2**31-1, since sizeof(long) << sizeof(void*).
> >
> > ```
> > from libc.stdint cimport int64_t
> >
> > class F(object):
> > def __getitem__(self, i):
> > print(i)
> >
> > cdef int64_t n = 2**53
> > f = F()
> > f[n+1]
> > ```
>
> Thanks for the report and the investigation. I can imagine why this is the
> case. "libc.stdint.int64_t" is hand-wavingly declared as "long" and the
> literal 1 is also of type "long" in Cython, so it infers that using "long"
> is good enough to hold the result of the sum.
>
> You can work around this by casting the 1 to , but that's clumsy
> and error prone. The problem is that Cython doesn't know the exact type of
> typedefs at translation time, only the C compilers will eventually know and
> might have diverging ideas about it. Your specific issue could be helped by
> preferring typedefs over standard integer types in the decision which type
> to use for arithmetic expressions, but that would then break the case where
> the typedef-ed type happens to be smaller than the standard one, e.g.
>
> cdef extern from "...":
> ctypedef long long SomeInt  # declared large enough, just in case
>
> def test(SomeInt x):
> cdef long long y = 1
> return x + y
>
> If Cython inferred "int64" for the type of the result, the C code would be
> correct if sizeof(SomeInt) >= sizeof(long long), but not if it's smaller.
>
> Also, what should happen in expressions that used two different user
> provided typedefs of the same declared base type? The decision here must
> necessarily be arbitrary.
>
> So, I agree that what you have found is a problem, it feels like fixing it
> by preferring typedefs would generally be a good idea, but on the other
> hand, it might break existing code (which usually means that it *will*
> break someone's code), and it would not fix all possible problematic cases.
>
> Not an easy decision...
>
> Stefan
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Stefan Behnel]

Robert McGibbon schrieb am 02.07.2015 um 09:49:
>> "libc.stdint.int64_t" is hand-wavingly declared as "long"
> 
> There are some deeper issues in the rest of your message, but as a
> preliminary matter, isn't this a clear error for linux-32 and windows?

No, it's not. That's just what Cython sees. The C compiler then sees the
exact platform specific type. And as the vast amount of Cython code out
there shows, it's usually not a problem in practice. Cython is designed to
handle most of these "platform specific type" issues at C compile time
rather than C code generation time. But as your example shows, it can't
always hide the details entirely. And there can be bugs.

Stefan

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Robert McGibbon]

Right, okay. I think I understand.

-Robert

On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel  wrote:

> Robert McGibbon schrieb am 02.07.2015 um 09:49:
> >> "libc.stdint.int64_t" is hand-wavingly declared as "long"
> >
> > There are some deeper issues in the rest of your message, but as a
> > preliminary matter, isn't this a clear error for linux-32 and windows?
>
> No, it's not. That's just what Cython sees. The C compiler then sees the
> exact platform specific type. And as the vast amount of Cython code out
> there shows, it's usually not a problem in practice. Cython is designed to
> handle most of these "platform specific type" issues at C compile time
> rather than C code generation time. But as your example shows, it can't
> always hide the details entirely. And there can be bugs.
>
> Stefan
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Ian Henriksen]

On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon  wrote:

> Right, okay. I think I understand.
>
> -Robert
>
> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel 
> wrote:
>
>> Robert McGibbon schrieb am 02.07.2015 um 09:49:
>> >> "libc.stdint.int64_t" is hand-wavingly declared as "long"
>> >
>> > There are some deeper issues in the rest of your message, but as a
>> > preliminary matter, isn't this a clear error for linux-32 and windows?
>>
>> No, it's not. That's just what Cython sees. The C compiler then sees the
>> exact platform specific type. And as the vast amount of Cython code out
>> there shows, it's usually not a problem in practice. Cython is designed to
>> handle most of these "platform specific type" issues at C compile time
>> rather than C code generation time. But as your example shows, it can't
>> always hide the details entirely. And there can be bugs.
>>
>> Stefan
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel


This is an interesting discussion. Thanks.
For the particular case of int64_t, wouldn't it still be wise to
make it a typedef of long long rather than just long so as to consistently
get the correct size on platforms where long
is only 32 bits?
Thanks!
-Ian Henriksen
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] cython-devel Digest, Vol 53, Issue 9

[Stephen LARROQUE]

Thank you very much Stefan.

I have just read the ticket, and it seems that Guido van Rossum consider
this to be a feature, not a bug (???), so I guess they won't fix it.

For me it would be a nice plus to have this problem fixed, but I can live
with my current workaround, so I'm not sure if Cython should spend dev time
to fix that if that's expensive.

At least, I think the workaround should be mentioned in the github wiki so
that others may implement finite fields extension types without MemoryError
issues. Should I add a wiki page?

2015-06-19 12:00 GMT+02:00 :

> Send cython-devel mailing list submissions to
> cython-devel@python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman/listinfo/cython-devel
> or, via email, send a message with subject or body 'help' to
> cython-devel-requ...@python.org
>
> You can reach the person managing the list at
> cython-devel-ow...@python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of cython-devel digest..."
>
>
> Today's Topics:
>
>1. Re: Bug: Extension Type inheriting from int cause a
>   MemoryError (Stefan Behnel)
>
>
> --
>
> Message: 1
> Date: Fri, 19 Jun 2015 08:38:32 +0200
> From: Stefan Behnel 
> To: cython-devel@python.org
> Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a
> MemoryError
> Message-ID: <5583b8e8.9050...@behnel.de>
> Content-Type: text/plain; charset=utf-8
>
> Stephen LARROQUE schrieb am 15.06.2015 um 12:34:
> > I am trying to make an extension type inheriting from int or cython.int
> (to
> > do arithmetic operations in Galois Fields). However, this causes a
> > MemoryError because it seems such extension type is not freed correctly.
> > Other than that, it works perfectly well.
> >
> > cdef class ExtendedInt(int): pass
> >
> > for j in xrange(1000):
> > ExtendedInt(j)
>
> It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this
> code in intobject.c:
>
> """
> static void
> int_dealloc(PyIntObject *v)
> {
> if (PyInt_CheckExact(v)) {
> Py_TYPE(v) = (struct _typeobject *)free_list;
> free_list = v;
> }
> else
> Py_TYPE(v)->tp_free((PyObject *)v);
> }
>
> static void
> int_free(PyIntObject *v)
> {
> Py_TYPE(v) = (struct _typeobject *)free_list;
> free_list = v;
> }
> """
>
> Your extension type automatically inherits the "int_free" slot function
> from its base type, so the "else" case in "int_dealloc()" will in fact call
> "int_free()" and append the object to the free list despite *not* being
> exactly of type PyInt. Then, when creating a new ExtendedInt instance,
> Python's int-subtype instantiation code *ignores* the free list and instead
> creates a completely new object.
>
> Thus, the free list keeps growing until it fills all memory and the process
> dies. I created a CPython ticket.
>
> https://bugs.python.org/issue24469
>
> I guess we could hack up a work around for this in Cython somehow (Python
> 2.7 is an easy target being a dead end, after all), but let's wait what the
> CPython devs have to say about this. Note, however, that any fix in a
> future CPython 2.7.x release will not fix it in earlier Python 2.x
> versions, so my guess is that we'd end up having to add a work-around on
> our side anyway.
>
> Stefan
>
>
>
> --
>
> Subject: Digest Footer
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
> --
>
> End of cython-devel Digest, Vol 53, Issue 9
> ***
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Robert Bradshaw]

I've fixed this particular case by making our integer ranking more
consistent: signedness is ignored for anything but chars and typedefs
are always preferred (as a last tiebreaker) over non-typedefs (rather
than arbitrarily picking the first or second argument).
https://github.com/cython/cython/commit/61ee8e544bd8802dfc313d832f5da97baf755c3d

I've also upgraded the definition of int64_t to be long long to avoid
issues with int64_t + unsigned long. There's still an issue of int32_t
+ unsigned int for any platforms where sizeof(int) == 16.

 -Robert

On Thu, Jul 2, 2015 at 1:11 PM, Ian Henriksen
 wrote:
> On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon  wrote:
>>
>> Right, okay. I think I understand.
>>
>> -Robert
>>
>> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel 
>> wrote:
>>>
>>> Robert McGibbon schrieb am 02.07.2015 um 09:49:
>>> >> "libc.stdint.int64_t" is hand-wavingly declared as "long"
>>> >
>>> > There are some deeper issues in the rest of your message, but as a
>>> > preliminary matter, isn't this a clear error for linux-32 and windows?
>>>
>>> No, it's not. That's just what Cython sees. The C compiler then sees the
>>> exact platform specific type. And as the vast amount of Cython code out
>>> there shows, it's usually not a problem in practice. Cython is designed
>>> to
>>> handle most of these "platform specific type" issues at C compile time
>>> rather than C code generation time. But as your example shows, it can't
>>> always hide the details entirely. And there can be bugs.
>>>
>>> Stefan
>>>
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
> This is an interesting discussion. Thanks.
> For the particular case of int64_t, wouldn't it still be wise to
> make it a typedef of long long rather than just long so as to consistently
> get the correct size on platforms where long
> is only 32 bits?
> Thanks!
> -Ian Henriksen
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Robert Bradshaw]

To clarify, any choice of int64_t + unsigned long will be wrong on one
platform or the others, but getting the wrong sign seems preferable to
getting the wrong number of bits.

On Thu, Jul 2, 2015 at 8:45 PM, Robert Bradshaw
 wrote:
> I've fixed this particular case by making our integer ranking more
> consistent: signedness is ignored for anything but chars and typedefs
> are always preferred (as a last tiebreaker) over non-typedefs (rather
> than arbitrarily picking the first or second argument).
> https://github.com/cython/cython/commit/61ee8e544bd8802dfc313d832f5da97baf755c3d
>
> I've also upgraded the definition of int64_t to be long long to avoid
> issues with int64_t + unsigned long. There's still an issue of int32_t
> + unsigned int for any platforms where sizeof(int) == 16.
>
>  -Robert
>
> On Thu, Jul 2, 2015 at 1:11 PM, Ian Henriksen
>  wrote:
>> On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon  wrote:
>>>
>>> Right, okay. I think I understand.
>>>
>>> -Robert
>>>
>>> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel 
>>> wrote:

 Robert McGibbon schrieb am 02.07.2015 um 09:49:
 >> "libc.stdint.int64_t" is hand-wavingly declared as "long"
 >
 > There are some deeper issues in the rest of your message, but as a
 > preliminary matter, isn't this a clear error for linux-32 and windows?

 No, it's not. That's just what Cython sees. The C compiler then sees the
 exact platform specific type. And as the vast amount of Cython code out
 there shows, it's usually not a problem in practice. Cython is designed
 to
 handle most of these "platform specific type" issues at C compile time
 rather than C code generation time. But as your example shows, it can't
 always hide the details entirely. And there can be bugs.

 Stefan

 ___
 cython-devel mailing list
 cython-devel@python.org
 https://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>> This is an interesting discussion. Thanks.
>> For the particular case of int64_t, wouldn't it still be wise to
>> make it a typedef of long long rather than just long so as to consistently
>> get the correct size on platforms where long
>> is only 32 bits?
>> Thanks!
>> -Ian Henriksen
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Ian Henriksen]

On Thu, Jul 2, 2015 at 9:50 PM Robert Bradshaw 
wrote:

> To clarify, any choice of int64_t + unsigned long will be wrong on one
> platform or the others, but getting the wrong sign seems preferable to
> getting the wrong number of bits.
>

That makes sens. Thanks for looking at it!
-Ian Henriksen


>
> On Thu, Jul 2, 2015 at 8:45 PM, Robert Bradshaw
>  wrote:
> > I've fixed this particular case by making our integer ranking more
> > consistent: signedness is ignored for anything but chars and typedefs
> > are always preferred (as a last tiebreaker) over non-typedefs (rather
> > than arbitrarily picking the first or second argument).
> >
> https://github.com/cython/cython/commit/61ee8e544bd8802dfc313d832f5da97baf755c3d
> >
> > I've also upgraded the definition of int64_t to be long long to avoid
> > issues with int64_t + unsigned long. There's still an issue of int32_t
> > + unsigned int for any platforms where sizeof(int) == 16.
> >
> >  -Robert
> >
> > On Thu, Jul 2, 2015 at 1:11 PM, Ian Henriksen
> >  wrote:
> >> On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon 
> wrote:
> >>>
> >>> Right, okay. I think I understand.
> >>>
> >>> -Robert
> >>>
> >>> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel 
> >>> wrote:
> 
>  Robert McGibbon schrieb am 02.07.2015 um 09:49:
>  >> "libc.stdint.int64_t" is hand-wavingly declared as "long"
>  >
>  > There are some deeper issues in the rest of your message, but as a
>  > preliminary matter, isn't this a clear error for linux-32 and
> windows?
> 
>  No, it's not. That's just what Cython sees. The C compiler then sees
> the
>  exact platform specific type. And as the vast amount of Cython code
> out
>  there shows, it's usually not a problem in practice. Cython is
> designed
>  to
>  handle most of these "platform specific type" issues at C compile time
>  rather than C code generation time. But as your example shows, it
> can't
>  always hide the details entirely. And there can be bugs.
> 
>  Stefan
> 
>  ___
>  cython-devel mailing list
>  cython-devel@python.org
>  https://mail.python.org/mailman/listinfo/cython-devel
> >>>
> >>>
> >>> ___
> >>> cython-devel mailing list
> >>> cython-devel@python.org
> >>> https://mail.python.org/mailman/listinfo/cython-devel
> >>
> >>
> >> This is an interesting discussion. Thanks.
> >> For the particular case of int64_t, wouldn't it still be wise to
> >> make it a typedef of long long rather than just long so as to
> consistently
> >> get the correct size on platforms where long
> >> is only 32 bits?
> >> Thanks!
> >> -Ian Henriksen
> >>
> >> ___
> >> cython-devel mailing list
> >> cython-devel@python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[David Vierra]

Very informative discussion. I see that Cython doesn't actually know the 
underlying types of typedefs because it doesn't parse any C headers. 
Cython doesn't even care about the exact length of the underlying type, 
just that it can order the types by rank to find which of two types is 
widest. When you write `ctypedef`, you're just telling Cython that two 
types are equivalent.


I second the opinion that the stdint.h typedefs in the standard library 
should have accurate definitions. This way `ctypedefs` that resolve to a 
stdint type will not be too wrong.


It might be safe just to change the ctypedefs for int64_t to `long long` 
- per the C99 standard, this type is *always* at least 64-bits long. (I 
don't know if all compilers adhere to this part of the spec and don't 
feel like getting a list of every compiler's opinion...but I know for 
sure it's 64 bits on MSVC, which doesn't claim to implement C99 nor even 
provide a stdint.h in the 2008 version.)


David Vierra
MCEdit, a Minecraft World Editor
http://www.mcedit.net
http://twitter.com/codewarrior0

On 7/2/2015 10:11 AM, Ian Henriksen wrote:
On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon > wrote:


Right, okay. I think I understand.

-Robert

On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel
mailto:stefan...@behnel.de>> wrote:

Robert McGibbon schrieb am 02.07.2015 um 09:49:
>> "libc.stdint.int64_t" is hand-wavingly declared as "long"
>
> There are some deeper issues in the rest of your message,
but as a
> preliminary matter, isn't this a clear error for linux-32
and windows?

No, it's not. That's just what Cython sees. The C compiler
then sees the
exact platform specific type. And as the vast amount of Cython
code out
there shows, it's usually not a problem in practice. Cython is
designed to
handle most of these "platform specific type" issues at C
compile time
rather than C code generation time. But as your example shows,
it can't
always hide the details entirely. And there can be bugs.

Stefan

___
cython-devel mailing list
cython-devel@python.org 
https://mail.python.org/mailman/listinfo/cython-devel


___
cython-devel mailing list
cython-devel@python.org 
https://mail.python.org/mailman/listinfo/cython-devel


This is an interesting discussion. Thanks.
For the particular case of int64_t, wouldn't it still be wise to
make it a typedef of long long rather than just long so as to 
consistently get the correct size on platforms where long

is only 32 bits?
Thanks!
-Ian Henriksen


___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel


___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Use of long type for intermediate integral variables

[Stefan Behnel]

David Vierra schrieb am 03.07.2015 um 02:26:
> I see that Cython doesn't actually know the
> underlying types of typedefs because it doesn't parse any C headers.

And even if it did parse header files, it still wouldn't know the
properties of the underlying types because they are platform specific and
only turn into concrete types and sizes at C compile time.

Stefan

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel