Re: [Cython] CEP: prange for parallel loops

[Nathaniel Smith]

On Mon, Apr 4, 2011 at 3:17 AM, Dag Sverre Seljebotn
 wrote:
>  * A simple and friendly solution that covers, perhaps, 80% of the cases,
> based on simply replacing range with prange.

This is a "merely" aesthetic objection, while remaining agnostic on
the larger discussion, but -- 'for i in prange(...)' looks Just Wrong.
This is not a regular loop over a funny range, it's a funny loop over
a regular range. Surely it should be 'pfor i in range(...)'. Or better
yet, spell it 'parallel_for'.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [SciPy-User] Scikits.sparse build issue

[Nathaniel Smith]

Whoops, looks like we passed each other :-)

My patch has a few other fixes you might want to apply too, since I
figured I should probably double-check the rest of the 'enum's in
there too -- I don't know if they'd cause problems in practice, but...

On Thu, May 5, 2011 at 11:46 AM, Dag Sverre Seljebotn
 wrote:
> On 05/05/2011 08:42 PM, Nathaniel Smith wrote:
>>
>> On Thu, May 5, 2011 at 3:03 AM, Anand Patil
>>   wrote:
>>>
>>> On May 4, 8:16 pm, Nathaniel Smith  wrote:
>>>>
>>>> On Tue, May 3, 2011 at 10:10 AM, Nathaniel Smith  wrote:
>>>>>
>>>>> On Tue, May 3, 2011 at 5:51 AM, Anand Patil
>>>>>   wrote:
>>>>>>
>>>>>> scikits/sparse/cholmod.c: In function
>>>>>> ‘__pyx_f_7scikits_6sparse_7cholmod__py_sparse’:
>>>>>> scikits/sparse/cholmod.c:1713: error: storage size of ‘__pyx_t_10’
>>>>>> isn’t known
>>>>
>>>>>> I've never used Cython and am having a hard time figuring this out.
>>>>
>>>>> Could you send me the file 'scikits/sparse/cholmod.c'? This means that
>>>>> there's some C type that was forward-declared, but never actually
>>>>> defined, and then we tried to instantiate an instance of it. But I'll
>>>>> need to see the generated code to figure out which type '__pyx_t_10'
>>>>> is supposed to be.
>>>>
>>>> Huh, this appears to be some bad interaction between numpy and cython,
>>>> rather than anything to do with my code. The offending variable comes
>>>> from doing 'cimport numpy as np' and then referring to
>>>> 'np.NPY_F_CONTIGUOUS' -- this is being translated to:
>>>>   enum requirements __pyx_t_10;
>>>>   __pyx_t_10 = NPY_F_CONTIGUOUS;
>>>> and then gcc is complaining that 'enum requirements' is an undefined
>>>> type.
>>>>
>>>> What version of Numpy and Cython do you have installed?
>>>
>>> Cython 0.14.1, Numpy 1.5.1. Which versions do you have?
>>
>> It looks like with Cython 0.12.1, which is what I was using before, it
>> happens not to generate a temporary variable in this case, but Cython
>> 0.14.1 generates the temporary variable.
>>
>> I've just committed a workaround to the scikits.sparse repository:
>>
>> https://code.google.com/p/scikits-sparse/source/detail?r=ad106e9c2c2d55f2022a3fb8b9282003b55666fc#
>> (I believe it works -- it does compile -- but technically I can't
>> guarantee it since for me the tests are now failing with an "illegal
>> instruction" error inside BLAS. But I think this must be an unrelated
>> Ubuntu screwup. Yay software.)
>>
>> And I'll see about poking Cython upstream to get this fixed...
>
> Awh. Thanks!
>
> https://github.com/cython/cython/commit/a6ec50077990a9767695896076a8b573a5bdccc0
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused types syntax

[Nathaniel Smith]

On Jun 2, 2011 2:18 PM, "Dag Sverre Seljebotn" 
wrote:
> If you only want this allowed in typedefs, then, being puristic, I think
that really a "fused type" is really different from a ctypedef, and that it
would warrant something like a new keyword.
>
> cdef fusedtype [list, dict, object] fused_t

+1 (to some variant of this, not necessarily this exactly)

I find the argument that defining the same fused type twice results in two
different, incompatible types to be very compelling. That's a fundamental
difference from typedefs.

- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] setup.py refusing to run cython

[Nathaniel Smith]

On Wed, Jun 15, 2011 at 12:18 AM, Stefan Behnel  wrote:
> Lisandro Dalcin, 14.06.2011 21:39:
>>
>> On 14 June 2011 16:20, Nathaniel Smith wrote:
>>>
>>> Hello Cython folks,
>>>
>>> This message (see below) is the second report I've gotten of a very
>>> strange build problem with a cython module. I'm just using the
>>> standard 'from Cython.Distutils import build_ext', 'cmdclass =
>>> {"build_ext": build_ext}' incantation, but for some reason setup.py is
>>> attempting to just run gcc directly without calling cython first to
>>> generate the .c file.
>>>
>>> You can see the setup.py here:
>>>
>>>  https://code.google.com/p/scikits-sparse/source/browse/setup.py?name=scikits.sparse-0.1
>>>
>>> Anyone have any ideas?
>>
>> Your setup.py is mixing setuptools and Cython.Distutils. Both are
>> monkeypatching distutils, and in general that's a very bad idea.
>
> Specifically, "setuptools" checks the list of source files and when it sees
> a .pyx extension, it checks if Pyrex (!) is installed. If not, which is
> rather unlikely for Cython users, it changes the file extension to ".c",
> thus preventing Cython from compiling it.
>
> There are two ways around this. The ugly way is to put a fake Pyrex into
> sys.path during installation, e.g.
>
> https://github.com/lxml/lxml/tree/master/fake_pyrex

Right, I'm already doing that, and it works for me...but not for all
my users, I guess?

> A better way is to switch from "setuptools" to "distribute".

Right, I'd just switch to plain distutils except this package uses the
'scikits' namespace package, and IIUC distutils can't handle namespace
packages.

I'm finding the docs for 'distribute' a bit confusing, though. Is the
way that I switch from 'setuptools' to 'distribute' to just leave my
code the same, and tell my users to install distribute? That seems
kind of error prone...

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.16

[Nathaniel Smith]

On Oct 29, 2011 4:41 AM, "mark florisson"  wrote:
> "
> Typed memoryviews can be used for efficient access to buffers. It is
> similar to the current buffer support, but has more features and
> cleaner syntax. A memoryview can be used in any context (function
> parameters, module-level, cdef class attribute, etc) and can be
> obtained from any object that exposes the PEP 3118 buffer interface.
> "

FWIW, I do find this paragraph somewhat confusing, because the main
description of what a typed memoryview is assumes that I already know the
current buffer support. I think that's actually true (the ndarray[int32]
syntax, right?), but I'm not sure, and people coming to this for the first
time probably won't even know that buffers are what they're looking for.

I'd say something like: "Typed memoryviews can be used for efficient access
to buffers. For example, you can use them to read and modify numpy arrays or
 without incurring any python overhead." And put
a compare/contrast with the old syntax later, like the second paragraph or
so.

My 2¢,
- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fwd: Re: [Numpy-discussion] Proposed Roadmap Overview

[Nathaniel Smith]

On Tue, Feb 21, 2012 at 8:02 PM, Robert Bradshaw
 wrote:
> On Tue, Feb 21, 2012 at 9:19 AM, mark florisson
>  wrote:
>> On 21 February 2012 04:42, Robert Bradshaw  
>> wrote:
>>> Python bytecode -> LLVM is a great idea for creating ufuncs, the
>>> overhead of Cython + GCC is atrocious for stuff like this. (I think
>>> Cython could make a good frontent as well, especially if we generated
>>> just the .c code for the function rather than a full extension module
>>> and used a good compiler to generate machine code in-memory
>>> on-the-fly.)
>>
>> I want elementwise functions and reductions for Cython as well, but
>> they would be specialized according to the types you use them for, so
>> it wouldn't be atrocious bloat (at least, no more then you'd already
>> be having).
>> I've been giving profile guided optimizations some thoughts as well, I
>> think they are most neatly handled through a python tracer that is
>> installed e.g. through sitecustomize which collects data about types,
>> time spent in which code regions etc during testing and development.
>> You can then tell Cython to compile the module, so e.g. if it
>> registered "a numpy array" it could generate some specializations for
>> dtypes either statically or dynamically through OpenCL. The
>> alternative approach using code instrumentation would be more of a
>> chore but would work just as well for Cython code with object-typed
>> arguments. That's all rather far off, and this looks like an
>> interesting development.
>
> One of the reasons JIT compilers can do so well is that they can use
> runtime profile information to write specialized code. It'd be great
> to be able to incorporate this kind of profiling into a Cython compile
> too. One of the big missing features here is a "fallback mode" where
> we can insert guards and fall back to generic code if they're not met,
> which I think would open us up to a huge number of optimizations
> without violating Python semantics.

A really neat first-step would be to write a straight LLVM backend for
Cython -- instead of generating code, one could generate equivalent
in-memory IR. (The really hacky way to do this would be to throw the
output of the current C code generator through Clang.) That seems
achievable (at least as pie-in-the-sky projects go), and even if
no-one ever got around to adding fancier profile-guided optimizations,
it would still be useful. For instance, it would be possible to not
just import .pyx modules directly, but reload() them after editing
(!!). Or have a cython.inline() function that works like weave.inline,
but more awesome.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Gsoc project

[Nathaniel Smith]

On Thu, Mar 29, 2012 at 5:25 PM, David Malcolm  wrote:
> On Thu, 2012-03-29 at 11:10 +0100, mark florisson wrote:
>> > Then what happens if you have
>> >
>> > #ifdef FOO
>> > #define BAR 3
>> > #else
>> > #define BAR 4
>> > #endif
>> >
>> > ?? I'm not saying it is hard, but perhaps no longer completely trivial :-)
> Yeah.  I have no idea to what extent the C preprocessor stuff is exposed
> internally, and if the branching logic is preserved in any way that's
> usable.

I'm pretty sure macros and such are gone before the C parser ever gets
to see the token stream.

But, it should be easy to pull out #define's on a second pass, using
something like
  gcc $CFLAGS -E -dM myfile.h
and then just scanning the output with a trivial "parser". (I'm not
sure which of -dM, -dD, -dU you want.)

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Fri, Apr 13, 2012 at 9:52 AM, Dag Sverre Seljebotn
 wrote:
> On 04/13/2012 01:38 AM, Robert Bradshaw wrote:
>> Also, I'm not sure the type registration will scale, especially if
>> every callable type wanted to get registered. (E.g. currently closures
>> and generators are new types...) Where to draw the line? (Perhaps
>> things could get registered lazily on the first __nativecall__ lookup,
>> as they're likely to be looked up again?)
>
>
> Right... if we do some work to synchronize the types for Cython modules
> generated by the same version of Cython, we're left with 3-4 types for
> Cython, right? Then a couple for numba and one for f2py; so on the order of
> 10?
>
> An alternative is do something funny in the type object to get across the
> offset-in-object information (abusing the docstring, or introduce our own
> flag which means that the type object has an additional non-standard field
> at the end).

In Python 2.7, it looks like there may be a few TP_FLAG bits free --
15 and 16 are labeled "reserved for stackless python", and 2, 11, 22
don't have anything defined.

There may also be an unused ssize_t field ob_size at the beginning of
the type object -- for some reason PyTypeObject is declared as
variable size (using PyObject_VAR_HEAD), but I don't see any
variable-size fields in it, the docs claim that the ob_size field is a
"historical artifact that is maintained for binary
compatibility...Always set this field to zero", and Include/object.h
has a definition for a PyHeapTypeObject which has a PyTypeObject as
its first member, which would not work if PyTypeObject had variable
size. Grep says that the only place where ob_type->ob_size is accessed
is in Objects/typeobject.c:object_sizeof(), which at first glance
appears to be a bug, and anyway I don't think anyone cares whether
__sizeof__ on C-callable objects is exactly correct. One could use
this for an offset, or even a pointer.

One could also add a field easily by just subclassing PyTypeObject.

The Signature thing seems like a distraction to me. Signature is
intended as just a nice convenient format for looking up stuff that's
otherwise stored in more obscure ways -- the API equivalent of
pretty-printing. The important thing here is getting the C-level
dispatch right.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Fri, Apr 13, 2012 at 12:59 PM, Dag Sverre Seljebotn
 wrote:
> I'll go one further: Intern Python bytes objects. It's just a PyObject*, but
> it's *required* (or just strongly encouraged) to have gone through
>
> sig = sys.modules['_nativecall']['interned_db'].setdefault(sig, sig)
>
> Obviously in a PEP you'd have a C-API function for such interning
> (completely standalone utility). Performance of interning operation itself
> doesn't matter...
>
> Unless CPython has interning features itself, like in Java? Was that present
> back in the day and then ripped out?

http://docs.python.org/library/functions.html#intern ? (C API:
PyString_InternInPlace, moved from __builtin__.intern to sys.intern in
Py3.)

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Fri, Apr 13, 2012 at 9:27 PM, Dag Sverre Seljebotn
 wrote:
> Ah, I didn't think about 6-bit or huffman. Certainly helps.
>
> I'm almost +1 on your proposal now, but a couple of more ideas:
>
> 1) Let the key (the size_t) spill over to the next specialization entry if
> it is too large; and prepend that key with a continuation code (two size-ts
> could together say "iii)-d\0\0" on 32 bit systems with 8bit encoding, using
> - as continuation). The key-based caller will expect a continuation if it
> knows about the specialization, and the prepended char will prevent spurios
> matches against the overspilled slot.
>
> We could even use the pointers for part of the continuation...

I am really lost here. Why is any of this complicated encoding stuff
better than interning? Interning takes one line of code, is incredibly
cheap (one dict lookup per call site and function definition), and it
lets you check any possible signature (even complicated ones involving
memoryviews) by doing a single-word comparison. And best of all, you
don't have to think hard to make sure you got the encoding right. ;-)

On a 32-bit system, pointers are smaller than a size_t, but more
expressive! You can still do binary search if you want, etc. Is the
problem just that interning requires a runtime calculation? Because I
feel like C users (like numpy) will want to compute these compressed
codes at module-init anyway, and those of us with a fancy compiler
capable of computing them ahead of time (like Cython) can instruct
that fancy compiler to compute them at module-init time just as
easily?

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Fri, Apr 13, 2012 at 11:22 PM, Dag Sverre Seljebotn
 wrote:
>
>
> Robert Bradshaw  wrote:
>
>>On Fri, Apr 13, 2012 at 2:24 PM, Nathaniel Smith  wrote:
>>> On Fri, Apr 13, 2012 at 9:27 PM, Dag Sverre Seljebotn
>>>  wrote:
>>>> Ah, I didn't think about 6-bit or huffman. Certainly helps.
>>>>
>>>> I'm almost +1 on your proposal now, but a couple of more ideas:
>>>>
>>>> 1) Let the key (the size_t) spill over to the next specialization
>>entry if
>>>> it is too large; and prepend that key with a continuation code (two
>>size-ts
>>>> could together say "iii)-d\0\0" on 32 bit systems with 8bit
>>encoding, using
>>>> - as continuation). The key-based caller will expect a continuation
>>if it
>>>> knows about the specialization, and the prepended char will prevent
>>spurios
>>>> matches against the overspilled slot.
>>>>
>>>> We could even use the pointers for part of the continuation...
>>>
>>> I am really lost here. Why is any of this complicated encoding stuff
>>> better than interning? Interning takes one line of code, is
>>incredibly
>>> cheap (one dict lookup per call site and function definition), and it
>>> lets you check any possible signature (even complicated ones
>>involving
>>> memoryviews) by doing a single-word comparison. And best of all, you
>>> don't have to think hard to make sure you got the encoding right. ;-)
>>>
>>> On a 32-bit system, pointers are smaller than a size_t, but more
>>> expressive! You can still do binary search if you want, etc. Is the
>>> problem just that interning requires a runtime calculation? Because I
>>> feel like C users (like numpy) will want to compute these compressed
>>> codes at module-init anyway, and those of us with a fancy compiler
>>> capable of computing them ahead of time (like Cython) can instruct
>>> that fancy compiler to compute them at module-init time just as
>>> easily?
>>
>>Good question.
>>
>>The primary disadvantage of interning that I see is memory locality. I
>>suppose if all the C-level caches of interned values were co-located,
>>this may not be as big of an issue. Not being able to compare against
>>compile-time constants may thwart some optimization opportunities, but
>>that's less clear.

I would like to see some demonstration of this. E.g., you can run this:

echo -e '#include \nint main(int argc, char ** argv) {
return strcmp(argv[0], "a"); }' | gcc -S -x c - -o - -O2 | less

Looks to me like for a short, known-at-compile-time string, with
optimization on, gcc implements it by basically sticking the string in
a global variable and then using a pointer... (If I do argv[0] ==
(char *)0x1234, then it places the constant value directly into the
instruction stream. Strangely enough, it does *not* inline the
constant value even if I do memcmp(&argv[0], "\1\2\3\4", 4), which
should be exactly equivalent...!)

I think gcc is just as likely to stick a bunch of
  static void * interned_dd_to_d;
  static void * interned_ll_to_l;
next to each other in the memory image as it is to stick a bunch of
equivalent manifest constants. If you're worried, make it static void
* interned_signatures[NUM_SIGNATURES] -- then they'll definitely be
next to each other.

>>It also requires coordination common repository, but I suppose one
>>would just stick a set in some standard module (or leverage Python's
>>interning).
>
> More problems:
>
> 1) It doesn't work well with multiple interpreter states. Ok, nothing works 
> with that at the moment, but it is on the roadmap for Python and we should 
> not make it worse.

This isn't a criticism, but I'd like to see a reference to the work in
this direction! My impression was that it's been on the roadmap for
maybe a decade, in a really desultory fashion:
  
http://docs.python.org/faq/library.html#can-t-we-get-rid-of-the-global-interpreter-lock
So if it's actually happening that's quite interesting.

> You basically *need* a thread safe store separate from any python 
> interpreter; though pythread.h does not rely on the interpreter state; which 
> helps.

Anyway, yes, if you can't rely on the interpreter than you'd need some
place to store the intern table, but I'm not sure why this would be a
problem (in Python 3.6 or whenever it becomes relevant).

> 2) you end up with the known comparison values in read-write memory segments 
> rather than readonly segments, which is probably worse on multicore systems?

Is it? Can you elaborate? Cache p

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Sun, Apr 15, 2012 at 9:15 AM, Dag Sverre Seljebotn
 wrote:
> Do you really think it complicates the spec? SHA-1 is pretty standard, and
> Python ships with hashlib (the hashing part isn't performance critical).
>
> I prefer hashing to string-interning as it can still be done compile-time
> etc. 160 bits isn't worse than the second-to-best strcmp case of a 256-bit
> function entry.

If you're *so* set on compile-time calculation, one could also
accommodate these within the intern framework pretty easily. Any
PyString/PyBytes * will be aligned, which means the low bit will not
be set, which means there are at least 2**31 bit-patterns that will
never be used by a run-time interned string. So we could write down a
lookup table in the spec that assigns arbitrary, well-known numbers to
every common signature. "dd->d" is 1, "ii->i" is 2, etc. If you have
15 standard types, then you can assign such an id to every 0, 1, 2, 3,
4, 5, and 6 argument function with space left over.

And this could all be abstracted away inside the intern() function.
The only thing is that if you wanted to look at the characters in the
interned string, you'd have to call a disintern() function instead of
just following the pointer.

I still think all this stuff would be complexity for its own sake, though.

> Shortening the hash to 120 bits (truncation) we could have a spec like this:
>
>  - Short signature: [64 bit encoded signature. 64 bit funcptr]
>  - Long signature: [64 bit hash, 64 bit pointer to full signature,
>                    8 bit guard byte, 56 bits remaining hash,
>                    64 bit funcptr]

This is a fixed length encoding, so why does it need a guard byte?

BTW, the guard byte design in the last version of the CEP looks buggy
to me -- there's no guarantee that a valid pointer might not contain
the guard byte by accident. A solution would be to move the
to-be-continued byte (or bit) to the first word. This would also mean
that if you're looking for a one-word signature via switch(), you
won't hit signatures which have your signature as a prefix. In the
variable-length encoding with the lookup rule you suggested you'd also
want a second bit to mark the actual beginning of each structure, so
you don't get hits on the middle of structures.

> Anyway: Looks like it's about time to do some benchmarks. I'll try to get
> around to it next week.

 Agreed :-).

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Sun, Apr 15, 2012 at 9:07 AM, Dag Sverre Seljebotn
 wrote:
> On 04/15/2012 09:30 AM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 15.04.2012 08:58:
>>>
>>> Ah, Cython objects. Didn't think of that. More below.
>>>
>>> On 04/14/2012 11:02 PM, Stefan Behnel wrote:

 thanks for writing this up. Comments inline as I read through it.

 Dag Sverre Seljebotn, 14.04.2012 21:08:
>
> each described by a function pointer and a signature specification
> string, such as "id)i" for {{{int f(int, double)}}}.


 How do we deal with object argument types? Do we care on the caller
 side?
 Functions might have alternative signatures that differ in the type of
 their object parameters. Or should we handle this inside of the caller
 and
 expect that it's something like a fused function with internal dispatch
 in
 that case?

 Personally, I think there is not enough to gain from object parameters
 that
 we should handle it on the caller side. The callee can dispatch those if
 necessary.

 What about signatures that require an object when we have a C typed
 value?

 What about signatures that require a C typed argument when we have an
 arbitrary object value in our call parameters?

 We should also strip the "self" argument from the parameter list of
 methods. That's handled by the attribute lookup before even getting at
 the
 callable.
>>>
>>>
>>> On 04/15/2012 07:59 AM, Robert Bradshaw wrote:

 It would certainly be useful to have special syntax for memory views
 (after nailing down a well-defined ABI for them) and builtin types.
 Being able to declare something as taking a
 "sage.rings.integer.Integer" could also prove useful, but could result
 in long (and prefix-sharing) signatures, favoring the
 runtime-allocated ids.
>>>
>>>
>>> I do think describing Cython objects in this cross-tool CEP would work
>>> nicely, this is for standardized ABIs only (we can't do memoryviews
>>> either
>>> until their ABI is standard).
>>
>>
>> It just occurred to me that an object's type can safely be represented at
>> runtime as a pointer, i.e. an integer. Even if the type is heap allocated
>> and replaced by another one later, a signature that uses that pointer
>> value
>> in its encoding would only ever match if both sides talk about the same
>> type at call time (because at least one of them would hold a life
>> reference
>> to the type in order to actually use it).
>
>
> The missing piece here is that both me and Robert are huge fans of Go-style
> polymorphism. If you haven't read up on that I highly recommend it, basic
> idea is if you agree on method names and their signatures, you don't have to
> have access to the same interface declaration (you don't have to call the
> interface the same thing).

Go style polymorphism is certainly a neat idea, but two points:

- You can't do this kind of matching via signature comparison. If I
have a type with methods "foo", "bar" and "baz", then that should
match the interface {"foo", "bar", "baz"}, but also {"foo", "bar"},
{"foo", "baz"}, {"bar"}, {}, etc. To find the right function for such
a type, you need to decode each function signature and check them in
some structured way. Unless your plan is to precompute the hash of all
2**n interfaces that each object fulfills.

- Adding a whole new type system with polymorphic dispatch is a heck
of a thing to do in a spec for boxing and unboxing pointers. Honestly
at this level I'm even leery of describing Python objects via their
type, as opposed to just "PyObject *". Just let the callee do the type
checking if they need to, and if it later turns out that there are
actually enough cases where Cython knows the exact type at compile
time and is dispatching through a boxed pointer and the callee type
checking is significant overhead, then extend the spec then.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Tue, Apr 17, 2012 at 12:54 PM, Dag Sverre Seljebotn
 wrote:
> I don't believe doing interning right without a common dependency .so is all
> that easy. I'd love to see a concrete spec for it (e.g., if you use Python
> bytes in a dict in sys.modules['_nativecall'], the bytes objects could be
> deallocated before callables containing the interned string -- unless you
> Py_INCREF once too many, but then valgrind complains -- and so on).

I don't understand. A C-callable object would hold a reference to each
interned string it contains, just like any other Python data structure
does. When the C-callable object is deallocated, then it drops this
reference, and if it there are no other live C-callable objects that
use the same signature, and there are no callers that are statically
caching the signature, then it will drop out of the intern table.
Otherwise it doesn't.

Basically you do the memory management like you would for any other
bytes object, and everything just works.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Tue, Apr 17, 2012 at 1:53 PM, Dag Sverre Seljebotn
 wrote:
> On 04/17/2012 02:40 PM, Nathaniel Smith wrote:
>>
>> On Tue, Apr 17, 2012 at 12:54 PM, Dag Sverre Seljebotn
>>   wrote:
>>>
>>> I don't believe doing interning right without a common dependency .so is
>>> all
>>> that easy. I'd love to see a concrete spec for it (e.g., if you use
>>> Python
>>> bytes in a dict in sys.modules['_nativecall'], the bytes objects could be
>>> deallocated before callables containing the interned string -- unless you
>>> Py_INCREF once too many, but then valgrind complains -- and so on).
>>
>>
>> I don't understand. A C-callable object would hold a reference to each
>> interned string it contains, just like any other Python data structure
>> does. When the C-callable object is deallocated, then it drops this
>> reference, and if it there are no other live C-callable objects that
>> use the same signature, and there are no callers that are statically
>> caching the signature, then it will drop out of the intern table.
>> Otherwise it doesn't.
>
>
> Thanks!, I'm just being dense.
>
> In fact I was the one who first proposed just storing interned
> PyBytesObject* way back in the start of the thread, but it met opposition in
> favour of an interned char* or an allocated "string id"; perhaps that's why
> I shut the possibility out of my mind.
>
> Would we store just PyBytesObject* or a char* in addition? Is bytes a vararg
> object or does it wrap a char*? If the former I think we should just store
> the PyBytesObject*.

In 2.7, PyBytesObject is #defined to PyStringObject, and
stringobject.h says that PyStringObject is variable-size. So that's
fine.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Tue, Apr 17, 2012 at 1:24 PM, Dag Sverre Seljebotn
 wrote:
> OK, here's the benchmark code I've written:
>
> https://github.com/dagss/cep1000

This is great!

> Assumptions etc.:
>
>  - (Very) warm cache case is tested
>
>  - I compile and link libmycallable.so, libmycaller.so and ./bench; with
> -fPIC, to emulate the Python environment
>
>  - I use mostly pure C but use PyTypeObject in order to get the offsets
> to tp_flags etc right (I emulate the checking that would happen on a
> PyObject* according to CEP1000).
>
>  - The test function is "double f(double x) { return x * x; }
>
>  - The benchmark is run in a loop J=100 times (and time divided by
> J). This is repeated K=1 times and the minimum walltime of the K run
> is used. This gave very stable readings on my system.
>
> Fixing loop iterations:
>
> In the initial results I just scanned the overload list until
> NULL-termination. It seemed to me that the code generated for this
> scanning was the most important factor.
>
> Therefore I fixed the number of overloads as a known compile-time macro
> N *in the caller*. This is somewhat optimistic; however I didn't want to
> play with figuring out loop unrolling etc. at the same time, and
> hardcoding the length of the overload list sort of took that part out of
> the equation.

Since you've set this up... I have a suggestion for something that may
be worth trying, though I've hesitated to propose it seriously. And
that is, an API where instead of scanning a table, the C-callable
exposes a pointer-to-function, something like
  int get_funcptr(PyObject * self, PyBytesObject * signature, struct
c_function_info * out)

The rationale is, if we want to support JITed functions where new
function pointers may be generated on the fly, the array approach has
a serious problem. You have to decide how many array slots to allocate
ahead of time, and if you run out, then... too bad. I guess you get to
throw away one of the existing pointers (i.e., leak memory) to make
room. Adding an indirection here for the actual lookup means that a
JIT could use a more complex lookup structure if justified, while a
simple C function pointer could just hardcode this to a
signature-check + return, with no lookup step at all. It also would
give us a lot of flexibility for future optimizations (e.g., is it
worth sorting the lookup table in LRU order?). And it would allow for
a JIT to generate a C function pointer on first use, rather than
requiring the first use to go via the Python-level __call__ fallback.
(Which is pretty important when the first use is to fetch the function
pointer before entering an inner loop!)

OTOH the extra indirection will obviously have some overhead, so it'd
be nice to know if it's actually a problem.

> Table explanation:
>
>  - N: Number of overloads in list. For N=10, there's 9 non-matching
> overloads in the list before the matching 10 (but caller doesn't know
> this). For N=1, the caller knows this and optimize for a hit in the
> first entry.
>
>  - MISMATCHES: If set, the caller tries 4 non-matching signatures before
> hitting the final one. If not set, only the correct signature is tried.
>
>  - LIKELY: If set, a GCC likely() macro is used to expect that the signature
> matches.
>
>
> RESULTS:
>
> Direct call (and execution of!) the function in benchmark loop took 4.8 ns.
>
> An indirect dispatch through a function pointer of known type took 5.4 ns
>
> Notation below is (intern key), in ns
>
> N=1:
>  MISMATCHES=False:
>    LIKELY=True:     6.44  6.44
>    LIKELY=False:    7.52  8.06
>  MISMATCHES=True:   8.59  8.59
> N=10:
>  MISMATCHES=False:  17.19  19.20
>  MISMATCHES=True:   36.52  37.59
>
> To be clear, "intern" is an interned "char*" (comparison with a 64 bits
> global variable), while key is comparison of a size_t (comparison of a
> 64-bit immediate in the instruction stream).
>
> PRELIMINARY BENCHMARK CONCLUSION:
>
> Intern appears to be as fast or faster than strcmp.
>
> I don't know why (is the pointer offset to the global variable stored in
> less than 64 bits in the x86-64 instruction stream? What gdb (or other)
> commands would I use to figure that out?)

I don't know why. It's entirely possible that this is just an accident
of alignment or something. You're probably using, what, a 2 GHz CPU or
so? So we're talking about a difference on the order of 2-4 cycles.
(Actually, I'm surprised that LIKELY made any difference. The CPU
knows which branch you're going to take regardless; all the compiler
can do is try to improve memory locality for the "expected" path. But
your entire benchmark probably fits in L1, so why would memory
locality matter? Unless you got unlucky with cache associativity or
something...)

Generally, I think the conclusion I draw from these numbers is that in
the hot-cache case, the lookup overhead is negligible regardless of
how we do it. On my laptop (i7 L640 @ 2.13 GHz), a single L3 cache
miss costs 150 ns, and even an L2 miss is 30 ns.

-- Nathaniel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Tue, Apr 17, 2012 at 3:34 PM, Dag Sverre Seljebotn
 wrote:
> On 04/17/2012 04:20 PM, Nathaniel Smith wrote:
>> Since you've set this up... I have a suggestion for something that may
>> be worth trying, though I've hesitated to propose it seriously. And
>> that is, an API where instead of scanning a table, the C-callable
>> exposes a pointer-to-function, something like
>>   int get_funcptr(PyObject * self, PyBytesObject * signature, struct
>> c_function_info * out)
>
>
> Hmm. There's many ways to implement that function though. It shifts the
> scanning logic from the caller to the callee;

Yes, that's part of the point :-). Or, well, I guess the point is more
that it shifts the scanning logic from the ABI docs to the callee.

> you would need to call it
> multiple times for different signatures...

Yes, I'm not sure what I think about this -- there are arguments
either way for who should handle promotion. E.g., imagine the
following situation:

We have a JITable function
We have already JITed the int64 version of this function
Now we want to call it with an int32
Question: should we promote to int64, or should we JIT?

Later you write:
> if found in table:
>   do dispatch
> else if object supports get_funcptr:
>   call get_funcptr
> else:
>   python dispatch

If we do promotion during the table scanning, then we'll never call
get_funcptr and we'll never JIT an int32 version. OTOH, if we call
get_funcptr before doing promotion, then we'll end up calling
get_funcptr multiple times for different signatures regardless.

OTOOH, there are a *lot* of possible coercions for, say, a 3-argument
function with return, so just enumerating them is not necessarily a
good strategy. Possibly if get_functpr can't handle the initial
signature, it should return a table of signatures that it *is* willing
to handle... assuming that most callees will either be able to handle
a fixed set of types (cython variants) or else handle pretty much
anything (JIT), and only the former will reach this code path. Or we
could write down the allowed promotions (stealing from the C99 spec),
and require the callee to pick the best promotion if it can't handle
the initial request. Or we could put this part off until version 2,
once we see how eager callers are to actually implement a real
promotion engine.

> But if the overhead can be shown to be miniscule then it does perhaps make
> the API nicer, even if it feels like paying for nothing at the moment. But
> see below.
>
> Will definitely not get around to this today; anyone else feel free...
>
>
>>
>> The rationale is, if we want to support JITed functions where new
>> function pointers may be generated on the fly, the array approach has
>> a serious problem. You have to decide how many array slots to allocate
>> ahead of time, and if you run out, then... too bad. I guess you get to
>
>
> Note that the table is jumped to by a pointer in the PyObject, i.e. the
> PyObject I've tested with is
>
> [object data, &table, table]

Oh, I see! I thought you were embedding it in the object, to avoid an
extra indirection (and potential cache miss). That's probably
necessary, for the reasons you say, but also makes the get_funcptr
approach potentially more competitive.

> So a JIT could have the table in a separate location on the heap, then it
> can allocate a new table, copy over the contents, and when everything is
> ready, then do an atomic pointer update (using the assembly instructions/gcc
> intrinsics, not pthreads or locking).
>
> The old table would need to linger for a bit, but could at latest be
> deallocated when the PyObject is deallocated.

IMHO we should just hold the GIL through lookups, which would simplify
tihs, but that's mostly based on the naive intuition that we shouldn't
be passing around Python boxes in no-GIL code. Maybe there are good
reasons to.

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Tue, Apr 17, 2012 at 8:07 PM, Dag Sverre Seljebotn
 wrote:
>
>
> Nathaniel Smith  wrote:
>
>>On Tue, Apr 17, 2012 at 3:34 PM, Dag Sverre Seljebotn
>> wrote:
>>> On 04/17/2012 04:20 PM, Nathaniel Smith wrote:
>>>> Since you've set this up... I have a suggestion for something that
>>may
>>>> be worth trying, though I've hesitated to propose it seriously. And
>>>> that is, an API where instead of scanning a table, the C-callable
>>>> exposes a pointer-to-function, something like
>>>>   int get_funcptr(PyObject * self, PyBytesObject * signature, struct
>>>> c_function_info * out)
>>>
>>>
>>> Hmm. There's many ways to implement that function though. It shifts
>>the
>>> scanning logic from the caller to the callee;
>>
>>Yes, that's part of the point :-). Or, well, I guess the point is more
>>that it shifts the scanning logic from the ABI docs to the callee.
>
> Well, really it shifts the logic to the getfuncptr argument specification -- 
> is the signature argument an interned string, encoded string, sha1 hash,...
>
> Part of the table storage format is shifted from the CEP but that is so 
> unimportant it has not even been discussed.
>
>>
>>> you would need to call it
>>> multiple times for different signatures...
>>
>>Yes, I'm not sure what I think about this -- there are arguments
>>either way for who should handle promotion. E.g., imagine the
>>following situation:
>>
>>We have a JITable function
>>We have already JITed the int64 version of this function
>>Now we want to call it with an int32
>>Question: should we promote to int64, or should we JIT?
>
> I think we got close to a good solution to this dilemma earlier in this 
> thread:
>
>  - Callers promote scalars to 64 bit if no exact match is found (and JITs 
> only use 64 bit scalars)
>
>  - Arrays and pointers are the real issue. In this case the caller request 
> another signature (and the JIT kicks in)
>
> The utility of re-jiting for scalars is very limited; it is vital for arrays 
> and pointers.

Nonetheless, I think these rules will run into some trouble (starting
with, JITs only use long double?), and esp. if you want to convince
python-dev of them. But again, I don't think it's so terrible if the
caller just picks some different signatures that it's willing to deal
with, for now.

>
>>
>>Later you write:
>>> if found in table:
>>>   do dispatch
>>> else if object supports get_funcptr:
>>>   call get_funcptr
>>> else:
>>>   python dispatch
>>
>>If we do promotion during the table scanning, then we'll never call
>>get_funcptr and we'll never JIT an int32 version. OTOH, if we call
>>get_funcptr before doing promotion, then we'll end up calling
>>get_funcptr multiple times for different signatures regardless.
>>
>>OTOOH, there are a *lot* of possible coercions for, say, a 3-argument
>>function with return, so just enumerating them is not necessarily a
>>good strategy. Possibly if get_functpr can't handle the initial
>>signature, it should return a table of signatures that it *is* willing
>>to handle... assuming that most callees will either be able to handle
>>a fixed set of types (cython variants) or else handle pretty much
>>anything (JIT), and only the former will reach this code path. Or we
>>could write down the allowed promotions (stealing from the C99 spec),
>>and require the callee to pick the best promotion if it can't handle
>>the initial request. Or we could put this part off until version 2,
>>once we see how eager callers are to actually implement a real
>>promotion engine.
>
> I wanted to leave getfuncptr for another CEP.
>
> There's all kind of stuff -- how does the JIT determine that the argument 
> arrays are large enough to justify JITing? Etc.

I'm sort of inclined to follow KISS here, and say that this isn't
PyPy, we aren't trying to get optimal performance on large, arbitrary
programs. If someone took the trouble to write a function in a special
JIT-able Python subset/dialect and then passed it to a C code, it's
because they know that JITing is worth it we and should just do it
unconditionally. Maybe that'll have to be revised later, but it seems
like a plausible way to get started...

Anyway, getfuncptr alone is actually simpler spec-wise than the array
lookup approach, and the flexibility is an added bonus; it's just a
question of whether it will work.

>>
>>> But if the overhead can be shown to be miniscule then it does

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Wed, Apr 18, 2012 at 10:58 PM, Dag Sverre Seljebotn
 wrote:
> On 04/18/2012 11:35 PM, Dag Sverre Seljebotn wrote:
>>
>> On 04/17/2012 02:24 PM, Dag Sverre Seljebotn wrote:
>>>
>>> On 04/13/2012 12:11 AM, Dag Sverre Seljebotn wrote:

 Travis Oliphant recently raised the issue on the NumPy list of what
 mechanisms to use to box native functions produced by his Numba so
 that SciPy functions can call it, e.g. (I'm making the numba part
 up):

 @numba # Compiles function using LLVM def f(x): return 3 * x

 print scipy.integrate.quad(f, 1, 2) # do many callbacks natively!

 Obviously, we want something standard, so that Cython functions can
 also be called in a fast way.
>>>
>>>
>>> OK, here's the benchmark code I've written:
>>>
>>> https://github.com/dagss/cep1000
>>>
>>> Assumptions etc.:
>>>
>>> - (Very) warm cache case is tested
>>>
>>> - I compile and link libmycallable.so, libmycaller.so and ./bench; with
>>> -fPIC, to emulate the Python environment
>>>
>>> - I use mostly pure C but use PyTypeObject in order to get the offsets
>>> to tp_flags etc right (I emulate the checking that would happen on a
>>> PyObject* according to CEP1000).
>>>
>>> - The test function is "double f(double x) { return x * x; }
>>>
>>> - The benchmark is run in a loop J=100 times (and time divided by
>>> J). This is repeated K=1 times and the minimum walltime of the K run
>>> is used. This gave very stable readings on my system.
>>>
>>> Fixing loop iterations:
>>>
>>> In the initial results I just scanned the overload list until
>>> NULL-termination. It seemed to me that the code generated for this
>>> scanning was the most important factor.
>>>
>>> Therefore I fixed the number of overloads as a known compile-time macro
>>> N *in the caller*. This is somewhat optimistic; however I didn't want to
>>> play with figuring out loop unrolling etc. at the same time, and
>>> hardcoding the length of the overload list sort of took that part out of
>>> the equation.
>>>
>>>
>>> Table explanation:
>>>
>>> - N: Number of overloads in list. For N=10, there's 9 non-matching
>>> overloads in the list before the matching 10 (but caller doesn't know
>>> this). For N=1, the caller knows this and optimize for a hit in the
>>> first entry.
>>>
>>> - MISMATCHES: If set, the caller tries 4 non-matching signatures before
>>> hitting the final one. If not set, only the correct signature is tried.
>>>
>>> - LIKELY: If set, a GCC likely() macro is used to expect that the
>>> signature matches.
>>>
>>>
>>> RESULTS:
>>>
>>> Direct call (and execution of!) the function in benchmark loop took
>>> 4.8 ns.
>>>
>>> An indirect dispatch through a function pointer of known type took 5.4 ns
>>>
>>> Notation below is (intern key), in ns
>>>
>>> N=1:
>>> MISMATCHES=False:
>>> LIKELY=True: 6.44 6.44
>>> LIKELY=False: 7.52 8.06
>>> MISMATCHES=True: 8.59 8.59
>>> N=10:
>>> MISMATCHES=False: 17.19 19.20
>>> MISMATCHES=True: 36.52 37.59
>>>
>>> To be clear, "intern" is an interned "char*" (comparison with a 64 bits
>>> global variable), while key is comparison of a size_t (comparison of a
>>> 64-bit immediate in the instruction stream).
>>
>>
>> First: My benchmarks today are a little inconsistent with earlier
>> results. I think I have converged now in terms of number of iterations
>> (higher than last time), but that doesn't explain why indirect dispatch
>> through function pointer is now *higher*:
>>
>> Direct took 4.83 ns
>> Dispatch took 5.91 ns
>>
>> Anyway, even if crude, hopefully this will tell us something. Order of
>> benchmark numbers are:
>>
>> intern key get_func_intern get_func_key
>>
>> where the get_func_XX versions retrieve a function pointer taking either
>> a single interned signature or a single key as argument (just see
>> mycallable.c).
>>
>> In the MISMATCHES case, the get_func_XX is called 4 times with a miss
>> and then with the match.
>>
>> N=1
>> - MISMATCHES=False:
>> --- LIKELY=True: 5.91 6.44 8.59 9.13
>> --- LIKELY=False: 7.52 7.52 9.13 9.13
>> - MISMATCHES=True: 11.28 11.28 22.56 22.56
>>
>> N=10
>> - MISMATCHES=False: 17.18 18.80 29.75 10.74(*)
>> - MISMATCHES=True: 36.06 38.13 105.00 36.52
>>
>> Benchmark comments:
>>
>> The one marked (*) is implemented as a switch statement with known keys
>> compile-time. I tried shifting around the case label values a bit but
>> the result persists; it could just be that the compiler does a very good
>> job of the switch as well.
>
>
> I should make this clearer: The issue is that the compiler may have
> reordered the labels so that the hit came close to first; in the intern case
> the code is written so that the hit is always after 9 mismatches.
>
> So I redid the (*) test using 10 cases with very different numeric values,
> and then tried each 10 as the matching case. Timings were stable for each
> choice of label (so this is not noise), with values:
>
> 13.4 11.8 11.8 12.3 10.7 11.2 12.3
>
> Guess this is the binary decision tree Mark talk

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Thu, Apr 19, 2012 at 11:56 AM, Dag Sverre Seljebotn
 wrote:
> I thought of some drawbacks of getfuncptr:
>
>  - Important: Doesn't allow you to actually inspect the supported
> signatures, which is needed (or at least convenient) if you want to use an
> FFI library or do some JIT-ing. So an iteration mechanism is still needed in
> addition, meaning the number of things for the object to implement grows a
> bit large. Default implementations help -- OTOH there really wasn't a major
> drawback with the table approach as long as JIT's can just replace it?

But this is orthogonal to the table vs. getfuncptr discussion. We're
assuming that the table might be extended at runtime, which means you
can't use it to determine which signatures are supported. So we need
some sort of extra interface for the caller and callee to negotiate a
type anyway. (I'm intentionally agnostic about whether it makes more
sense for the caller or the callee to be doing the iterating... in
general type negotiation could be quite complicated, and I don't think
we know enough to get that interface right yet.)

The other other option would be to go to the far other end of
simplicity, and just forget for now about allowing multiple signatures
in the same object. Do signature selection by having the user select
one explicitly:

@cython.inline
def square(x):
return x * x

# .specialize is an un-standardized Cython interface
# square_double is an object implementing the standardized C-callable interface
square_double = square.specialize("d->d")
scipy.integrate.quad(square_double)

That'd be enough to get started, and doesn't rule out later extensions
that do automatic type selection, once we have more experience.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Nathaniel Smith]

On Thu, Apr 19, 2012 at 2:18 PM, Dag Sverre Seljebotn
 wrote:
> On 04/19/2012 01:20 PM, Nathaniel Smith wrote:
>> @cython.inline
>> def square(x):
>>     return x * x
>>
>> # .specialize is an un-standardized Cython interface
>> # square_double is an object implementing the standardized C-callable
>> interface
>> square_double = square.specialize("d->d")
>> scipy.integrate.quad(square_double)
>>
>> That'd be enough to get started, and doesn't rule out later extensions
>> that do automatic type selection, once we have more experience.
>
>
> Well, I want np.sin to replace "cdef extern from 'math.h'", and then this
> seems to be needed... at least the possibility to have both "d->d" and
> "O->O".

Except, the C function implementing np.sin on doubles actually has a
signature that's something like "&&t&i&i&t->"
(PyUFuncGenericFunction), not "d->d"... so maybe this is a good
example to work through! It isn't at all obvious to me how this should
be made to work in any of these proposals.

(Isn't "O->O" just obj->ob_type->tp_call in any case?)

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Julialang

[Nathaniel Smith]

On Mon, Apr 23, 2012 at 6:09 PM, Dimitri Tcaciuc  wrote:
> I may be misuderstanding the intent here, but here it goes.
>
> If the main idea is to be able to call functions that are written in
> Julia or other languages, I think an effort to create an LLVM backend
> for Cython would go a long way towards inter-language connections as
> the one discussed here. It should be possible to take Cython- and
> Julia- produced LLVM bytecode and assemble it all together, applying
> whatever bytecode optimizers that are available (eg. SSE
> vectorization). A big advantage of that approach is that there's no
> need for one language to know syntax conventions of the other one (or
> at least not to full extent). Continuing the effort, it should be
> possible to eliminate the need for writing an intermediate .c/.cpp
> file if Clang compiler is used, which is also LLVM based.

You'd still need some way to translate between the Cython and Julia
calling conventions, runtimes, error handling, garbage collection
regimes, etc. IIUC, LLVM IR isn't like the CLR -- it doesn't force
languages into a common system for these things.

Which might be great and worth the effort, I don't know, and don't
want to discourage anyone. But there are literally hundreds of new
languages designed every year, and a new *successful* language comes
along maybe twice in a decade? And one of those recent ones was PHP,
which shows you how important pure technical quality is in determining
which ones survive (i.e., not much). Building a self-sustaining
ecosystem requires a ton of work and a ton of luck. And here I'm still
trying to *reduce* the number of languages I need in each analysis
pipeline... so even though there are a number of really exciting
things about Julia, and its author seems to know what he's doing, I'm
still in wait-and-see mode.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Wacky idea: proper macros

[Nathaniel Smith]

Was chatting with Wes today about the usual problem many of us have
encountered with needing to use some sort of templating system to
generate code handling multiple types, operations, etc., and a wacky
idea occurred to me. So I thought I'd through it out here.

What if we added a simple macro facility to Cython, that worked at the
AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
Basically some way to write arbitrary Python code into a .pyx file
that gets executed at compile time and can transform the AST, plus
some nice convenience APIs for simple transformations.

E.g., if we steal the illegal token sequence @@ as our marker, we
could have something like:

@@ # alone on a line, starts a block of Python code
from Cython.MacroUtil import replace_ctype
def expand_types(placeholder, typelist):
  def my_decorator(function_name, ast):
functions = {}
for typename in typelist:
  new_name = "%s_%s" % (function_name, typename)
  functions[name] = replace_ctype(ast, placeholder, typename)
return functions
  return function_decorator
@@ # this token sequence cannot occur in Python, so it's a safe end-marker

# Compile-time function decorator
# Results in two cdef functions named sum_double and sum_int
@@expand_types("T", ["double", "int"])
cdef T sum(np.ndarray[T] arr):
  cdef T start = 0;
  for i in range(arr.size):
start += arr[i]
  return start

I don't know if this is a good idea, but it seems like it'd be very
easy to do on the Cython side, fairly clean, and be dramatically less
horrible than all the ad-hoc templating stuff people do now.
Presumably there'd be strict limits on how much backwards
compatibility we'd be willing to guarantee for code that went poking
around in the AST by hand, but a small handful of functions like my
notional "replace_ctype" would go a long way, and wouldn't impose much
of a compatibility burden.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Wacky idea: proper macros

[Nathaniel Smith]

On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
 wrote:
> On 28 April 2012 22:04, Nathaniel Smith  wrote:
>> Was chatting with Wes today about the usual problem many of us have
>> encountered with needing to use some sort of templating system to
>> generate code handling multiple types, operations, etc., and a wacky
>> idea occurred to me. So I thought I'd through it out here.
>>
>> What if we added a simple macro facility to Cython, that worked at the
>> AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
>> Basically some way to write arbitrary Python code into a .pyx file
>> that gets executed at compile time and can transform the AST, plus
>> some nice convenience APIs for simple transformations.
>>
>> E.g., if we steal the illegal token sequence @@ as our marker, we
>> could have something like:
>>
>> @@ # alone on a line, starts a block of Python code
>> from Cython.MacroUtil import replace_ctype
>> def expand_types(placeholder, typelist):
>>  def my_decorator(function_name, ast):
>>    functions = {}
>>    for typename in typelist:
>>      new_name = "%s_%s" % (function_name, typename)
>>      functions[name] = replace_ctype(ast, placeholder, typename)
>>    return functions
>>  return function_decorator
>> @@ # this token sequence cannot occur in Python, so it's a safe end-marker
>>
>> # Compile-time function decorator
>> # Results in two cdef functions named sum_double and sum_int
>> @@expand_types("T", ["double", "int"])
>> cdef T sum(np.ndarray[T] arr):
>>  cdef T start = 0;
>>  for i in range(arr.size):
>>    start += arr[i]
>>  return start
>>
>> I don't know if this is a good idea, but it seems like it'd be very
>> easy to do on the Cython side, fairly clean, and be dramatically less
>> horrible than all the ad-hoc templating stuff people do now.
>> Presumably there'd be strict limits on how much backwards
>> compatibility we'd be willing to guarantee for code that went poking
>> around in the AST by hand, but a small handful of functions like my
>> notional "replace_ctype" would go a long way, and wouldn't impose much
>> of a compatibility burden.
>>
>> -- Nathaniel
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
>
> In general I would like better meta-programming support, maybe even
> allow defining new operators (although I'm not sure any of it is very
> pythonic), but for templates I think fused types should be used, or
> improved when they fall short. Maybe a plugin system could also help
> people.

I hadn't seen that, no -- thanks for the link.

I have to say that the examples in that link, though, give me the
impression of a cool solution looking for a problem. I've never wished
I could symbolically differentiate Python expressions at compile time,
or create a mutant Python+SQL hybrid language. Actually I guess I've
only missed define-syntax once in maybe 10 years of hacking in
Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
will peek at the caller's syntax tree to automagically label the axes
as "x" and "log(y)", and that can't be done in Python. But that's not
exactly a convincing argument for a macro system.

But generating optimized code is Cython's whole selling point, and
people really are doing klugey tricks with string-based preprocessors
just to generate multiple copies of loops in Cython and C.

Also, fused types are great, but: (1) IIUC you can't actually do
ndarray[fused_type] yet, which speaks to the feature's complexity, and
(2) to handle Wes's original example on his blog (duplicating a bunch
of code between a "sum" path and a "product" path), you'd actually
need something like "fused operators", which aren't even on the
horizon. So it seems unlikely that fused types will grow to cover all
these cases in the near future.

Of course some experimentation would be needed to find the right
syntax and convenience functions for this feature too, so maybe I'm
just being over-optimistic and it would also turn out to be very
complicated :-). But it seems like some simple AST search/replace
functions would get you a long way.

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Wacky idea: proper macros

[Nathaniel Smith]

On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn
 wrote:
> JIT is really the way to go. It is one thing that a JIT could optimize the
> case where you pass a callback to a function and inline it run-time. But
> even if it doesn't get that fancy, it'd be great to just be able to write
> something like "cython.eval(s)" and have that be compiled (I guess you could
> do that now, but the sheer overhead of the C compiler and all the .so files
> involved means nobody would sanely use that as the main way of stringing
> together something like pandas).

The overhead of running a fully optimizing compiler over pandas on
every import is pretty high, though. You can come up with various
caching mechanisms, but they all mean introducing some kind of compile
time/run time distinction. So I'm skeptical we'll just be able to get
rid of that concept, even in a brave new LLVM/PyPy/Julia world.

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP 1001 - Custom PyTypeObject extensions

[Nathaniel Smith]

On Fri, May 11, 2012 at 2:25 PM, Dag Sverre Seljebotn
 wrote:
> This comes from a refactor of the work on CEP 1000: A PEP proposal, with a
> hack for use in current Python versions and in the case of PEP rejection,
> that allows 3rd party libraries to agree on extensions to PyTypeObject.
>
> http://wiki.cython.org/enhancements/cep1001
>
> If this makes it as a PEP, I don't think we need to think about having CEP
> 1000 accepted as a PEP.
>
> Comments?

There should probably be some discussion of memory management for the
tpe_data pointers. (I assume it's "guaranteed to be valid for as long
as the associated PyTypeObject, and the PyTypeObject is responsible
for making sure any necessary cleanup happens if it gets deallocated",
but a note to this effect would be good.)

What happens if I want to inherit from PyTypeObject (a "metaclass")
and also implement this interface? It is possible? What if I want to
inherit from an existing subclass of PyTypeObject and add on this
interface? I don't know enough gnarly details about how new style
classes are implemented to tell. Would it make sense to make this
memory-layout-equivalent to a PyTypeObject subclass with extra fields?

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP 1001 - Custom PyTypeObject extensions

[Nathaniel Smith]

On Sun, May 13, 2012 at 8:35 PM, Dag Sverre Seljebotn
 wrote:
> On 05/12/2012 08:44 PM, Nathaniel Smith wrote:
>>
>> On Fri, May 11, 2012 at 2:25 PM, Dag Sverre Seljebotn
>>   wrote:
>>>
>>> This comes from a refactor of the work on CEP 1000: A PEP proposal, with
>>> a
>>> hack for use in current Python versions and in the case of PEP rejection,
>>> that allows 3rd party libraries to agree on extensions to PyTypeObject.
>>>
>>> http://wiki.cython.org/enhancements/cep1001
>>>
>>> If this makes it as a PEP, I don't think we need to think about having
>>> CEP
>>> 1000 accepted as a PEP.
>>>
>>> Comments?
>>
>>
>> There should probably be some discussion of memory management for the
>> tpe_data pointers. (I assume it's "guaranteed to be valid for as long
>> as the associated PyTypeObject, and the PyTypeObject is responsible
>> for making sure any necessary cleanup happens if it gets deallocated",
>> but a note to this effect would be good.)
>>
>> What happens if I want to inherit from PyTypeObject (a "metaclass")
>> and also implement this interface? It is possible? What if I want to
>> inherit from an existing subclass of PyTypeObject and add on this
>> interface? I don't know enough gnarly details about how new style
>> classes are implemented to tell. Would it make sense to make this
>> memory-layout-equivalent to a PyTypeObject subclass with extra fields?
>
>
> Hmm. You know what -- this whole thing could probably be a metaclass.

Well, yes, conceptually, that's exactly what it is -- the question is
how and whether it relates to the Python metaclass machinery, since
you are speed freaks :-).

> Except
> I think a PyObject_TypeCheck on the type would be a bit more expensive than
> just checking a flag. I think I like having a flag better...

A number of existing flags are actually used exactly to make
type-checking faster for some key types (PY_TPFLAGS_INT_SUBCLASS,
etc.). I guess doing it the same way would put the flag in
obj->tp_class->tp_class->tp_flags, though, instead of
obj->tp_class->tp_flags.

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP 1001 - Custom PyTypeObject extensions

[Nathaniel Smith]

On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn
 wrote:
> On 05/14/2012 01:34 PM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 13.05.2012 21:37:
>>>
>>> Anyway, thanks for the heads up, this seems to need a bit more work.
>>> Input
>>> from somebody more familiar with this corner of the CPython API very
>>> welcome.
>>
>>
>> Wouldn't you consider python-dev an appropriate place to discuss this?
>
>
> Propose something for a PEP that's primarily useful to Cython without even
> understanding the full implications myself first?
>
> I'd rather try to not annoy people; I figured the time I have the CPython
> patches ready and tested is the time I ping python-dev...

If you want to eventually propose a PEP, you really really really
should be talking to them before. Otherwise you'll get everything
worked out just the way you want and they'll be like "what is this?
re-do it all totally differently". And they might be wrong, but then
you have to reconstruct for them the whole debate and reasoning
process and implicit assumptions that you're making and not realizing
you need to articulate, so easier to just get all the interested
people at the table to begin with. And they might be right, in which
case you just wasted however much time digging yourself into a hole
and reverse-engineering bits of CPython.

Don't propose it as a PEP, just say "hey, we have this problem and
these constraints, and we're thinking we could solve them by something
like this; but of course that has these limitations, so I dunno. What
do you think?" And expect to spend some time figuring out what your
requirements actually are (even if you think you know already, see
above about implicit assumptions).

-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [Python-Dev] C-level duck typing

[Nathaniel Smith]

On Sun, May 27, 2012 at 10:24 PM, Dag Sverre Seljebotn
 wrote:
> On 05/18/2012 10:30 AM, Dag Sverre Seljebotn wrote:
>>
>> On 05/18/2012 12:57 AM, Nick Coghlan wrote:
>>>
>>> I think the main things we'd be looking for would be:
>>> - a clear explanation of why a new metaclass is considered too complex a
>>> solution
>>> - what the implications are for classes that have nothing to do with the
>>> SciPy/NumPy ecosystem
>>> - how subclassing would behave (both at the class and metaclass level)
>>>
>>> Yes, defining a new metaclass for fast signature exchange has its
>>> challenges - but it means that *our* concerns about maintaining
>>> consistent behaviour in the default object model and avoiding adverse
>>> effects on code that doesn't need the new behaviour are addressed
>>> automatically.
>>>
>>> Also, I'd consider a functioning reference implementation using a custom
>>> metaclass a requirement before we considered modifying type anyway, so I
>>> think that's the best thing to pursue next rather than a PEP. It also
>>> has the virtue of letting you choose which Python versions to target and
>>> iterating at a faster rate than CPython.
>>
>>
>> This seems right on target. I could make a utility code C header for
>> such a metaclass, and then the different libraries can all include it
>> and handshake on which implementation becomes the real one through
>> sys.modules during module initialization. That way an eventual PEP will
>> only be a natural incremental step to make things more polished, whether
>> that happens by making such a metaclass part of the standard library or
>> by extending PyTypeObject.
>
>
> So I finally got around to implementing this:
>
> https://github.com/dagss/pyextensibletype
>
> Documentation now in a draft in the NumFOCUS SEP repo, which I believe is a
> better place to store cross-project standards like this. (The NumPy
> docstring standard will be SEP 100).
>
> https://github.com/numfocus/sep/blob/master/sep200.rst
>
> Summary:
>
>  - No common runtime dependency
>
>  - 1 ns overhead per lookup (that's for the custom slot *alone*, no
> fast-callable signature matching or similar)
>
>  - Slight annoyance: Types that want to use the metaclass must be a
> PyHeapExtensibleType, to make the binary layout work with how CPython makes
> subclasses from Python scripts
>
> My conclusion: I think the metaclass approach should work really well.

Few quick comments on skimming the code:

The complicated nested #ifdef for __builtin_expect could be simplified to
  #if defined(__GNUC__) && (__GNUC__ > 2 || __GNUC_MINOR__ > 95)

PyCustomSlots_Check should be called PyCustomSlots_CheckExact, surely?
And given that, how can this code work if someone does subclass this
metaclass?

Stealing a flag bit (but now to indicate this metaclass) would allow
us to make a real PyCustomSlots_Check function that was still fast. It
would also mean that different implementations didn't have to
rendezvous on a single PyExtensibleType_Type, so long as they all used
the same flag bit. That would let us skip monkeying around with
sys.modules.

Speaking of which, surely we should not be using sys.modules for this?
Stashing it in sys itself or something would make more sense, if we're
going to do it at all.

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [Python-Dev] C-level duck typing

[Nathaniel Smith]

On Mon, May 28, 2012 at 10:13 AM, mark florisson
 wrote:
> On 28 May 2012 09:54, mark florisson  wrote:
>> On 27 May 2012 23:12, Nathaniel Smith  wrote:
>>> On Sun, May 27, 2012 at 10:24 PM, Dag Sverre Seljebotn
>>>  wrote:
>>>> On 05/18/2012 10:30 AM, Dag Sverre Seljebotn wrote:
>>>>>
>>>>> On 05/18/2012 12:57 AM, Nick Coghlan wrote:
>>>>>>
>>>>>> I think the main things we'd be looking for would be:
>>>>>> - a clear explanation of why a new metaclass is considered too complex a
>>>>>> solution
>>>>>> - what the implications are for classes that have nothing to do with the
>>>>>> SciPy/NumPy ecosystem
>>>>>> - how subclassing would behave (both at the class and metaclass level)
>>>>>>
>>>>>> Yes, defining a new metaclass for fast signature exchange has its
>>>>>> challenges - but it means that *our* concerns about maintaining
>>>>>> consistent behaviour in the default object model and avoiding adverse
>>>>>> effects on code that doesn't need the new behaviour are addressed
>>>>>> automatically.
>>>>>>
>>>>>> Also, I'd consider a functioning reference implementation using a custom
>>>>>> metaclass a requirement before we considered modifying type anyway, so I
>>>>>> think that's the best thing to pursue next rather than a PEP. It also
>>>>>> has the virtue of letting you choose which Python versions to target and
>>>>>> iterating at a faster rate than CPython.
>>>>>
>>>>>
>>>>> This seems right on target. I could make a utility code C header for
>>>>> such a metaclass, and then the different libraries can all include it
>>>>> and handshake on which implementation becomes the real one through
>>>>> sys.modules during module initialization. That way an eventual PEP will
>>>>> only be a natural incremental step to make things more polished, whether
>>>>> that happens by making such a metaclass part of the standard library or
>>>>> by extending PyTypeObject.
>>>>
>>>>
>>>> So I finally got around to implementing this:
>>>>
>>>> https://github.com/dagss/pyextensibletype
>>>>
>>>> Documentation now in a draft in the NumFOCUS SEP repo, which I believe is a
>>>> better place to store cross-project standards like this. (The NumPy
>>>> docstring standard will be SEP 100).
>>>>
>>>> https://github.com/numfocus/sep/blob/master/sep200.rst
>>>>
>>>> Summary:
>>>>
>>>>  - No common runtime dependency
>>>>
>>>>  - 1 ns overhead per lookup (that's for the custom slot *alone*, no
>>>> fast-callable signature matching or similar)
>>>>
>>>>  - Slight annoyance: Types that want to use the metaclass must be a
>>>> PyHeapExtensibleType, to make the binary layout work with how CPython makes
>>>> subclasses from Python scripts
>>>>
>>>> My conclusion: I think the metaclass approach should work really well.
>>>
>>> Few quick comments on skimming the code:
>>>
>>> The complicated nested #ifdef for __builtin_expect could be simplified to
>>>  #if defined(__GNUC__) && (__GNUC__ > 2 || __GNUC_MINOR__ > 95)
>>>
>>> PyCustomSlots_Check should be called PyCustomSlots_CheckExact, surely?
>>> And given that, how can this code work if someone does subclass this
>>> metaclass?
>>
>> I think we should provide a wrapper for PyType_Ready, which just
>> copies the pointer to the table and the count directly into the
>> subclass. If a user then wishes to add stuff, the user can allocate a
>> new memory region dynamically, memcpy the base class' stuff in there,
>> and append some entries.
>
> Maybe we should also allow each custom type to set a deallocator,
> since they are then heap types which can go out of scope. The
> metaclass can then call this deallocator to deallocate the table.

Custom types are plain old Python objects, they can use tp_dealloc.

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [Python-Dev] C-level duck typing

[Nathaniel Smith]

On Mon, May 28, 2012 at 11:55 AM, mark florisson
 wrote:
> On 28 May 2012 11:41, Nathaniel Smith  wrote:
>> On Mon, May 28, 2012 at 10:13 AM, mark florisson
>>  wrote:
>>> On 28 May 2012 09:54, mark florisson  wrote:
>>>> On 27 May 2012 23:12, Nathaniel Smith  wrote:
>>>>> On Sun, May 27, 2012 at 10:24 PM, Dag Sverre Seljebotn
>>>>>  wrote:
>>>>>> On 05/18/2012 10:30 AM, Dag Sverre Seljebotn wrote:
>>>>>>>
>>>>>>> On 05/18/2012 12:57 AM, Nick Coghlan wrote:
>>>>>>>>
>>>>>>>> I think the main things we'd be looking for would be:
>>>>>>>> - a clear explanation of why a new metaclass is considered too complex 
>>>>>>>> a
>>>>>>>> solution
>>>>>>>> - what the implications are for classes that have nothing to do with 
>>>>>>>> the
>>>>>>>> SciPy/NumPy ecosystem
>>>>>>>> - how subclassing would behave (both at the class and metaclass level)
>>>>>>>>
>>>>>>>> Yes, defining a new metaclass for fast signature exchange has its
>>>>>>>> challenges - but it means that *our* concerns about maintaining
>>>>>>>> consistent behaviour in the default object model and avoiding adverse
>>>>>>>> effects on code that doesn't need the new behaviour are addressed
>>>>>>>> automatically.
>>>>>>>>
>>>>>>>> Also, I'd consider a functioning reference implementation using a 
>>>>>>>> custom
>>>>>>>> metaclass a requirement before we considered modifying type anyway, so 
>>>>>>>> I
>>>>>>>> think that's the best thing to pursue next rather than a PEP. It also
>>>>>>>> has the virtue of letting you choose which Python versions to target 
>>>>>>>> and
>>>>>>>> iterating at a faster rate than CPython.
>>>>>>>
>>>>>>>
>>>>>>> This seems right on target. I could make a utility code C header for
>>>>>>> such a metaclass, and then the different libraries can all include it
>>>>>>> and handshake on which implementation becomes the real one through
>>>>>>> sys.modules during module initialization. That way an eventual PEP will
>>>>>>> only be a natural incremental step to make things more polished, whether
>>>>>>> that happens by making such a metaclass part of the standard library or
>>>>>>> by extending PyTypeObject.
>>>>>>
>>>>>>
>>>>>> So I finally got around to implementing this:
>>>>>>
>>>>>> https://github.com/dagss/pyextensibletype
>>>>>>
>>>>>> Documentation now in a draft in the NumFOCUS SEP repo, which I believe 
>>>>>> is a
>>>>>> better place to store cross-project standards like this. (The NumPy
>>>>>> docstring standard will be SEP 100).
>>>>>>
>>>>>> https://github.com/numfocus/sep/blob/master/sep200.rst
>>>>>>
>>>>>> Summary:
>>>>>>
>>>>>>  - No common runtime dependency
>>>>>>
>>>>>>  - 1 ns overhead per lookup (that's for the custom slot *alone*, no
>>>>>> fast-callable signature matching or similar)
>>>>>>
>>>>>>  - Slight annoyance: Types that want to use the metaclass must be a
>>>>>> PyHeapExtensibleType, to make the binary layout work with how CPython 
>>>>>> makes
>>>>>> subclasses from Python scripts
>>>>>>
>>>>>> My conclusion: I think the metaclass approach should work really well.
>>>>>
>>>>> Few quick comments on skimming the code:
>>>>>
>>>>> The complicated nested #ifdef for __builtin_expect could be simplified to
>>>>>  #if defined(__GNUC__) && (__GNUC__ > 2 || __GNUC_MINOR__ > 95)
>>>>>
>>>>> PyCustomSlots_Check should be called PyCustomSlots_CheckExact, surely?
>>>>> And given that, how can this code work if someone does subclass this
>>>>> metaclass?
>>>>
>>>> I think we should provide a wrapper for PyType_Ready, which just
>>>> copies the pointer to the table and the count directly into the
>>>> subclass. If a user then wishes to add stuff, the user can allocate a
>>>> new memory region dynamically, memcpy the base class' stuff in there,
>>>> and append some entries.
>>>
>>> Maybe we should also allow each custom type to set a deallocator,
>>> since they are then heap types which can go out of scope. The
>>> metaclass can then call this deallocator to deallocate the table.
>>
>> Custom types are plain old Python objects, they can use tp_dealloc.
>>
> If I set etp_custom_slots to something allocated on the heap, then the
> (shared) metaclass would have to deallocate it. The tp_dealloc of the
> type itself would be called for its instances (which can be used to
> deallocate dynamically allocated memory in the objects if you use a
> custom slot "pointer offset").

Oh, I see. Right, the natural way to handle this would be have each
user define their own metaclass with the behavior they want. Another
argument for supporting multiple metaclasses simultaneously I guess...

- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [Python-Dev] C-level duck typing

[Nathaniel Smith]

On Mon, May 28, 2012 at 12:09 PM, mark florisson
 wrote:
> On 28 May 2012 12:01, Nathaniel Smith  wrote:
>> On Mon, May 28, 2012 at 11:55 AM, mark florisson
>>  wrote:
>>> On 28 May 2012 11:41, Nathaniel Smith  wrote:
>>>> On Mon, May 28, 2012 at 10:13 AM, mark florisson
>>>>  wrote:
>>>>> On 28 May 2012 09:54, mark florisson  wrote:
>>>>>> On 27 May 2012 23:12, Nathaniel Smith  wrote:
>>>>>>> On Sun, May 27, 2012 at 10:24 PM, Dag Sverre Seljebotn
>>>>>>>  wrote:
>>>>>>>> On 05/18/2012 10:30 AM, Dag Sverre Seljebotn wrote:
>>>>>>>>>
>>>>>>>>> On 05/18/2012 12:57 AM, Nick Coghlan wrote:
>>>>>>>>>>
>>>>>>>>>> I think the main things we'd be looking for would be:
>>>>>>>>>> - a clear explanation of why a new metaclass is considered too 
>>>>>>>>>> complex a
>>>>>>>>>> solution
>>>>>>>>>> - what the implications are for classes that have nothing to do with 
>>>>>>>>>> the
>>>>>>>>>> SciPy/NumPy ecosystem
>>>>>>>>>> - how subclassing would behave (both at the class and metaclass 
>>>>>>>>>> level)
>>>>>>>>>>
>>>>>>>>>> Yes, defining a new metaclass for fast signature exchange has its
>>>>>>>>>> challenges - but it means that *our* concerns about maintaining
>>>>>>>>>> consistent behaviour in the default object model and avoiding adverse
>>>>>>>>>> effects on code that doesn't need the new behaviour are addressed
>>>>>>>>>> automatically.
>>>>>>>>>>
>>>>>>>>>> Also, I'd consider a functioning reference implementation using a 
>>>>>>>>>> custom
>>>>>>>>>> metaclass a requirement before we considered modifying type anyway, 
>>>>>>>>>> so I
>>>>>>>>>> think that's the best thing to pursue next rather than a PEP. It also
>>>>>>>>>> has the virtue of letting you choose which Python versions to target 
>>>>>>>>>> and
>>>>>>>>>> iterating at a faster rate than CPython.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This seems right on target. I could make a utility code C header for
>>>>>>>>> such a metaclass, and then the different libraries can all include it
>>>>>>>>> and handshake on which implementation becomes the real one through
>>>>>>>>> sys.modules during module initialization. That way an eventual PEP 
>>>>>>>>> will
>>>>>>>>> only be a natural incremental step to make things more polished, 
>>>>>>>>> whether
>>>>>>>>> that happens by making such a metaclass part of the standard library 
>>>>>>>>> or
>>>>>>>>> by extending PyTypeObject.
>>>>>>>>
>>>>>>>>
>>>>>>>> So I finally got around to implementing this:
>>>>>>>>
>>>>>>>> https://github.com/dagss/pyextensibletype
>>>>>>>>
>>>>>>>> Documentation now in a draft in the NumFOCUS SEP repo, which I believe 
>>>>>>>> is a
>>>>>>>> better place to store cross-project standards like this. (The NumPy
>>>>>>>> docstring standard will be SEP 100).
>>>>>>>>
>>>>>>>> https://github.com/numfocus/sep/blob/master/sep200.rst
>>>>>>>>
>>>>>>>> Summary:
>>>>>>>>
>>>>>>>>  - No common runtime dependency
>>>>>>>>
>>>>>>>>  - 1 ns overhead per lookup (that's for the custom slot *alone*, no
>>>>>>>> fast-callable signature matching or similar)
>>>>>>>>
>>>>>>>>  - Slight annoyance: Types that want to use the metaclass must be a
>>>>>>>> PyHeapExtensibleType, to make the binary layout work with h

Re: [Cython] 2d buffer interface with aligned data

[Nathaniel Smith]

On Tue, Jul 17, 2012 at 9:55 PM, Christian Heimes  wrote:
> Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
>> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
>> your cdef class (don't know if it's documented but you can see example
>> in tests/run/buffer.pyx).
>
> The new buffer interface from PEP 3118 is only available for Python 2.6
> and newer. I was hoping for some abstraction layer in Cython. Well, I
> don't have to support Python 2.5 and older. Thanks for the hint!

If you're worried about supporting older Python versions then the
simplest thing is probably to just implement the __array_interface__
or __array_struct__ interface. They're pretty trivial, and while not
as standards-compliant and Correct as PEP3118, they'll work just fine
with numpy across any version of Python. (Of course you can also
implement both PEP3118 and __array_struct__ together.)

http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Fwd: [numpy] MAINT: silence Cython warnings about changes dtype/ufunc size. (#432)

[Nathaniel Smith]

Hi Cythoneers,

Ralf just proposed this pull request for numpy, to unconditionally install
a warning filter to silence all "numpy.dtype size changed", "numpy.ufunc
size changed" warnings that Cython likes to spit out. See the links below
for details. Figured you should have a chance to comment, anyway.

-n

-- Forwarded message --
From: Ralf Gommers 
Date: Sat, Sep 8, 2012 at 9:00 PM
Subject: [numpy] MAINT: silence Cython warnings about changes dtype/ufunc
size. (#432)
To: numpy/numpy 


These warnings are visible whenever you import scipy (or another package)
that
was compiled against an older numpy than is installed. For example compiled
against 1.5.1, like current scipy binaries are, and used with 1.7.0.

These warnings aren't useful; if numpy would really break its ABI it would
be
noticed in no time without these warnings.
--
You can merge this Pull Request by running:

  git pull https://github.com/rgommers/numpy silence-cython-warnings

Or view, comment on, or merge it at:

  https://github.com/numpy/numpy/pull/432
Commit Summary

   - MAINT: silence Cython warnings about changes dtype/ufunc size.

File Changes

   - *M* numpy/__init__.py (6)

Patch Links

   - https://github.com/numpy/numpy/pull/432.patch
   - https://github.com/numpy/numpy/pull/432.diff

 —
Reply to this email directly or view it on
GitHub.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Be more forgiving about memoryview strides

[Nathaniel Smith]

On Thu, Feb 28, 2013 at 5:50 PM, Robert Bradshaw  wrote:
> On Thu, Feb 28, 2013 at 7:13 AM, Sebastian Berg
>  wrote:
>> Hey,
>>
>> Maybe someone here already saw it (I don't have a track account, or I
>> would just create a ticket), but it would be nice if Cython was more
>> forgiving about contiguous requirements on strides. In the future this
>> would make it easier for numpy to go forward with changing the
>> contiguous flags to be more reasonable for its purpose, and second also
>> to allow old (and maybe for the moment remaining) corner cases in numpy
>> to slip past (as well as possibly the same for other programs...). An
>> example is (see also https://github.com/numpy/numpy/issues/2956 and the
>> PR linked there for more details):
>>
>> def add_one(array):
>> cdef double[::1] a = array
>> a[0] += 1.
>> return array
>>
>> giving:
>>
> add_one(np.ascontiguousarray(np.arange(10.)[::100]))
>> ValueError: Buffer and memoryview are not contiguous in the same
>> dimension.
>>
>> This could easily be changed if MemoryViews check the strides as "can be
>> interpreted as contiguous". That means that if shape[i] == 1, then
>> strides[i] are arbitrary (you can just change them if you like). This is
>> also the case for 0-sized arrays, which are arguably always contiguous,
>> no matter their strides are!
>
> I was under the impression that the primary value for contiguous is
> that it a foo[::1] can be interpreted as a foo*. Letting strides be
> arbitrary completely breaks this, right?

Nope. The natural definition of "C contiguous" is "the array entries
are arranged in memory in the same way they would be if they were a
multidimensional C array" (i.e., what you said.) But it turns out that
this is *not* the definition that numpy and cython use!

The issue is that the above definition is a constraint on the actual
locations of items in memory, i.e., given a shape, it tells you that
for every index,
 (a)  sum(index * strides) == sum(index * cumprod(shape[::-1])[::-1] * itemsize)
Obviously this equality holds if
 (b)  strides == cumprod(shape[::-1])[::-1] * itemsize
(Or for F-contiguity, we have
 (b')  strides == cumprod(shape) * itemsize
)

(a) is the natural definition of "C contiguous". (b) is the definition
of "C contiguous" used by numpy and cython. (b) implies (a). But (a)
does not imply (b), i.e., there are arrays that are C-contiguous which
numpy and cython think are discontiguous. (Also in numpy there are
some weird cases where numpy accidentally uses the correct definition,
I think, which is the point of Sebastian's example.)

In particular, if shape[i] == 1, then the value of stride[i] really
should be irrelevant to judging contiguity, because the only thing you
can do with strides[i] is multiply it by index[i], and if shape[i] ==
1 then index[i] is always 0. So an array of int8's with shape = (10,
1), strides = (1, 73) is contiguous according to (a), but not
according to (b). Also if shape[i] is 0 for any i, then the entire
contents of the strides array becomes irrelevant to judging
contiguity; all zero-sized arrays are contiguous according to (a), but
not (b).

(This is really annoying for numpy because given, say, a column vector
with shape (n, 1), it is impossible to be both C- and F-contiguous
according to the (b)-style definition. But people expect expect
various operations to preserve C versus F contiguity, so there are
heuristics in numpy that try to guess whether various result arrays
should pretend to be C- or F-contiguous, and we don't even have a
consistent idea of what it would mean for this code to be working
correctly, never mind test it and keep it working. OTOH if we just fix
numpy to use the (a) definition, then it turns out a bunch of
third-party code breaks, like, for example, cython.)

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Upcoming cython/numpy breakage with stride checking

[Nathaniel Smith]

Hi all,

If you build current numpy master with
  NPY_RELAXED_STRIDES_CHECKING=1 python setup.py install
then Cython code using ndarrays starts blowing up, e.g.:

# foo.pyx
def add_one(array):
cdef double[::1] a = array
a[0] += 1.
return array

>>> foo.add_one(np.ascontiguousarray(np.arange(10.)[::100]))
Traceback (most recent call last):
  File "", line 1, in 
  File "foo.pyx", line 2, in foo.add_one (foo.c:1210)
cdef double[::1] a = array
ValueError: Buffer and memoryview are not contiguous in the same dimension.

The problem (as discussed before) is that Cython has an unnecessarily
strict definition of "contiguous", so NPY_RELAXED_STRIDES_CHECKING=1
pretty much breaks all existing compiled Cython modules.

Our plan is to make NPY_RELAXED_STRIDES_CHECKING=1 into the default
sooner or later, and Cython is a major blocker on this plan. It may
become the default as soon as the 1.8 pre-releases (with the
expectation that we'll probably have to switch back again before the
actual release, but still).

References:

Previous thread:
  http://thread.gmane.org/gmane.comp.python.cython.devel/14634
Detailed discussion of the difference between numpy/cython's current
definition of "contiguity", and the correct definition:
  http://thread.gmane.org/gmane.comp.python.cython.devel/14634/focus=14640
The PR implementing NPY_RELAXED_STRIDES_CHECKING:
  https://github.com/numpy/numpy/pull/3162
Another test case:
  https://github.com/numpy/numpy/issues/2956

We're hoping that Cython will also switch soon to the more accurate
check for contiguity. This shouldn't cause any backwards compatibility
problems -- it just means Cython code would make strictly fewer
copies, and error out due to lack of contiguity strictly less often,
even with older numpys. And it seems like a necessary step for getting
this untangled and minimizing user pain. What do you think?

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Upcoming cython/numpy breakage with stride checking

[Nathaniel Smith]

On Mon, Apr 8, 2013 at 7:42 AM, Dag Sverre Seljebotn
 wrote:
> I guess you have changed your implementation of PEP 3118 too slightly on the 
> NumPy side? Is this undefined in the PEP or are you now not in strict 
> adherence to it?

I just checked, and PEP 3118 just says that if a {C,F,ANY}_CONTIGUOUS
buffer is requested, then that should be provided, and the strides
array should contain something valid (not be NULL). They don't define
any algorithm for matching {C,F,ANY}_CONTIGUOUS to particular strides.
My interpretation would be, we're satisfying this just fine so long as
our memory layout actually does match what was requested (i.e., the
strides are not needed).

Obviously Cython goes above and beyond in checking the strides -- if
you just *trusted* us then it would all work out ;-). If there's other
code that's similarly picky then it might be useful for us to take
some special effort of "canonicalize" the strides when exporting a PEP
3118 buffer, like Sebastian says, but eh. This is arguably just
papering over buggy code, and such code may or may not exist... (Or is
it only the PEP 3118 buffer strides that Cython actually checks? If
there's a simple hack we can do that will let us avoid forcing
everyone to rebuild their Cython modules then it will make this
transition significantly easier...)

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking)

[Nathaniel Smith]

On 9 Apr 2013 13:05, "mark florisson"  wrote:
> However, the memoryview class is duplicated in every cython module, which
means a memoryview object from another module will fail this check. This is
a general problem in Cython that could be worked around for memoryviews,
but in general the lack of a Cython runtime is a blessing for distribution
purposes and a wart for most other purposes.

This seems easy to fix - a Cython runtime would be a python module like
cython.runtime that Cython modules imported and used, right? You could
emulate this perfectly by just checking for such a module in sys.modules at
import time, and injecting a copy if none was found. (With some versioning
in the name so modules compiled with different versions of Cython could
coexist.) This would be much easier than coming up with elaborate object
model hacks, since it just uses existing standard python mechanisms...

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking)

[Nathaniel Smith]

On 9 Apr 2013 13:50, "Stefan Behnel"  wrote:
>
> Nathaniel Smith, 09.04.2013 14:25:
> > On 9 Apr 2013 13:05, "mark florisson" wrote:
> >> However, the memoryview class is duplicated in every cython module,
which
> >> means a memoryview object from another module will fail this check.
This is
> >> a general problem in Cython that could be worked around for
memoryviews,
> >> but in general the lack of a Cython runtime is a blessing for
distribution
> >> purposes and a wart for most other purposes.
> >
> > This seems easy to fix - a Cython runtime would be a python module like
> > cython.runtime that Cython modules imported and used, right? You could
> > emulate this perfectly by just checking for such a module in
sys.modules at
> > import time, and injecting a copy if none was found. (With some
versioning
> > in the name so modules compiled with different versions of Cython could
> > coexist.)> This would be much easier than coming up with elaborate
object
> > model hacks, since it just uses existing standard python mechanisms...
>
> "easy" may not be the optimal way of putting this - otherwise, it would
> have been done long ago, back when we decided that it should eventually
> work that way.

Of course. But by this definition there are only two kinds of code: already
written, and hard. Might be true, but it's easier to discuss things if we
have more gradations than that ;-)

> The main problems is that this would move code out of the module, and thus
> out of the module's control and out of the sight of the C compiler, so
we'd
> need to decide rather carefully what to externalise, based on criteria
like
> API stability, performance, actual doability, ...

Sure, but obviously the memoryview type object would be a good candidate,
from what Mark said. Inlineable functions would remain in each compilation
unit, no real point in trying to share those.

> There's also the problem of dependency hell and getting rid of old modules
> once they are no longer used on the user side. And also, how to get them
> there in the first place. Having one package overwrite the files of
another
> during its installation is just asking for trouble.

The system I described does not involve the addition of any new files to
any package.

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking)

[Nathaniel Smith]

On Tue, Apr 9, 2013 at 2:11 PM, Stefan Behnel  wrote:
> Nathaniel Smith, 09.04.2013 15:00:
>> On 9 Apr 2013 13:50, "Stefan Behnel" wrote:
>>> Nathaniel Smith, 09.04.2013 14:25:
>>> There's also the problem of dependency hell and getting rid of old modules
>>> once they are no longer used on the user side. And also, how to get them
>>> there in the first place. Having one package overwrite the files of
>>> another during its installation is just asking for trouble.
>>
>> The system I described does not involve the addition of any new files to
>> any package.
>
> I take it then that you were envisaging a separate "cython-runtime" package
> on PyPI that Cython compiled modules would have to depend on?

No, I must have been unclear -- I envisage that each cython-compiled
module continues to contain all the code it depends on (just like
now). But, the first cython-compiled module that is imported tweaks
sys.modules *as if* there were a separate cython-runtime package, and
sticks the necessary pieces (like the memoryview TypeObject) into this
module, and later modules import it from there.

There's no additional dependencies anywhere. You have to be careful to
make sure you version the runtime module appropriately so that you
don't end up with a module compiled with cython version X using the
runtime injected by a module compiled with the incompatible cython
version Y, but this is no harder than any other versioning problem.
Just make sure to change the runtime module name whenever you break
compatibility.

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking)

[Nathaniel Smith]

On Tue, Apr 9, 2013 at 3:15 PM, Stefan Behnel  wrote:
> Ok, got it now. That solves the distribution problem, assuming that all
> installed runtimes with a given version are equivalent. Basically, we'd
> move the code out and then cimport the stuff back that we need, which would
> then let the first import of a given runtime version insert it into
> sys.modules.

Yeah, that's the definition of "given version" :-), and any kind of
shared runtime does require versioning. If we wanted to be extra
careful we could put the shared code into its own block of boilerplate
that gets injected into each generated .c file, and then have the
"version" be the sha1 of that block of boilerplate... the trade-offs
depend on what exactly is getting shared.

> I so can't wait seeing the surprise in the eyes of our users when they let
> their IDE reorganise their import order and their code starts crashing. :]

That's easy to fix, just don't create such bugs ;-).

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Shared Cython runtime (was: Upcoming cython/numpy breakage with stride checking)

[Nathaniel Smith]

On Tue, Apr 9, 2013 at 3:32 PM, mark florisson
 wrote:
> On 9 April 2013 14:55, Nikita Nemkin  wrote:
>> One alternative for code reuse in large Cython projects
>> could be packaging multiple modules into one shared library.
>
> We have 'include'! :) Seriously though, that wouldn't work well with the
> import mechanism, and probably not for C compile time either.

You can link multiple .c files into a single shared library.

This is off-topic again, but I've often thought in the past it would
be nice if one could easily build a single module out of a combination
of multiple .c and .pyx files. Specifically it'd be nice to be able to
port bits of numpy/core/multiarray.so to Cython, but it's already
50,000 lines of C code spread out over 38 files and defining a single
module; no real way to move just part of it to Cython so far as I
know...

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CF based type inference

[Nathaniel Smith]

On Sun, Jun 2, 2013 at 7:22 AM, Stefan Behnel  wrote:
> mark florisson, 21.05.2013 15:32:
>> On 21 May 2013 14:14, Vitja Makarov wrote:
>>>
>>> def foo(int N):
>>> x = 1
>>> y = 0
>>> for i in range(N):
>>> x = x * 0.1 + y * 0.2
>>> y = x * 0.3 + y * 0.4
>>> print typeof(x), typeof(y)
>>>
>>> Here both x and y will be inferred as double
>>
>> Ok, so I assume it promotes the incoming types (all reaching
>> definitions)? If N == 0, then when using objects you get an int,
>> otherwise a double.
>
> I'm not sure what you mean here. I certainly don't think the inferred type
> of x and y should depend on the value of N. It should always be a double,
> because that's the spanning type for all paths. In the very unlikely case
> that that's not what the user wants, explicit typing will easily fix it for
> them.

But 'double' does not actually span 'int', floats and integers are
different in all kinds of corner cases. Both have values that are
unrepresentable in the other, etc. So this optimization as stated
is... not an optimization, it's just wrong.

I mean obviously in this example double would be fine, but how do you
decide when it's okay to randomly reinterpret users' code as meaning
something different than what they wrote, and when it isn't? It's not
that I think the Python rules here are particularly awesome, or that I
on purpose write code that sometimes returns ints and sometimes
doubles. But at least I know what the Python rules are, which means I
can always look at a chunk of code and figure out what the interpreter
will do. This is why people writing serious C compilers are so anal
about obscure problems like aliasing and guaranteeing that you get
segfaults at the right time, and generally insisting that
optimizations must *exactly* preserve semantics. I'm worried from this
discussion that in Cython, the rule for how variables are typed will
become "well, you get whatever types our type inference engine
guessed; dropping ordinary Python code into Cython might change the
outputs or might not; if you want to know 100% what your code will do
then your only option is to either put explicit types on every single
variable or else go read the source code for the inference engine in
the specific version of Cython you're using".

-n
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Declaration syntax change

[Nathaniel Smith]

On Thu, Oct 3, 2013 at 1:23 PM, Stefan Behnel  wrote:
> Greg Ewing, 03.10.2013 14:10:
>> Robert Bradshaw wrote:
>>> cdef int *a, b, c, *d[3]
>>>
>>> is IMHO quite ugly but also adds a lot of complexity to the parser.
>>> What if instead we required
>>>
>>> cdef int* a
>>> cdef int b, c
>>> cdef int[3]* d
>
> The last line looks ambiguous, BTW, hadn't even noticed it before. Is that
> an array of int pointers or a pointer to an array (pointer)? We should make
> sure the way this is declared is really obvious and not unexpected to C users.
>
>
>> What would be the benefit of this? You're proposing to change
>> from something identical to C declaration syntax, which is
>> second nature for a great many people, to something that
>> looks deceptively like C syntax but isn't.
>
> The reasoning is that the C syntax is error prone and less readable than it
> could be, because you have to spot stars in the right places of a
> potentially long list of variable names to know if something is a value or
> a pointer. If there was only one type declaration, right after the cdef, it
> would be much clearer. It would just say: "this is a list of variables
> declared as int*", not mixing any further types into it.
>
> Also, C is only second nature to some people. A great many people actually
> use Cython specifically to *avoid* having to write C.

The two halves of this email seem to sort of contradict each other,
don't you think? At least the C syntax has the advantage that it's
well-defined and many people *do* know it (and if they don't then
there are bazillions of references around, plus you can just copy it
out of header files if you're wrapping a C library), whereas as noted
above, in fact there are *no* people who know how to look at int[3]*
and be confident about what it means, even you...?

(I'm not against improving on C in general, but I'm far from convinced
that there's any syntax for encoding C types that's sufficiently
better than what C does to be worth the switching costs.)

If what really bothers you is having objects of different types
declared within the same statement then you could just litigate *that*
out of existence directly... not convinced this would be worthwhile
(though I do tend to use that style myself already), but it seems more
viable than trying to reinvent C's type syntax.

-n
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Declaration syntax change

[Nathaniel Smith]

On Thu, Oct 3, 2013 at 3:00 PM, Stefan Behnel  wrote:
> Nathaniel Smith, 03.10.2013 14:35:
>> On Thu, Oct 3, 2013 at 1:23 PM, Stefan Behnel wrote:
>>> Greg Ewing, 03.10.2013 14:10:
>>>> Robert Bradshaw wrote:
>>>>> cdef int *a, b, c, *d[3]
>>>>>
>>>>> is IMHO quite ugly but also adds a lot of complexity to the parser.
>>>>> What if instead we required
>>>>>
>>>>> cdef int* a
>>>>> cdef int b, c
>>>>> cdef int[3]* d
>>>
>>> The last line looks ambiguous, BTW, hadn't even noticed it before. Is that
>>> an array of int pointers or a pointer to an array (pointer)? We should make
>>> sure the way this is declared is really obvious and not unexpected to C 
>>> users.
>> [...]
>> The two halves of this email seem to sort of contradict each other,
>> don't you think? At least the C syntax has the advantage that it's
>> well-defined and many people *do* know it (and if they don't then
>> there are bazillions of references around, plus you can just copy it
>> out of header files if you're wrapping a C library), whereas as noted
>> above, in fact there are *no* people who know how to look at int[3]*
>> and be confident about what it means, even you...?
>
> Well, it's still better than looking at "*d[3]", now, isn't it? Maybe I'm
> just confused (by both, actually) because I'm not really breathing C.

Yeah, personally in either case I'd have to look it up (and it's
simply impossible that you're going to make it as easy to lookup this
funky Cython-specific syntax as it is to look up standard C syntax).
But also, the reason I don't know the C version already is that I've
probably never seen such a declaration in real life, which makes it
hard to see why this is a really pressing problem. I don't really come
to Cython because I want idiosyncratic tweaks to things C already does
perfectly well, you know? I come to Cython because I want a nice way
to get Python and C to talk to each other, so sticking to familiar
Python and C things is rather nice...

-n
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Declaration syntax change

[Nathaniel Smith]

On Thu, Oct 3, 2013 at 5:03 PM, Robert Bradshaw  wrote:
> On Thu, Oct 3, 2013 at 7:13 AM, Nathaniel Smith  wrote:
>> On Thu, Oct 3, 2013 at 3:00 PM, Stefan Behnel  wrote:
>>> Nathaniel Smith, 03.10.2013 14:35:
>>>> On Thu, Oct 3, 2013 at 1:23 PM, Stefan Behnel wrote:
>>>>> Greg Ewing, 03.10.2013 14:10:
>>>>>> Robert Bradshaw wrote:
>>>>>>> cdef int *a, b, c, *d[3]
>>>>>>>
>>>>>>> is IMHO quite ugly but also adds a lot of complexity to the parser.
>>>>>>> What if instead we required
>>>>>>>
>>>>>>> cdef int* a
>>>>>>> cdef int b, c
>>>>>>> cdef int[3]* d
>>>>>
>>>>> The last line looks ambiguous, BTW, hadn't even noticed it before. Is that
>>>>> an array of int pointers or a pointer to an array (pointer)? We should 
>>>>> make
>>>>> sure the way this is declared is really obvious and not unexpected to C 
>>>>> users.
>>>> [...]
>>>> The two halves of this email seem to sort of contradict each other,
>>>> don't you think? At least the C syntax has the advantage that it's
>>>> well-defined and many people *do* know it (and if they don't then
>>>> there are bazillions of references around, plus you can just copy it
>>>> out of header files if you're wrapping a C library), whereas as noted
>>>> above, in fact there are *no* people who know how to look at int[3]*
>>>> and be confident about what it means, even you...?
>>>
>>> Well, it's still better than looking at "*d[3]", now, isn't it? Maybe I'm
>>> just confused (by both, actually) because I'm not really breathing C.
>>
>> Yeah, personally in either case I'd have to look it up (and it's
>> simply impossible that you're going to make it as easy to lookup this
>> funky Cython-specific syntax as it is to look up standard C syntax).
>> But also, the reason I don't know the C version already is that I've
>> probably never seen such a declaration in real life, which makes it
>> hard to see why this is a really pressing problem.
>
> Cause or effect :).
>
>> I don't really come
>> to Cython because I want idiosyncratic tweaks to things C already does
>> perfectly well, you know?
>
> I wouldn't classify this as something that "C already does perfectly
> well." The fact that people commonly write
>
> int* ptr;
>
> rather than
>
> int *ptr;
>
> means that it's parsed differently in people's heads than the grammar,
> and though it's hard to miss given the context of this thread I've
> seen people gloss right over things like
>
> char* a, b;
> a = malloc(n);
> b = malloc(n);
> strcpy(b, a);
>
> which, yes, is perfectly valid C (though any sane compiler will throw
> out a warning). It should also be noted that an increasing percentage
> of Cython users don't know C at all.

So, like I said upthread, if this is the problem you're really trying
to solve, just tackle it directly by making 'char *a, b' an error of
some kind.

> The rule would be very simple--type decorations would be left
> associative, so an int*[5]** would be a pointer to a pointer to an
> array of pointers to ints.

That's a simple rule, but not necessarily a memorable one -- it's
exactly the opposite of how both English and C work. English: you'd
say "array of pointers" but write pointer-array *[]. C: *const * is a
const-pointer-to-pointer, but here it would be a
pointer-to-const-pointer.

In real life of course I'd parenthesize an expression like this to
make it unambiguous even for readers who don't remember the
precedence/associativity rules, but I guess in this notation you can't
parenthesize things, readers will have to memorize/look-up the
associativity regardless.

> Now, you're right that this doesn't come up
> often, which is why it'll be easy to change, but it does complicate
> the compiler (and hypothetical grammar). Ideally people shouldn't be
> copying C headers in the long run, they should be parsed automatically
> or by wrappers like xdress.

But we don't live in the long run. Maybe that will happen eventually,
maybe it won't -- it's hard to make predictions, especially about the
future...

-n
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] "relaxed_strides" test broken with NumPy 1.8

[Nathaniel Smith]

On 4 Jan 2014 11:53, "Stefan Behnel"  wrote:
>
> mark florisson, 03.01.2014 23:28:
> > On 3 January 2014 18:22, Stefan Behnel wrote:
> >> I enabled the NumPy build for our Py3.3 test runs and while I was at
it, I
> >> got it to use the latest NumPy release 1.8. This made one of the tests
fail:
> >>
> >> """
> >> Traceback (most recent call last):
> >>   File ".../doctest.py", line 1313, in __run
> >> compileflags, 1), test.globs)
> >>   File " >> 29)[3]>", line 1, in 
> >> test_one_sized(a)[0]
> >>   File "relaxed_strides.pyx", line 38, in
> >> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
> >>   File "stringsource", line 622, in
View.MemoryView.memoryview_cwrapper
> >> (relaxed_strides.cpp:7568)
> >>   File "stringsource", line 327, in
> >> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
> >>
> >> ValueError: ndarray is not C-contiguous
> >> """
> >>
> >>
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
> >>
> >> According to the comments in the test file and the corresponding NumPy
pull
> >> request, this seems to be somewhat expected.
> >>
> >>
https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
> >>
> >> https://github.com/numpy/numpy/pull/3162
> >>
> >> Does someone know enough about this to figure out what to do?
> >
> > It seems to come from the call to __Pyx_GetBuffer, which is
> > PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
> > for an extent of 1, but instead only checking the stride (which is a
> > multiple of the itemsize)?
>
> No, Py3 doesn't do any validation here, it does a straight call into the
> object's slot function, i.e. into the NumPy array itself.
>
> So, the question is: who's wrong here? The test or NumPy?
>
> Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
> environment variable for the NumPy build. Let's try that first.
>
> http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking

It's probably nicer though to write the Cython tests in such a way that
they can pass against a default numpy installation? To avoid user confusion
and all that.

If those tests depend on numpy having relaxed strides enabled, then I'd
suggest checking for this a test time and skipping the tests if not found.
The relaxed strides docs give a recipe for determining how numpy was built.

-n
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] "relaxed_strides" test broken with NumPy 1.8

[Nathaniel Smith]

On 4 Jan 2014 17:07, "Robert Bradshaw"  wrote:
>
> Yes, that'd be good. Do you know how? I won't have time to look at this
'till next week.

Something like

if not np.ones((10, 1), order="C").flags.f_contiguous:
  # numpy without relaxed stride support
  raise SkipTest

> On Jan 4, 2014 9:54 AM, "Nathaniel Smith"  wrote:
>>
>> On 4 Jan 2014 11:53, "Stefan Behnel"  wrote:
>> >
>> > mark florisson, 03.01.2014 23:28:
>> > > On 3 January 2014 18:22, Stefan Behnel wrote:
>> > >> I enabled the NumPy build for our Py3.3 test runs and while I was
at it, I
>> > >> got it to use the latest NumPy release 1.8. This made one of the
tests fail:
>> > >>
>> > >> """
>> > >> Traceback (most recent call last):
>> > >>   File ".../doctest.py", line 1313, in __run
>> > >> compileflags, 1), test.globs)
>> > >>   File "> > >> 29)[3]>", line 1, in 
>> > >> test_one_sized(a)[0]
>> > >>   File "relaxed_strides.pyx", line 38, in
>> > >> relaxed_strides.test_one_sized (relaxed_strides.cpp:1414)
>> > >>   File "stringsource", line 622, in
View.MemoryView.memoryview_cwrapper
>> > >> (relaxed_strides.cpp:7568)
>> > >>   File "stringsource", line 327, in
>> > >> View.MemoryView.memoryview.__cinit__ (relaxed_strides.cpp:3717)
>> > >>
>> > >> ValueError: ndarray is not C-contiguous
>> > >> """
>> > >>
>> > >>
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/1787/ARCH=m64,BACKEND=cpp,PYVERSION=py33m/console
>> > >>
>> > >> According to the comments in the test file and the corresponding
NumPy pull
>> > >> request, this seems to be somewhat expected.
>> > >>
>> > >>
https://github.com/cython/cython/blob/master/tests/memoryview/relaxed_strides.pyx
>> > >>
>> > >> https://github.com/numpy/numpy/pull/3162
>> > >>
>> > >> Does someone know enough about this to figure out what to do?
>> > >
>> > > It seems to come from the call to __Pyx_GetBuffer, which is
>> > > PyObject_GetBuffer in python 3. Maybe this is Python 3 not checking
>> > > for an extent of 1, but instead only checking the stride (which is a
>> > > multiple of the itemsize)?
>> >
>> > No, Py3 doesn't do any validation here, it does a straight call into
the
>> > object's slot function, i.e. into the NumPy array itself.
>> >
>> > So, the question is: who's wrong here? The test or NumPy?
>> >
>> > Hmm, or maybe just me. I didn't define the NPY_RELAXED_STRIDES_CHECKING
>> > environment variable for the NumPy build. Let's try that first.
>> >
>> >
http://docs.scipy.org/doc/numpy/release.html#npy-relaxed-strides-checking
>>
>> It's probably nicer though to write the Cython tests in such a way that
they can pass against a default numpy installation? To avoid user confusion
and all that.
>>
>> If those tests depend on numpy having relaxed strides enabled, then I'd
suggest checking for this a test time and skipping the tests if not found.
The relaxed strides docs give a recipe for determining how numpy was built.
>>
>> -n
>>
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] "relaxed_strides" test broken with NumPy 1.8

[Nathaniel Smith]

No, I'm missing something now; AFAIK there are only two numpy behaviors:
with relaxed strides and without relaxed strides, and version number should
be irrelevant beyond that. What's different between
1.8-without-relaxed-strides and 1.7 that makes the test break?
On 4 Jan 2014 22:01, "Stefan Behnel"  wrote:

> Stefan Behnel, 04.01.2014 22:51:
> > Stefan Behnel, 04.01.2014 22:47:
> >> Nathaniel Smith, 04.01.2014 18:36:
> >>> On 4 Jan 2014 17:07, "Robert Bradshaw" wrote:
> >>>> Yes, that'd be good. Do you know how? I won't have time to look at
> this
> >>> 'till next week.
> >>>
> >>> Something like
> >>>
> >>> if not np.ones((10, 1), order="C").flags.f_contiguous:
> >>>   # numpy without relaxed stride support
> >>>   raise SkipTest
> >>
> >>
> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
> >
> > Hmm, but this doesn't seem to work for me in older NumPy versions,
> although
> > the original test used to work there. Should we explicitly test for NumPy
> > 0.18+ as well?
>
> (Looks like the Cython versioning scheme starts messing up my sense for
> good versioning...)
>
> I meant NumPy 1.8, 'obviously', i.e. this:
>
>
> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
>
> Stefan
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] "relaxed_strides" test broken with NumPy 1.8

[Nathaniel Smith]

On Sun, Jan 5, 2014 at 8:03 AM, Stefan Behnel  wrote:
> Nathaniel Smith, 05.01.2014 02:07:
>> On 4 Jan 2014 22:01, "Stefan Behnel" wrote:
>>> Stefan Behnel, 04.01.2014 22:51:
>>>> Stefan Behnel, 04.01.2014 22:47:
>>>>> Nathaniel Smith, 04.01.2014 18:36:
>>>>>>
>>>>>> if not np.ones((10, 1), order="C").flags.f_contiguous:
>>>>>>   # numpy without relaxed stride support
>>>>>>   raise SkipTest
>>>>>
>>>>> https://github.com/cython/cython/commit/e1982505564125714d2010391eecfb8de61626fa
>>>>
>>>> Hmm, but this doesn't seem to work for me in older NumPy versions, although
>>>> the original test used to work there. Should we explicitly test for NumPy
>>>> 1.8+ as well?
>>>
>>> https://github.com/cython/cython/commit/a95d8f912c995300a13fc244ee71bc277668cb9a
>>
>> No, I'm missing something now; AFAIK there are only two numpy behaviors:
>> with relaxed strides and without relaxed strides, and version number should
>> be irrelevant beyond that. What's different between
>> 1.8-without-relaxed-strides and 1.7 that makes the test break?
>
> Mark would certainly know better than me.
>
> In any case, the test works with both NumPy 1.7 (tested with Py 2.x and
> 3.[12] on Jenkins) and NumPy 1.8 with relaxed strides support, but not with
> NumPy 1.8 without relaxed strides. The last two were tested in Py3.3 only,
> in case that matters. I also tested it locally now (in 3.3) and your
> snippet successfully distinguishes the two builds for me, but the test
> starts to fail when I disable relaxed strides in NumPy and works when it's
> enabled.

Then either there's a bug in numpy or in cython -- 1.7 and
1.8-without-relaxed-strides are supposed to be compatible, so if your
test works on one and not on the other, that's a bug in the libraries
somewhere, not in the test suite. Adding version checks will just hide
this bug, not fix anything.

(Personally I'd double-check whether the cython memoryview code is
doing its own version check, and wrongly assuming that if numpy
version >= 1.8, then relaxed strides = True...)

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] License information on each individual files

[Nathaniel Smith]

On Fri, Jul 18, 2014 at 10:13 PM, Sturla Molden  wrote:
> Benjamin Lerman  wrote:
>
>>  Would cython accept to add such a copyright header on its files?
>
> You want to display the Apache licence in every single file, even those
> with utility C code?

It's annoying, but a pretty standard request. Debian once made me do
this for some package or another (forget which) before they would
distribute it, though obviously that rule is inconsistently applied.
Still, it's pretty trivial and has real-world consequences, so why
reject a patch like this?

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] License information on each individual files

[Nathaniel Smith]

On Fri, Jul 18, 2014 at 10:53 PM, Robert Bradshaw 
wrote:

> On Fri, Jul 18, 2014 at 2:28 PM, Nathaniel Smith  wrote:
>
>> On Fri, Jul 18, 2014 at 10:13 PM, Sturla Molden 
>> wrote:
>> > Benjamin Lerman  wrote:
>> >
>> >>  Would cython accept to add such a copyright header on its files?
>> >
>> > You want to display the Apache licence in every single file, even those
>> > with utility C code?
>>
>> It's annoying, but a pretty standard request. Debian once made me do
>> this for some package or another (forget which) before they would
>> distribute it, though obviously that rule is inconsistently applied.
>> Still, it's pretty trivial and has real-world consequences, so why
>> reject a patch like this?
>
>
> Debian currently distributes Cython without these headers.
>

Yes, they're inconsistent, as I noted.


>
> I am curious why a licence in a top level directory that explicitly states
> it applies to everything in that directory is not sufficiently clear. What
> about auto-generated files? What about binary blobs? All 1000+ test files?
>
> http://www.apache.org/dev/apply-license.html#copy-per-file
>

That link is about sticking a copy of the apache license text in every
file, this is about having a few line header saying "This is part of
Cython, copyright Cython devs, released under Apache-2, see LICENSE.txt for
details".

I'm just saying, you can argue about how chromium's lawyers ought to think,
or you can just say "whatever, patches welcome" and merge it if it arrives.
Your time is yours to spend as you wish :-) I was also very annoyed
initially when asked to do this, so thought I'd share my experience in case
it was useful.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] License information on each individual files

[Nathaniel Smith]

On Sat, Jul 19, 2014 at 7:39 AM, Stefan Behnel  wrote:
>
> Sturla Molden, 19.07.2014 00:36:
> > Robert Bradshaw wrote:
> >> It's not just the initial patch; I'm primarily worried about the
> >> maintenance burden
> >
> > And also, will it break utility code and binary blobs? It might not even be
> > safe to put it in every file.
> >
> > And when put in files with utility C code, will it be included in the
> > generated .c file and taint this file with a notification about Cython's
> > Apache license?
>
> Well, on the technical side, stuff at the beginning of a utility code file
> should just be ignored, up to the first header line (starting with a
> sequence of at least 5 comment characters). Some of them already contain
> leading comments anyway.
>
> On the legal side, the licensing state of the generated code does not
> change by copying the license description from the global license file into
> each code file, because the global license already applies to the complete
> code base anyway (unless stated otherwise).
>
> However, what *is* the licensing state of the generated code? Strictly
> speaking, the code generated by Cython, all parts of which are embedded in
> Cython's own source code, could be considered a "derivative work" by some.

I suspect that "some" includes most lawyers and judges ;-). There's
tons of copyrightable code copied into Cython output.

[...]
> From my side, that statement definitely still applies. I really don't want
> to have anything to say on what users do with the code generated from their
> own source code. I do not even consider the generated code a "derivative
> work" myself, but IANAL...
>
> Anyway, as far as I understand it, the worst case is that people who ship
> Cython generated code have to add a copy of the Apache License to their
> package as well as any attribution notices we ship. That's annoying, but
> not really the end of the world.

No, the worst case is that GPL2 projects can't legally use code
written in Cython (perhaps even indirectly via other projects) :-(.
This might be okay because of the language in the GPL that's designed
to make it possible to write e.g. GPL'ed Win32 programs, but I'm not
at all sure.

> Also, utility code files could be exempt from the license explicitly by
> stating so (although they'd then need another license to allow for safe
> contributions to them).

There's substantial prior art for this kind of thing, see e.g. the GCC
runtime exception, the Classpath exception, etc.

You might want to send an email to h...@softwarefreedom.org describing
the situation and asking for advice. (See
https://www.softwarefreedom.org/about/contact/)

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython sometimes fails to build on Travis

[Nathaniel Smith]

On Fri, Nov 7, 2014 at 5:13 PM, Ondrej Certik  wrote:
> Hi,
>
> Cython sometimes fails to build on Travis for various projects. If it
> fails, it always ends with this error:
>
> https://gist.github.com/certik/08f16dd572170c17d956

I don't have any insight into the error, but for CI purposes I install
cython with:

pip install --install-option="--no-cython-compile" cython

which skips the compilation of cython itself (i.e. your cython code is
still translated from .pyx to .c, but the cython compiler that does
this translation runs as a pure python program). This is ridiculously
faster than a normal install -- skipping compiling cython takes
minutes off the build, which is far far far more than one loses by
making cythonization faster.

I mention it here because it seems like aside from being a good idea
in general, it might also serve as a workaround for whatever this
problem is.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] Cython inserting unqualified module name into sys.module on python 3?

[Nathaniel Smith]

Hi all,

Can anyone shed any light on this?

https://github.com/numpy/numpy/issues/5680

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython inserting unqualified module name into sys.module on python 3?

[Nathaniel Smith]

On Mar 14, 2015 2:03 PM, "Robert Bradshaw"  wrote:
>
> That is strange, looks like it was an attempt to support relative imports?
>
> https://github.com/cython/cython/blob/384cc660f5c7958524b8839ba24099fdbc6eaffd/Cython/Compiler/ModuleNode.py#L2271

Ah, I see.

I don't see how this could affect relative imports, because if foo.bar
does 'import .baz', this doesn't actually trigger any access to
sys.modules["foo.bar"]. (Exception: if you have foo/__init__.pyx. Is
that actually supported?) The critical thing for relative imports is
having correct __name__ and/or __package__ module-level global
variables, and AFAICT cython is not currently doing anything to set
these up. But it probably should, because relative imports are a
thing.

OTOH, putting the module into sys.modules *is* crucial to handle
recursive imports, i.e. where foo.pyx's module init function imports
bar.py, and bar.py imports foo. For regular python modules or for
python 2 extension modules, this works because even while foo is still
initializing, you can already get its (incomplete) module object from
sys.modules; for python 3 extension modules this won't work unless we
do it by hand. So the code that Cython is generating seems to be
correct and necessary, it just has the wrong idea about what the
fully-qualified module name is, and this breaks things.

So I'm convinced that Cython has to know the fully-qualified module
name for correct operation, and if it's wrong then weird real bugs
will happen. Next question: how am I supposed to make this work? Maybe
I'm just missing it, but I can't find anything in the docs about how I
should tell cython that mtrand.pyx is really numpy.random.mtrand...?

-n

> On Sat, Mar 14, 2015 at 1:17 AM, Nathaniel Smith  wrote:
> > Hi all,
> >
> > Can anyone shed any light on this?
> >
> > https://github.com/numpy/numpy/issues/5680
> >
> > -n
> >
> > --
> > Nathaniel J. Smith -- http://vorpus.org
> > ___
> > cython-devel mailing list
> > cython-devel@python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] Is it easy to do an "AST grep" of Cython code?

[Nathaniel Smith]

I have a few hundred files worth of Cython code, and I'd like to find
all instances where a variable/expression has a certain cdef class
type. (More specifically, I'd like to find all accesses to the cdef
fields of a particular cdef class, but even just finding all 'cdef
MYTYPE x" and "x" statements would probably get me pretty
close.) Is there any easy way to do this?

(The files are "every cython file on github or searchcode.com that
mentions the word "ufunc"", and I'm trying to find code that does
direct field access to the internals of the PyUFuncObject struct.)

-- 
Nathaniel J. Smith -- http://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] [ANN] Python compilers workshop at SciPy this year

[Nathaniel Smith]

Hi all,

I wanted to announce a workshop I'm organizing at SciPy this year, and
invite you to attend!

What: A two-day workshop bringing together folks working on JIT/AOT
compilation in Python.

When/where: July 11-12, in Austin, Texas.

(This is co-located with SciPy 2016, at the same time as the tutorial
sessions, just before the conference proper.)

Website: https://python-compilers-workshop.github.io/

Note that I anticipate that we'll be able to get sponsorship funding
to cover travel costs for folks who can't get their employers to foot
the bill.

Cheers,
-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Manylinux wheels for Cython

[Nathaniel Smith]

On Mon, Mar 28, 2016 at 4:04 PM, Emmanuel Gil Peyrot
 wrote:
> On Mon, Mar 28, 2016 at 10:54:21AM -0700, Matthew Brett wrote:
> […]
>> I uploaded manylinux wheels for Cython 0.23.5.
>>
>> If you're on Linux, and you upgrade pip to 8.1.1 (current) you should
>> now get Cython via a manylinux wheel by default.
>>
>> Please do test and let me know of any problems.
>
> Thanks, this significantly improves the install-time required for
> poezio for people who don’t already have cython installed.
>
> Do you plan on publishing AArch64 and ARMv7h wheels as well?  Those are
> the systems most impacted by the compile time of cython.

PEP 513 only standardizes linux wheels for x86-{32,64}, because its
strategy is to say "hey, in practice everyone is careful to maintain
backwards ABI compatibility with RHEL-cough-I-mean-CentOS 5, so let's
use that as our standard ABI". But CentOS 5 is only available for
x86-{32,64}.

It would be great if there's some way to standardize an ABI that lets
us distribute ARM wheels -- I just don't know enough about ARM to have
any idea how to do that :-). One possibility would be to standardize a
way to distribute wheels that target particular distributions, so you
could have e.g. a wheel for Raspbian specifically -- Nate Coraor was
interested in working on this. If you send an email to distutils-sig
volunteering to help then you might get some takers :-). Alternatively
maybe there is some minimum baseline ABI that most/all ARM
distributions actually do agree on in practice, similar to CentOS 5
for x86 variants... that could also be used to standardize something.
But either way it would need someone with the appropriate expertise
(or willingness to fake it) to step up and make a proposal on
distutils-sig.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Re: Cython 0.23.5 released

[Nathaniel Smith]

On Mar 31, 2016 6:40 AM, "Stefan Behnel"  wrote:
>
> Jason Madden schrieb am 31.03.2016 um 15:13:
> > On Thursday, 31 March 2016 00:50:57 UTC-5, Matthew Brett wrote:
> >>
> >> On Wed, Mar 30, 2016 at 2:35 PM, Forest Gregg  >> > wrote:
> >>> Thanks!
> >>>
> >>> Are there also plans to post Windows wheels for cython 0.23.5 to pypi
> >> like
> >>> there was for 0.23.4?
> >>
> >> I did that earlier today, using an Appveyor build system :
> >>
> >> https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml
> >> https://ci.appveyor.com/project/matthew-brett/cython-wheels
> >> http://win-wheels.scikit-image.org/
> >>
> >> Do they work for you?
> >
> > They work for building gevent on Appveyor. The cython binary wheels cut
> > gevent's total build time by more than half (they save almost 10
minutes).
>
> That makes me wonder if we shouldn't just provide pure Python wheels of
> Cython, i.e. without compiling any binary extensions in them. For one-time
> installs, it would only be marginally slower than with a compiled Cython,
> but it would work on all platforms with all Python versions.
>
> I guess the drawback would be that it would be more hassle for people who
> really want to install Cython permanently for themselves...
>
> Is there a way to tag a wheel somehow as "special"? I doubt that
> Nick&friends thought about the special Cython installation use case when
> designing the format...

The general case is "library with optional native acceleration". I think
one thing is that if there's both a generic and a specific wheel then pip
will prefer the specific one? So an arch-specific wheel will get higher
priority than a py2.py3-none-any wheel. Of course this only works if both
are available -- it's more of a fallback thing.

-n
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Re: Cython 0.23.5 released

[Nathaniel Smith]

Pip ignores trove classifiers entirely. You could overload PEP 440
prerelease versions, which pip ignores by default, but I'm not sure I'd
advise it, since those are rather useful for distributing prereleases :-).

I think I missed something in the initial motivation here: if there's a
precompiled binary wheel available for some platform, then that's what CI
servers should use, right? So the goal here is to provide some convenient
way for folks on unusual platforms without wheels -- like fbsd or arm Linux
-- to get the behavior that "pip install Cython" does the lengthy
build-from-source thing, but on CI servers they can instead run "pip
install [something]" to get a quick install? And what's wrong with the
current "pip install Cython --install-option="--no-cython-compile"?
On Mar 31, 2016 11:29 AM, "Stefan Behnel"  wrote:

> Nathaniel Smith schrieb am 31.03.2016 um 17:42:
> > On Mar 31, 2016 6:40 AM, "Stefan Behnel" wrote:
> >> That makes me wonder if we shouldn't just provide pure Python wheels of
> >> Cython, i.e. without compiling any binary extensions in them. For
> one-time
> >> installs, it would only be marginally slower than with a compiled
> Cython,
> >> but it would work on all platforms with all Python versions.
> >>
> >> I guess the drawback would be that it would be more hassle for people
> who
> >> really want to install Cython permanently for themselves...
> >>
> >> Is there a way to tag a wheel somehow as "special"? I doubt that
> >> Nick&friends thought about the special Cython installation use case when
> >> designing the format...
> >
> > The general case is "library with optional native acceleration". I think
> > one thing is that if there's both a generic and a specific wheel then pip
> > will prefer the specific one? So an arch-specific wheel will get higher
> > priority than a py2.py3-none-any wheel. Of course this only works if both
> > are available -- it's more of a fallback thing.
>
> We could provide a separate release version for the wheel, e.g. 0.24 for
> normal sdist and binary builds and 0.24py for the unaccelerated pure Python
> wheel. People could then "pip install Cython==0.24py" on their CI servers
> and get a fast wheel installation.
>
> The wheel is really easy to build:
>
> pip install wheel
> python setup.py bdist_wheel --universal --no-cython-compile
>
> This quickly creates a file
>
> dist/Cython-0.24b0-py2.py3-none-any.whl
>
> Py3.2 wouldn't be supported, but that's it.
>
> Since Cython shouldn't normally show up in any requirements.txt files
> itself, and thus would usually be installed separately in CI builds, this
> scheme should be easy to apply for users.
>
> Using different versions is better than having different package names
> because the later would quickly lead to confusion when both packages get
> installed by accident and try to unpack the same files. We need to take
> care that PyPI's version sorting doesn't get confused by this, though. "pip
> install Cython" should still install the latest *accelerated* version.
>
> Does anyone know if the trove classifier
>
> Development Status :: 7 - Inactive
>
> has any effect on pip? Or should we mark the wheel package as "3 - Alpha"
> release? That should prevent pip from installing it by default (IIRC).
>
> Stefan
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fwd: Question about how best require compiler options for C sources

[Nathaniel Smith]

Can you give a tiny concrete example? My questions are basic enough that I
feel like I'm missing something fundamental :-)

My first question is why you even need this, since AFAIK there are no cases
where it is correct to have a cython module dllexporting symbols that
appear in header files. This is what cimport is for, right?

My second question is why you would want to do this via the command line,
when compiling the dll means that you are compiling some cython-generated
.c, which means that you can put the #define directly in the source code,
no?

-n
On Apr 8, 2016 5:35 AM, "Erik Bray"  wrote:

> Hi all,
>
> I'd like to call attention to an issue I've been looking into for the
> past couple days:
>
> https://github.com/cython/cython/pull/360
>
> To summarize the discussion, when building DLLs for Windows, functions
> that should be exported by that DLL must be marked
> __declspec(dllexport) in their declaration.  However, when using the
> same header file in a project that links to that DLL the same function
> must be declared __declspec(dllimport).
>
> It's common practice to have a macro for this, whose value is
> controlled by whether or not a macro (often called something like
> "DLL_EXPORT").  When compiling the DLL we would define -DDLL_EXPORT to
> output __declspec(dllexport).  Otherwise it outputs
> __declspec(dllimport).
>
> Cython currently handles this with such a macro called DL_IMPORT which
> comes from Python. However, this macro was deprecated some time ago,
> and is removed in current Python 3 versions.  So Cython must replace
> it with its own.
>
> I'm working on a patch for this--to reduce confusion the macro is
> named specifically for the Cython module it's associated with.  For
> example, for a Cython module named "foo.bar" there are two macros
> DLL_EXPORT__foo__bar and EXPORT__foo__bar.  If the latter is defined
> then the former outputs dllexport, otherwise it outputs dllimport.
> I've attached the patch in progress.
>
> I'm open to comment on this, but where I'm stuck now is that in order
> for the "foo.bar" module to be compiled correctly it needs
> EXPORT__foo__bar to be defined at compile time.  And it's not clear to
> me what the best way is for the Cython compiler to pass down options
> that are passed to the C/C++ compiler.  In general the best way would
> be to attach this to the define_macros attribute of the associated
> distutils/setuptools Extension object and let the distutils compiler
> class generate the right compiler options.  But it's not clear to me
> from Cython's internals where the best place to do that would be.
>
> Thanks,
> Erik
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fwd: Question about how best require compiler options for C sources

[Nathaniel Smith]

On Apr 11, 2016 04:18, "Erik Bray"  wrote:
>
> On Fri, Apr 8, 2016 at 5:49 PM, Nathaniel Smith  wrote:
> > Can you give a tiny concrete example? My questions are basic enough
that I
> > feel like I'm missing something fundamental :-)
>
> Yes, I think you might be missing something, but I'm not sure exactly
> where.  In the issue I provided a tiny example which you can see here:
>
> https://gist.github.com/embray/12a67edb82b213217e31f408007898e6
>
> The C code generated by this example currently does not compile on
> Windows, because of how Cython uses DL_IMPORT incorrectly.  Regardless
> of what it *does* do, Cython should not be using the DL_IMPORT macro
> at all actually since that no longer even exists in Python.

Sure.

For that example, the correct thing to do is to *not* export the function.

Backing up, to make sure we're on the same page: There are three levels of
symbol visibility in C: file-internal, shared library internal (different
.c files that make up the library can see it, but users of the shared
library can't see it), and shared library exported (everyone can see it;
can also carry other consequences, e.g. on Linux then internal calls will
become noticeably slower, and it becomes possible for weird symbol
interposition issues to occur). So the rule of thumb is to make everything
as private as you can get away with.

Making this more interesting:
- vanilla C only includes 2 ways to mark symbol visibility, which is not
enough to make a 3-way distinction. Hence the need for an extra attribute
thingummy.
- everyone agrees that symbols marked 'static' should be file-internal, but
different platforms disagree about what should happen if the extra
attribute thingummy is missing.

So on Windows, the convention is:
'static' -> file internal
no marking -> shared library internal
'dllexport' -> public

And on Linux it's:
'static' -> file internal
'visibility (hidden)' -> shared library internal
no marking -> public

It's generally agreed that Linux got this wrong and that you should always
use '-fvisibility=hidden' to switch it to the windows style, but cython
doesn't control compiler options and thus should probably generate code
that works correctly regardless of such compiler settings. Fortunately,
Linux does provide some markings to explicitly make things public: you can
mark a symbol 'visibility (default)' (which means public), or you can use
the dllexport syntax, just like Windows, because gcc is helpful like that.

OTOH, windows is annoying because of this dllimport thing that started this
whole thread: on other systems just marking symbols as extern is enough to
handle both shared-object-internal and shared-library-exported symbols, and
the linker will sort it out. On Windows, you have to explicitly distinguish
between these. (And annoyingly, if you accidentally leave out the dllimport
making on functions then it will use some fallback hack that works but
silently degrades performance; on other symbols it just doesn't work, and
ditto for if you use it when it isn't needed.)

So final conclusion: for non-static symbols, cython should first decide
whether they are supposed to be shared-library-internal or actually
exported from the shared library.

For shared-library-internal symbols: their definition should be marked
'visibility(hidden)' on Linux, and unmarked on Windows. This is easy using
some preprocessor gunk. (Or maybe simplest is: marked that everywhere
except if using msvc, because I think everyone else will understand
'visibility (hidden)' even if it's a no op.) Their declaration in the
header file should just be 'extern' everywhere.

For shared-library-exported symbols: I am dubious about whether cython
should even support these at all. But if it does, then the definitions
should be marked 'dllexport' (no macro trickery needed, because everyone
understands this syntax), and their declaration in the header file needs
some extra hack that is the subject of this thread.

Now, back to your example: Here the caller and callee are both compiled
into the same shared library, so you don't want dllexport/dllimport at all,
you just want a shared-library-internal symbol, which as we see is much
easier.

NumPy also ran into some problems with this in our experiments with using
cython internally. Our temporary solution was to use the preprocessor to
monkeypatch DL_IMPORT into expanding to the appropriate
shared-library-internal thing :-).

> > My first question is why you even need this, since AFAIK there are no
cases
> > where it is correct to have a cython module dllexporting symbols that
appear
> > in header files. This is what cimport is for, right?
>
> I don't think this has anything to do with cimport.  Could you explain
>

Re: [Cython] Fwd: Question about how best require compiler options for C sources

[Nathaniel Smith]

On Apr 11, 2016 06:23, "Erik Bray"  wrote:
>
> On Mon, Apr 11, 2016 at 2:51 PM, Nathaniel Smith  wrote:
> > Now, back to your example: Here the caller and callee are both compiled
into
> > the same shared library, so you don't want dllexport/dllimport at all,
you
> > just want a shared-library-internal symbol, which as we see is much
easier.
>
> Sorry, I'll respond more to your (helpfully detailed and precise)
> message in a second.  But something I wanted to point out is that my
> example is incomplete and I should have arranged a more complete
> example.  In this case I really do want the symbol "hello" to be
> exported by the DLL, as well as be understood between translation
> units making up the same library.  A more complete example would have
> shown a separate library which links with "foo" and uses the "hello"
> function in it.
>
> Yes, it's arguable that exporting anything more than then initmodule
> function from a Python extension module is not best practice, but the
> possibility should not be ruled out either.
>
> So I think later on you hit correctly on the deeper problem, which is
> that Cython currently doesn't have a great way to distinguish between
> intra-library visibility and *inter*-library visibility.
>
> And if both are needed then some DL_IMPORT-like macro is needed that
> sets the visibility for a symbol correctly depending on the context in
> which it's being used. (And yes, this is not a problem on *nix, but it
> is on Windows due to the way __declspec(dllimport) causes name
> mangling :(

This is highly tangential to the main conversation, but FYI and in the
general interests of demystifying this stuff: the reason for all this
rigmarole on Windows is that dllimport doesn't just cause name mangling, it
causes *type* mangling. The way windows symbol relocations work, is that
you can only import pointers. So code like

  __declspec(dllimport) extern void f(...);
  __declspec(dllimport) extern int myvalue;

is syntactic sugar for something like:

  // pointer that will be filled in by loader
  void (*__imp_f)(...);
  // desugar direct calls into indirect calls
  #define f(...) (*__imp_f)(...)

  // similar
  int *__imp_myint;
  #define myint (*__imp_myint)

...except that (a) instead of using the preprocessor to perform the
substitution, it happens in the compiler frontend, so it can correctly
follow scoping rules and things that the preprocessor can't, and (b) the
linker will also automagically generate a function like

  void f(...) {
  return (*__imp_f)(...);
  }

So there shouldn't still be any mentions of 'f' in the resulting file --
they should all be replaced by mentions of (*__imp_f) -- but just in case
we missed any your code will still work, albeit at the cost of some icache
pollution and an extra indirect jump at every call.

Note in particular that this __imp_ nonsense *isn't* an approximation on my
part, there really will be a shared-library-internal symbol called
__imp_whatever whose value is a pointer that gets filled in my the loader.
(Except possibly I got my underscores wrong -- on phone so too lazy to
check.) You could literally write the code I wrote above with the #define's
and it would actually work on windows...

-n
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] bug report on cython-mode.el: freezes when using which-function-mode

[Nathaniel Smith]

Hi all,

Bug report here -- trying to edit some cython code in emacs just now,
emacs was repeatedly freezing until I'd hit C-g repeatedly. Made
things literally unusable -- I couldn't type characters into the
buffer. M-x toggle-debug-on-quit gives the backtrace:

Debugger entered--Lisp error: (quit)
  syntax-ppss()
  python-nav-beginning-of-statement()
  cython-beginning-of-block()
  cython-current-defun()
  run-hook-with-args-until-success(cython-current-defun)
  which-function()
  which-func-update-1(#)
  which-func-update()
  apply(which-func-update nil)
  timer-event-handler([t 0 0 50 t which-func-update nil idle 0])

Which strongly suggests that the problem has something to do with my
having which-function-mode enabled, and likely that something is wrong
with cython-current-defun. (which-function-mode is a minor mode built
into emacs.) Toggling which-function-mode off seems tentatively to
have fixed the problem. So there's a workaround, but really
cython-mode + which-function-mode shouldn't cause freezes :-).

Possible contributing factor: this emacs is built from a git snapshot
of master ("GNU Emacs 25.1.50.1 (x86_64-pc-linux-gnu, GTK+ Version
3.18.9) of 2016-04-22"), so it has the git versions of python-mode and
which-function-mode. (I'm just using the python.el that ships with
emacs, + elpy. But toggling elpy off didn't seem to affect the hangs.)
I don't know whether the same thing happens with released versions of
emacs.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Suggested action for "Python.h not found"

[Nathaniel Smith]

On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
 wrote:
> Spack gives you complete control over your compiler.  This is important if
> you're  building Python extensions, especially C++-based extensions, which
> must be built with the same compilers use to build Python.

Just to hopefully avoid confusing people -- you mostly only need to
worry about this on Windows. On OS X and Linux, I can't think of any
situation where using a different compiler for your extension and for
Python will cause a problem.

The one thing to watch out for these days is that if you have two
different C++ libraries that are being linked together (so this
doesn't apply to CPython itself -- it's not a C++ library), and the
interface between the libraries uses std::string or std::list, and
you're on Linux, then you need to be careful about the GCC 5 / C++11
ABI transition. The short version is that if you consistently use
versions of GCC <5 OR consistently use versions of GCC >=5, then you
should be fine; the long version can be found by googling :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Suggested action for "Python.h not found"

[Nathaniel Smith]

On Jun 9, 2016 6:28 AM, "Elizabeth A. Fischer" <
elizabeth.fisc...@columbia.edu> wrote:
>
>
>
> On Thu, Jun 9, 2016 at 2:49 AM, Nathaniel Smith  wrote:
>>
>> On Wed, Jun 8, 2016 at 10:37 PM, Elizabeth A. Fischer
>>  wrote:
>> > Spack gives you complete control over your compiler.  This is
important if
>> > you're  building Python extensions, especially C++-based extensions,
which
>> > must be built with the same compilers use to build Python.
>>
>> Just to hopefully avoid confusing people -- you mostly only need to
>> worry about this on Windows. On OS X and Linux, I can't think of any
>> situation where using a different compiler for your extension and for
>> Python will cause a problem.
>
>
> It could cause a problem if your extension is a wrapper around a library
that you wrote in  C++ or Fortran --- whether you wrote your extension by
hand, or especially if you used the Cython variant that generates  C++ code
(a logical choice, if you're interfacing to a C++ library).  If your C++
Cython extension requires a different compiler than that used to build
Cython, you will also find that standard  Cython ways to building your
extension break.

Right, if you're compiling cython to C++, and then that C++ is calling into
a different piece of C++, then you have two C++ modules that are calling
each other, and my caveat about that applies.

> Maybe you're right --- maybe the extension itself is built in C, and all
 C++ elements are safely "encapsulated" away from Python.

They are: Python itself has no C++ interfaces, so the interface between
Python and your code is totally stable and compiler-independent in all
cases I know of on Linux and OS X.

> Maybe you can coax Cython into building extensions with your chosen
compiler.  But who wants to spend time finding out exactly which
heterogeneous compilers work together, and how to avoid the standard Cython
build paths?

Sure, if you have the means to easily use the same compiler everywhere then
it's a reasonable choice and definitely safe. But there are also lots of
people who can't build everything from source all the time or who wish to
distribute binaries to users who can't build from source all the time, and
I want to make sure those people have accurate information available.

Lots of urban legends tend to accrete around this stuff. For example, it's
widely believed that on Windows, msvc uses the ".lib" format for describing
.dll interfaces, and mingw-w64 uses the ".a" format, and so every time you
want to use mingw-w64 and link against a msvc-compiled library, you have to
throw away the .lib that shipped shipped with the .dll and write some nasty
code to regenerate it from scratch in .a format.

It turns out that this is totally wrong: the truth is that both msvc and
mingw-w64 are happy to work with both .lib and .a files; the whole legend
mostly arose because of a tiny bug in how mingw-w64 was handling .lib files
(literally there was a flag field that could have two possible values but
it only checked for one of them -- fixing it was a ~6 line patch). But this
bug persisted for a decade because all the users who hit this bug decided
that it wasn't supposed to work so they never filed a bug report.

> The problem is more general than compilers --- it can actually bite you
ANY time you use two different versions of a shared library in a build that
you ultimately link together.  Suppose your Python stack was built with
NetCDF4 but you built your extension with NetCDF3?  In a software stack of
50 packages, the possibilities for this kind of problem are endless --- or
more precisely, there are 50 opportunities for this kind of problem.  And
they are almost impossible to prevent if you're doing things by hand.
Sometimes you can get away with it, and sometimes these problems result in
strange, time-sucking bugs.  Spack ensures that only ONE version of each
package is linked in your final binary.

Yeah, all major platforms make it *possible* to handle this, but it's
tricky and requires some platform specific knowledge that's very very
poorly documented. Definitely the easiest approach if you control your
whole stack is to just not use multiple different libraries versions
together. But if someone is reading this message who has this problem and
doesn't control their entire stack, then they should come ask on the
wheel-build...@python.org mailing list and we can help them figure out how
to make things work right. :-)

>> and the
>> interface between the libraries uses std::string or std::list, and
>> you're on Linux, then you need to be careful about the GCC 5 / C++11
>> ABI transition. The short version is that if you consistently use
>> versions of GCC <5 OR consistently use versions of GCC >=5, then you

Re: [Cython] Drop Cython wheels for 2.6?

[Nathaniel Smith]

On Mon, Nov 20, 2017 at 4:08 PM, Matthew Brett  wrote:
> Hi,
>
> The manylinux1 docker image has just stopped supporting Python 2.6, so
> we can no longer build Python 2.6 wheels without putting some hacks on
> top:
>
> https://github.com/pypa/manylinux/pull/125#issuecomment-345770870
>
> Do y'all care about that?

To clarify, it's not so much that the docker image officially dropped
support, as that pip dropped support and the last rebuild happened to
pick that up, so we should figure out whether to drop it officially or
what: https://github.com/pypa/manylinux/issues/126

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Python subinterpreters support problem in v0.29

[Nathaniel Smith]

(resending to cython-devel list since my first attempt bounced)

On Tue, Dec 11, 2018, 11:56 Nathaniel Smith  FYI – you should be aware that subinterpreters are poorly tested (AFAIK
> ceph is the third project to try using them, ever), not well supported in
> general, and there has even been some discussion of removing them from
> CPython. For example, numpy has never supported being used with
> subinterpreters, and currently has no plans to fix this. I suspect other
> extension modules are in similar positions, but since the bugs that
> subinterpreters trigger are often really hard to detect or debug, no one
> really knows.
>
> On Tue, Dec 11, 2018, 11:11 Ricardo Dias 
>> Hi Cython developers,
>>
>> In the recent Cython 0.29 version was introduced a commit [1] that
>> hinders the usage of python subinterpreters.
>>
>> I discovered this the hard way when suddenly a component I was working
>> on started to crash. The component in question is the ceph-mgr daemon
>> from the Ceph project [2].
>>
>> Python subinterpreters are the basic building block for the
>> plugin/module architecture of ceph-mgr. Each "manager module" runs in
>> its own python subinterpreter. Furthermore, all python bindings for the
>> client libraries of Ceph, such as librados, librbd, libcephfs, and
>> librgw, are implemented as Cython modules, and in the particular case of
>> librados, all ceph-mgr plugin modules import the rados Cython module
>> upon initialization.
>>
>> In practice, with Cython 0.29 we can only load one module, because the
>> following modules will refuse to load.
>>
>> After discovering this issue, we "temporarily" prevent the issue by
>> restricting the version of Cython as a dependency [3]. But we don't want
>> to keep this restriction indefinitely and would prefer a fix from the
>> Cython side.
>>
>> Do you think it's feasible to implement a flag to disable the safe guard
>> introduced in [1]? That way we could re-enable subinterpreters at our
>> own risk.
>>
>>
>> [1]
>>
>> https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50
>> [2] https://github.com/ceph/ceph
>> [3] https://github.com/ceph/ceph/pull/25328
>>
>> --
>> Ricardo Dias
>> Senior Software Engineer - Storage Team
>> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>> HRB 21284
>> (AG Nürnberg)
>>
>>
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>>
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel