Re: [Cython] Multiple modules in one compilation unit

[mark florisson]

On 3 March 2011 07:43, Stefan Behnel  wrote:
> Lisandro Dalcin, 03.03.2011 05:38:
>>
>> On 2 March 2011 21:01, Greg Ewing  wrote:
>>>
>>> Stefan Behnel wrote:

 you'd call "cython" on a package and it would output a directory with a
 single __init__.so that contains the modules compiled from all .pyx/.py
 files in that package. Importing the package would then trigger an
 import of
 that __init__.so, which in turn will execute code in its init__init__()
 function to register the other modules.
>>>
>>> I don't think it even has to be a directory with an __init__,
>>> it could just be an ordinary .so file with the name of the
>>> package.
>>>
>>> I just tried an experiment in Python:
>>>
>>> # onefilepackage.py
>>> import new, sys
>>> blarg = new.module("blarg")
>>> blarg.thing = "This is the thing"
>>> sys.modules["onefilepackage.blarg"] = blarg
>>>
>>> and two different ways of importing it:
>>>
>>> >>> from onefilepackage import blarg
>>> >>> blarg
>>> 
>>> >>> blarg.thing
>>> 'This is the thing'
>>>
>>> >>> import onefilepackage.blarg
>>> >>> onefilepackage.blarg.thing
>>> 'This is the thing'
>>>
>>
>> I'm hacking around these lines. However, I'm working to maintain
>> different modules in different C compilation units, in order to
>> workaround the obvious issue with duplicated global C symbols.
>
> That should be ok as a first attempt to get it working quickly. I'd still
> like to see the modules merged in the long term in order to increase the
> benefit of the more compact format. They'd all share the same code generator
> and Cython's C internals, C helper functions, constants, builtins, etc., but
> each of them would use a separate (name mangling) scope to keep the visible
> C names separate.

I was thinking that perhaps we could share declarations common to all
cython modules (compiled with that version of Cython) in a cython.h
header file (which is also imho nicer to maintain than a code.put() in
e.g. ModuleNode.generate_module_preamble), and put it in e.g.
Cython/Includes and set the -I C compiler flag to point to the
Includes directory.
Module-specific functions would still be declared static, of course.
And if users want to ship generated C files to avoid Cython as a
dependency, they could simply ship the header and adjust their
setup.py.

If you want to merge modules and the "package-approach" is chosen, it
should have well-defined semantics for in-place builds, and
package/__init__.py is preferred over package.so. So how would you
solve that problem without either placing package.so in the package
itself, or giving it another name (and perhaps star-importing it from
__init__.py)? Basically, if people want to combine several modules
into one they could use the 'include' statement.

e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include
"ham.pxd"'.

(although you'd probably rename ham.pyx to ham.pxi, and you'd probably
merge spam.pxd with ham.pxd)

In any case, I'm just wondering, would this functionality be more
useful than our current include statement and a cython.h header that
is shared by default?

>>> So assuming the same thing works with a .so instead of a .py,
>>> all you need to do is emit a .so whose init function stuffs
>>> appropriate entries into sys.modules to make it look like
>>> a package.
>>>
>>
>> However, the import machinery does not treat .so the same as *.pyx.
>> For example, I have a problem with Python 3. For .py modules, before
>> the module code starts to execute, the matching entries in sys.modules
>> is already there.
>
> And it has to be, in order to prevent accidental reimports.
>
>
>> The same happens in Python 2 for .so modules, as
>> Py_InitModule() add the entry in sys.modules early. However, in Python
>> 3 that is not te case (and only for the .so, for .py is the same as in
>> Py2), the import machinery adds the entry later, after the
>> finalization of the module init function. I'm tempted to workaround
>> this by setting the entry in sys.modules right after the call to
>> PyModule_Create() ... What do you think about this? Any potential
>> issue?
>
> No idea. I'd say, read the source and give it a try.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Multiple modules in one compilation unit

[mark florisson]

On 3 March 2011 10:48, Vitja Makarov  wrote:
> 2011/3/3 mark florisson :
>> On 3 March 2011 07:43, Stefan Behnel  wrote:
>>> Lisandro Dalcin, 03.03.2011 05:38:
>>>>
>>>> On 2 March 2011 21:01, Greg Ewing  wrote:
>>>>>
>>>>> Stefan Behnel wrote:
>>>>>>
>>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>>> files in that package. Importing the package would then trigger an
>>>>>> import of
>>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>>> function to register the other modules.
>>>>>
>>>>> I don't think it even has to be a directory with an __init__,
>>>>> it could just be an ordinary .so file with the name of the
>>>>> package.
>>>>>
>>>>> I just tried an experiment in Python:
>>>>>
>>>>> # onefilepackage.py
>>>>> import new, sys
>>>>> blarg = new.module("blarg")
>>>>> blarg.thing = "This is the thing"
>>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>>
>>>>> and two different ways of importing it:
>>>>>
>>>>> >>> from onefilepackage import blarg
>>>>> >>> blarg
>>>>> 
>>>>> >>> blarg.thing
>>>>> 'This is the thing'
>>>>>
>>>>> >>> import onefilepackage.blarg
>>>>> >>> onefilepackage.blarg.thing
>>>>> 'This is the thing'
>>>>>
>>>>
>>>> I'm hacking around these lines. However, I'm working to maintain
>>>> different modules in different C compilation units, in order to
>>>> workaround the obvious issue with duplicated global C symbols.
>>>
>>> That should be ok as a first attempt to get it working quickly. I'd still
>>> like to see the modules merged in the long term in order to increase the
>>> benefit of the more compact format. They'd all share the same code generator
>>> and Cython's C internals, C helper functions, constants, builtins, etc., but
>>> each of them would use a separate (name mangling) scope to keep the visible
>>> C names separate.
>>
>> I was thinking that perhaps we could share declarations common to all
>> cython modules (compiled with that version of Cython) in a cython.h
>> header file (which is also imho nicer to maintain than a code.put() in
>> e.g. ModuleNode.generate_module_preamble), and put it in e.g.
>> Cython/Includes and set the -I C compiler flag to point to the
>> Includes directory.
>> Module-specific functions would still be declared static, of course.
>> And if users want to ship generated C files to avoid Cython as a
>> dependency, they could simply ship the header and adjust their
>> setup.py.
>>
>> If you want to merge modules and the "package-approach" is chosen, it
>> should have well-defined semantics for in-place builds, and
>> package/__init__.py is preferred over package.so. So how would you
>> solve that problem without either placing package.so in the package
>> itself, or giving it another name (and perhaps star-importing it from
>> __init__.py)? Basically, if people want to combine several modules
>> into one they could use the 'include' statement.
>>
>> e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include
>> "ham.pxd"'.
>>
>> (although you'd probably rename ham.pyx to ham.pxi, and you'd probably
>> merge spam.pxd with ham.pxd)
>>
>> In any case, I'm just wondering, would this functionality be more
>> useful than our current include statement and a cython.h header that
>> is shared by default?
>>
>>>>> So assuming the same thing works with a .so instead of a .py,
>>>>> all you need to do is emit a .so whose init function stuffs
>>>>> appropriate entries into sys.modules to make it look like
>>>>> a package.
>>>>>
>>>>
>>>> However, the import machinery does not treat .so the same as *.pyx.
>>>> For example, I have a problem with Python 3. For .py modules, before
>>>> the module code starts to execute, the matching

[Cython] OpenMP support

[mark florisson]

I'd like to implement OpenMP support for Cython. Looking at
http://wiki.cython.org/enhancements/openmp I agree a close
1:1 mapping would be nice. It would probably make sense to start with
support for 'nogil' sections because GIL-holding
sections would be hard to deal with considering all the 'goto's that
are generated (because you may not leave the
OpenMP block). See this thread
http://comments.gmane.org/gmane.comp.python.cython.devel/8695 for a
previous discussion.

Looking also at http://wiki.cython.org/enhancements/parallel , for the
'parallel for' support I think the best syntax would
be to introduce a fake 'cython.openmp' module as Dag suggested. I
propose to start with the following syntax:

from python (c)import openmp

with nogil:
with openmp.parallel('sections'):
with openmp.section():
...

I'd like to support the directives as part of the parallel directive,
and the stand-alone counterparts.
So the equivalent of the above would read:

with nogil:
with openmp.parallel():
with openmp.sections():
with openmp.section():
...

Perhaps to apply an OpenMP directive to a single statement instead of
a structured block, we could follow OpenMP's
approach, i.e. allow writing the following instead of its 'with' equivalent:

openmp.task(untied=True)
a = 1

For the 'for' and 'parallel for' constructs I propose an
openmp.range() and openmp.prange() respectively, i.e.

with openmp.parallel():
for i in openmp.range(..., firstprivate=('a', 'b'),
reduction='+:result'):
...

could be written

for i in openmp.prange(..., firstprivate=('a', 'b'), reduction='+:result'):
...

for short.

In the previous thread Sturla mentioned that perhaps it would be
easier to implement OpenMP support manually instead of
generating OpenMP pragmas and relying on OpenMP compiler support.
However, considering that OpenMP has quite a few constructs,
not to mention the functionality provided by the OpenMP library itself
(all the omp_* functions), I think it will be
easier to go the "pragma route", especially if we choose to provide
all (or a substantial part) of the OpenMP functionality.
Then the OpenMP runtime library (e.g. libgomp when using gcc) can
simply be wrapped with an appropriate .pxd in Cython/Includes.

What do you guys think?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[mark florisson]

On 8 March 2011 16:10, Sturla Molden  wrote:
> Den 08.03.2011 11:34, skrev mark florisson:
>>
>> However, considering that OpenMP has quite a few constructs,
>
> No, OpenMP has very few contructs, not quite a few. And most of them are not
> needed, nor wanted, because the functionality is covered by the Python
> language (such as scoping rules).
>
> I.e. we do not need -- nor want -- things like private, shared, firstprivate
> or lastprivate in Cython, because the scope of variables inside closures are
> already covered by Python syntax. OpenMP is just making up for the lack of
> closures in C here.
>
> We don't need to implement #pragma omp critical, because we have
> threading.Lock.
>
> We don't need to implement #pragma omp atomic, because we have the GIL (as
> well as OS and compiler support for atomic operations).
>
> #pragma omp barrier becomes a butterfly of threading.Event.
>
> Etc.

But how useful is it to parallelize CPU-bound code while holding GIL?
Or do you mean to run the CPU-intensive section in a 'with nogil'
block and when you need to do locking, or when you need to deal with
Python objects you reaqcuire the GIL?

The point of OpenMP is convenience, i.e., having your CPU-bound
algorithm parallelized with just a few annotations. If you rewrite
your code as a closure for say, a parallel for construct, you'd have
to call your closure at every iteration. And then you still have to
take care of any reduction and corresponding synchronization (unless
you have the GIL already). And then there's still the issue of
ordered, single and master constructs.

Of course, using threads manually is always possible, it's just not
very convenient.

> There is not really much left to implement. We need a decorator to launch
> closures as parallel blocks (trivial) and classes to do static, dynamic and
> guided load balancing (trivial as well).
>
> It is hardly a page of code to write, just a handful of small classes, once
> Cython has closures working properly.

What do you mean with proper closures? We have closure support right now.

> Sturla
>
>
There is a lot of whitespace at the end of most of your messages.
>
>
>
>
>
>
>
>
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[mark florisson]

On 8 March 2011 16:28, Dag Sverre Seljebotn  wrote:
> On 03/08/2011 11:34 AM, mark florisson wrote:
>>
>> I'd like to implement OpenMP support for Cython. Looking at
>
> Great news! It looks like this will be a topic on the coming workshop, with
> Francesc coming as well (but nothing wrong with getting started before
> then).
>
> (And please speak up if you are in the right group for a GSoC.)

I would love to participate, the problem is that I'm really not sure
whether it will fit in my schedule. I will certainly figure this out.

>> http://wiki.cython.org/enhancements/openmp I agree a close
>> 1:1 mapping would be nice. It would probably make sense to start with
>> support for 'nogil' sections because GIL-holding
>> sections would be hard to deal with considering all the 'goto's that
>> are generated (because you may not leave the
>> OpenMP block). See this thread
>> http://comments.gmane.org/gmane.comp.python.cython.devel/8695 for a
>> previous discussion.
>>
>> Looking also at http://wiki.cython.org/enhancements/parallel , for the
>> 'parallel for' support I think the best syntax would
>> be to introduce a fake 'cython.openmp' module as Dag suggested. I
>> propose to start with the following syntax:
>>
>>     from python (c)import openmp
>>
>>     with nogil:
>>         with openmp.parallel('sections'):
>>             with openmp.section():
>
> I've changed my opinion a little bit since that thread; I'm now rather
> undecided.
>
> Pros:
>
>  - Easy to get results fast (nothing wrong about Sturla's approach, but
> realistically it *will* take longer to make it work)
>  - OpenMP already has significant mindshare. Saying "Cython supports OpenMP"
> will sound soothing to scientific Fortran and C programmers, even if it is
> technically inferior to other solutions we could find.
>
> Cons:
>
>  - Like Sturla says, closures maps this better in the Cython language
> (having "private=('a', 'b')" as a syntax for referring to the local
> variables a and b is rather ugly)
>  - The goto restriction that makes exception handling harder

With OpenMP code, exactly how common are exceptions and error handling?
I agree that listing privates that way doesn't look very appealing.
Maybe 'private=(a, b)' is somewhat better. Or perhaps we should really
divert to syntax like

cython.openmp("parallel sections private(a, b)"):
...

and implement a little OpenMP pragma parser (which shouldn't be very hard).

> I think that long-term we must find some middle ground that sits between
> just having the GIL and not having it. E.g., find some way of raising an
> exception from a nogil block, and also allow more simple code to "get more
> data to work on" within the context of a single thread. So what you're
> saying about goto's making me lean against OpenMP.
>
> But, perfect is the enemy of good. I certainly won't vote against having
> this implemented -- because *anything* is better than nothing, and nothing
> prevents us from building something better in addition to or on top of an
> initial OpenMP implementation.
>
> BTW, I found this framework interesting ...  it cites some other frameworks
> as well (cilk, OpenMP, Intel Threading Blocks), which it beats in a specific
> situation.
>
> http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:about
>
> I'm NOT saying we should "support FastFlow instead". What I'm thinking is
> that perhaps the place to start is to ask ourselves: What are the current
> obstacles to using parallelization frameworks written for C or C++ with
> Cython? How would one use Cython with these, and do it in a practical way
> (i.e. not just C code in a nogil block without even exception handling).

I think OpenMP is not really designed for error handling, so if we'd
go the OpenMP route I think the only way to support exceptions would
be to save a pending exception, and "raise" it after the construct
ends (in the sequential part), just outside the nogil block (in which
case it makes sense to make directives implicitly nogil).
Perhaps we can allow GIL acquisition in master constructs and as you
mentioned on the wiki, default critical constructs. Exceptions would
of course remain pending. Perhaps it's possible to guard against that,
i.e., finish the OpenMp construct but skip the actual work in case of
a pending exception. I must say, it doesn't sound very promising.

So I think we should also ask ourselves what users want, i.e., do they
usually want to deal with Python objects in their worksharing code, or
do they often want to

Re: [Cython] OpenMP support

[mark florisson]

On 8 March 2011 17:38, Sturla Molden  wrote:
> Den 08.03.2011 17:10, skrev mark florisson:
>>
>> But how useful is it to parallelize CPU-bound code while holding GIL?
>> Or do you mean to run the CPU-intensive section in a 'with nogil'
>> block and when you need to do locking, or when you need to deal with
>> Python objects you reaqcuire the GIL?
>
> The Python C API is not thread-safe, thus we cannot allow threads concurrent
> access.
>
> It does not matter if we use the GIL or something else as mutex. A
> user-space spinlock would probably be faster than kernel objects like the
> GIL. But that is just implementation details.

Right, but if you want to deal with Python objects, then you can't
just use another lock then the GIL, because there might still be other
Python threads. But perhaps you were talking of non-python-object
synchronization in nogil blocks.

>> The point of OpenMP is convenience, i.e., having your CPU-bound
>> algorithm parallelized with just a few annotations. If you rewrite
>> your code as a closure for say, a parallel for construct, you'd have
>> to call your closure at every iteration.
>
> No, that would be hidden away with a decorator.
>
> for i in range(n):
> 
>
> becomes
>
> @openmp.parallel_range(n)
> def loop(i):
> 
>
Sure, that's not what I was hinting at. What I meant was that the
wrapper returned by the decorator would have to call the closure for
every iteration, which introduces function call overhead.

>
>> And then you still have to
>> take care of any reduction and corresponding synchronization (unless
>> you have the GIL already). And then there's still the issue of
>> ordered, single and master constructs.
>
> Yes, and this is not difficult to implement. Ordered can be implemented with
> a Queue, master is just a check on thread id. Single can be implemented with
> an atomic CAS operation. This is just a line or two of library code each.
>>
>> Of course, using threads manually is always possible, it's just not
>> very convenient.
>
> No it's not, but I am not taking about that. I am talking about how to best
> map OpenMP to Python.
>
> Closure is one method, another that might be possible is a context manager
> (with-statement). I am not sure if this would be doable or not:
>
> with OpenMP( private=(i,j,k), shared=(x,y,z) ) as openmp:
> 
>
> instead of #pragma omp parallel.
>
> But should we care if this is implemented with OpenMP or Python threads?
> It's just an implementation detail in the library, not visible to the user.

Indeed. I guess we just have to establish what we want to do: do we
want to support code with Python objects (and exceptions etc), or just
C code written in Cython? If it's the latter, then I still think using
OpenMP directly would be easier to implement and more convenient for
the user than decorators with closures, but maybe I'm too focussed on
OpenMP. And indeed, the syntax I proposed (apart from maybe
cython.openmp.(p)range) does not look very attractive.

> Also I am not against OpenMP, I use it all the time in Fortran :-)
>
> Another problem with using OpenMP inside the compiler, as opposed to an
> external library, is that it depends on a stabile ABI. If an ABI change to
> Cython's generated C code is made, even a minor change, OpenMP support will
> be broken.
>
>
> Sturla
>
>
>
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[mark florisson]

On 11 March 2011 01:46, Robert Bradshaw  wrote:
> On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted  wrote:
>> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigué:
>>> mark florisson, 08.03.2011 18:00:
>>> > What I meant was that the
>>> > wrapper returned by the decorator would have to call the closure
>>> > for every iteration, which introduces function call overhead.
>>> >
>>> >[...]
>>> >
>>> > I guess we just have to establish what we want to do: do we
>>> > want to support code with Python objects (and exceptions etc), or
>>> > just C code written in Cython?
>>>
>>> I like the approach that Sturla mentioned: using closures to
>>> implement worker threads. I think that's very pythonic. You could do
>>> something like this, for example:
>>>
>>>      def worker():
>>>          for item in queue:
>>>              with nogil:
>>>                  do_stuff(item)
>>>
>>>      queue.extend(work_items)
>>>      start_threads(worker, count)
>>>
>>> Note that the queue is only needed to tell the thread what to work
>>> on. A lot of things can be shared over the closure. So the queue may
>>> not even be required in many cases.
>>
>> I like this approach too.  I suppose that you will need to annotate the
>> items so that they are not Python objects, no?  Something like:
>>
>>     def worker():
>>         cdef int item  # tell that item is not a Python object!
>>         for item in queue:
>>             with nogil:
>>                 do_stuff(item)
>>
>>     queue.extend(work_items)
>>     start_threads(worker, count)
>
> On a slightly higher level, are we just trying to use OpenMP from
> Cython, or are we trying to build it into the language? If the former,
> it may make sense to stick closer than one might otherwise be tempted
> in terms of API to the underlying C to leverage the existing
> documentation. A library with a more Pythonic interface could perhaps
> be written on top of that. Alternatively, if we're building it into
> Cython itself, I'd it might be worth modeling it after the
> multiprocessing module (though I understand it would be implemented
> with threads), which I think is a decent enough model for managing
> embarrassingly parallel operations. The above code is similar to that,
> though I'd prefer the for loop implicit rather than as part of the
> worker method (or at least as an argument). If we went this route,
> what are the advantages of using OpenMP over, say, pthreads in the
> background? (And could the latter be done with just a library + some
> fancy GIL specifications?) One thing that's nice about OpenMP as
> implemented in C is that the serial code looks almost exactly like the
> parallel code; the code at http://wiki.cython.org/enhancements/openmp
> has this property too.

Indeed, and this works very nicely for for loops because they already
constitute blocks. However, if you want to support other OpenMP-like
functionality (and not saying that we want to, but if), then you have
two options:

1) you need a way to explicitly declare blocks in Cython, or
2) you need to write all your blocks as functions and call those,
instead (most likely quite bothersome).

If we want an interface like multiprocessing, I think we can just
existing batteries, of which just one example is the functionality
described in PEP 3148 (concurrent.futures, new in 3.2). People would
only have to write Python functions that could release the GIL if
needed.

> Also, I like the idea of being able to hold the GIL by the invoking
> thread and having the "sharing" threads do the appropriate locking
> among themselves when needed if possible, e.g. for exception raising.
>
> Another thought I had is, there might be other usecases for being able
> to emit generic pragmas statements, how far would that get us?

I gave that, but more generally, a 'verbatim' clause some thought.
However, syntax would be bothersome. I thought about e.g.

cdef verbatim:
 C code here

Access to variables (i.e., substitute Cython variable names with their
name-mangled counterparts) might be provided through e.g. '$var'
syntax, as used by string.Template. But I can already hear people
object rather loudly, so I rejected the thought :). If, of course,
access to variables is not allowed but instead we allow only simple
pragmas, a simple cython.pragma("#pragma bla") should probably
suffice.

On the other hand, do we really need to mangle (non-temporary)
variable names? Global variables are declared static and variables may
be redeclared in inner C scopes.

> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[mark florisson]

On 11 March 2011 08:56, Dag Sverre Seljebotn  wrote:
> On 03/11/2011 08:20 AM, Stefan Behnel wrote:
>>
>> Robert Bradshaw, 11.03.2011 01:46:
>>>
>>> On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted
>>>  wrote:
>>>>
>>>> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigué:
>>>>>
>>>>> mark florisson, 08.03.2011 18:00:
>>>>>>
>>>>>> What I meant was that the
>>>>>> wrapper returned by the decorator would have to call the closure
>>>>>> for every iteration, which introduces function call overhead.
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>> I guess we just have to establish what we want to do: do we
>>>>>> want to support code with Python objects (and exceptions etc), or
>>>>>> just C code written in Cython?
>>>>>
>>>>> I like the approach that Sturla mentioned: using closures to
>>>>> implement worker threads. I think that's very pythonic. You could do
>>>>> something like this, for example:
>>>>>
>>>>>      def worker():
>>>>>          for item in queue:
>>>>>              with nogil:
>>>>>                  do_stuff(item)
>>>>>
>>>>>      queue.extend(work_items)
>>>>>      start_threads(worker, count)
>>>>>
>>>>> Note that the queue is only needed to tell the thread what to work
>>>>> on. A lot of things can be shared over the closure. So the queue may
>>>>> not even be required in many cases.
>>>>
>>>> I like this approach too.  I suppose that you will need to annotate the
>>>> items so that they are not Python objects, no?  Something like:
>>>>
>>>>     def worker():
>>>>         cdef int item  # tell that item is not a Python object!
>>>>         for item in queue:
>>>>             with nogil:
>>>>                 do_stuff(item)
>>>>
>>>>     queue.extend(work_items)
>>>>     start_threads(worker, count)
>>>
>>> On a slightly higher level, are we just trying to use OpenMP from
>>> Cython, or are we trying to build it into the language? If the former,
>>> it may make sense to stick closer than one might otherwise be tempted
>>> in terms of API to the underlying C to leverage the existing
>>> documentation. A library with a more Pythonic interface could perhaps
>>> be written on top of that. Alternatively, if we're building it into
>>> Cython itself, I'd it might be worth modeling it after the
>>> multiprocessing module (though I understand it would be implemented
>>> with threads), which I think is a decent enough model for managing
>>> embarrassingly parallel operations.
>>
>> +1
>>
>>
>>> The above code is similar to that,
>>> though I'd prefer the for loop implicit rather than as part of the
>>> worker method (or at least as an argument).
>>
>> It provides a simple way to write per-thread initialisation code, though.
>> And it's likely easier to make looping fast than to speed up the call into a
>> closure. However, eventually, both ways will need to be supported anyway.
>>
>>
>>> If we went this route,
>>> what are the advantages of using OpenMP over, say, pthreads in the
>>> background? (And could the latter be done with just a library + some
>>> fancy GIL specifications?)
>>
>> In the above example, basically everything is explicit and nothing more
>> than a simplified threading setup is needed. Even the implementation of
>> "start_threads()" could be done in a couple of lines of Python code,
>> including the collection of results and errors. If someone thinks we need
>> more than that, I'd like to see a couple of concrete use cases and code
>> examples first.
>>
>>
>>> One thing that's nice about OpenMP as
>>> implemented in C is that the serial code looks almost exactly like the
>>> parallel code; the code at http://wiki.cython.org/enhancements/openmp
>>> has this property too.
>>
>> Writing it with a closure isn't really that much different. You can put
>> the inner function right where it would normally get executed and add a bit
>> of calling/load distributing code below it. Not that bad IMO.
>>
>> It may be worth providing some ready-to-use decorators to d

Re: [Cython] OpenMP support

[mark florisson]

On 11 March 2011 12:13, Sturla Molden  wrote:
> Den 11.03.2011 01:46, skrev Robert Bradshaw:
>>
>> On a slightly higher level, are we just trying to use OpenMP from
>> Cython, or are we trying to build it into the language?
>
> OpenMP is a specification, not a particular implementation. Implementation
> for Cython should either be compiler pragmas or a library.
>
> I'd like it to be a library, as it should also be usable from Python. I have
> made some progress on the library route, depending on Cython's closures.
>
> nogil makes things feel a bit awkward, though.
>
> We could for example imagine code like this:
>
>    with openmp.ordered(i):
> 
>
> Context managers are forbidden in nogil AFAIK.  So we end up with ugly hacks
> like this:
>
>    with nogil:
>       if openmp._ordered(i): # always returns 1, but will synchronize
> 
>
> Would it be possible to:
>
> - Make context managers that are allowed without the GIL? We don't need to
> worry about exceptions, but it should be possible to short-circuit from
> __enter__ to __exit__.
>
> - Have cpdefs that are callable without the GIL?
>
> This would certainly make OpenMP syntax look cleaner.
>
>
> Sturla
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

The reason I wanted to support OpenMP as a compiler modification is
exactly that you can use Pythonic syntax without actually dealing with
for instance, context managers. You'd just rewrite the WithStatNode in
a Transform. By additionally stubbing the context managers in the
Shadow module you get normal, sequential behaviour when running
directly in the python interpreter (provided that Cython is installed
or the Shadow module is shipped as cython.py).
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[mark florisson]

On 11 March 2011 14:54, Sturla Molden  wrote:
> Den 11.03.2011 11:42, skrev Matej Laitl:
>>
>> #pragma omp parallel for private(var1) reduction(+:var2) schedule(guided)
>> for i in range(n):
>>     do_work(i)
>>

> I do like this, as it is valid Python and can be turned on/off with a
> compiler flag to Cython.

That's the same thing I proposed with different syntax :) The problem
with the pragmas is that we can't declare arbitrary blocks in Cython,
to which the pragmas should be applied, in which case get the kind of
awkwardness seen below.

What I liked about your proposed closures is the neat pythonic syntax
for the scoping rules of variables in the loops. I think I might have
changed my mind entirely about OpenMP(-like) support, as users can
already do a lot, and if they really want OpenMP then it will be
likely most convenient to write it in C and simply wrap it.

> Issues to warn about:
>
> - We cannot jump out of a parallel block from C/C++/Fortran (goto, longjmp,
> C++ exception). That applies to Python exceptions as well, and the generated
> Cython code.
>
> - GIL issue: CPython interpreter actually call GetCurrentThreadId(), e.g. in
> thread_nt.h. So the OpenMP thread using the Python CAPI must be the OS
> thread holding the GIL. It is not sufficient that the master thread does.
>
> - Remember that NumPy arrays are unboxed. Those local variables should be
> silently passed as firstprivate.
>
> - Refcounting with Python objects and private variables.
>
> None of the above applies if we go with the library approach. But then it
> would look less like OpenMP in C.
>
> Also, do we want this?
>
>   #pragma omp parallel
>   if 1:
> 
>
> It is a consequence of just re-using the C-syntax for OpenMP, as intendation
> matters in Cython. There are no anonymous blocks similar to C in Cython:
>
>   #pragma omp parallel
>   {
> 
>   }
>
>
>
>
> Sturla
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] 'with gil:' statement

[mark florisson]

I implemented the 'with gil:' statement, and have added error checking
for nested 'with gil' or 'with nogil' statements. For instance, with
the patch applied Cython wil issue an error when you have e.g.

with nogil:
with nogil:
...

(or the same thing for 'with gil:'). This because nested 'nogil' or
'gil' blocks, without intermediate GIL-acquiring/releasing, will abort
the Python interpreter. However, if people are acquiring the GIL
manually in nogil blocks and then want to release the GIL with a
'nogil' block, it will incorrectly issue an error. I do think this
kind of code is uncommon, but perhaps we want to issue a warning
instead?

The 'with gil:' statement can now be used in the same way as 'with
nogil:'. Exceptions raised from GIL blocks will be propagated if
possible (i.e., if the return type is 'object'). Otherwise it will
jump to the end of the function and use the usual
__Pyx_WriteUnraisable, there's not really anything new there.

For functions declared 'nogil' that contain 'with gil:' statements, it
will safely lock around around the initialization of any Python
objects and set up the refnanny context (with appropriate preprocessor
guards). At the end of the function it will safely lock around the
teardown of the refnanny context. With 'safely' I mean that it will
create a thread state if it was not already created and may be called
even while the GIL is already held (using PyGILState_Ensure()). This
means variables are declared and initialized in the same way as in
normal GIL-holding functions (except that there is additional
locking), and of course the GIL-checking code ensures that errors are
issued if those variables are attempted to be used outside any GIL
blocks.

Could someone review the patch (which is attached)? Maybe check if I
haven't missed any side cases and such?


with_gil_statement.patch
Description: Binary data
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] PyCharm

[mark florisson]

The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the
Cython project a free Open Source license, which means that anyone
developing Cython may freely use PyCharm to develop Cython. They
prefer to license to remain unpublic, so if you develop Cython and
want a free PyCharm license, send me an email and I will send you the
license key. It is valid until 13 March 2012.

Cheers,

Mark
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 11:58, Dag Sverre Seljebotn  wrote:
> On 03/16/2011 11:28 AM, mark florisson wrote:
>
> I implemented the 'with gil:' statement, and have added error checking
> for nested 'with gil' or 'with nogil' statements. For instance, with
> the patch applied Cython wil issue an error when you have e.g.
>
> with nogil:
> with nogil:
> ...
>
> (or the same thing for 'with gil:'). This because nested 'nogil' or
> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort
> the Python interpreter. However, if people are acquiring the GIL
> manually in nogil blocks and then want to release the GIL with a
> 'nogil' block, it will incorrectly issue an error. I do think this
> kind of code is uncommon, but perhaps we want to issue a warning
> instead?
>
> I think we should make nested nogil-s noops, i.e.
>
> with nogil:
>     with nogil: # => if True:
>
> This is because one may want to change "with nogil" to "with gil" for
> debugging purposes (allow printing debug information).

Interesting, that does sound convenient, but I'm not if mere
convenience should move us to simply ignore what is technically most
likely incorrect code (unless there is intermediate manual locking).
In any case, I wouldn't really be against that. If you simply want to
allow this for debugging, we could also allow print statements in
nogil sections, by either rewriting it using 'with gil:', or by
inserting a simple printf (in which case you probably want to place a
few restrictions).

> Another feedback is that I wonder whether we should put the "gil" and
> "nogil" psuedo-context managers both in cython namespace, and sort of
> deprecate the "global" nogil, rather than introduce yet another name that
> can't be used safely for all kinds of variables.

Hmm, good catch. Actually, 'with cython.nogil:' is already possible,
but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to
simply be ignored).
So I guess I'll have to fix that :)

> -- Dag
>
> The 'with gil:' statement can now be used in the same way as 'with
> nogil:'. Exceptions raised from GIL blocks will be propagated if
> possible (i.e., if the return type is 'object'). Otherwise it will
> jump to the end of the function and use the usual
> __Pyx_WriteUnraisable, there's not really anything new there.
>
> For functions declared 'nogil' that contain 'with gil:' statements, it
> will safely lock around around the initialization of any Python
> objects and set up the refnanny context (with appropriate preprocessor
> guards). At the end of the function it will safely lock around the
> teardown of the refnanny context. With 'safely' I mean that it will
> create a thread state if it was not already created and may be called
> even while the GIL is already held (using PyGILState_Ensure()). This
> means variables are declared and initialized in the same way as in
> normal GIL-holding functions (except that there is additional
> locking), and of course the GIL-checking code ensures that errors are
> issued if those variables are attempted to be used outside any GIL
> blocks.
>
> Could someone review the patch (which is attached)? Maybe check if I
> haven't missed any side cases and such?
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

>> Another feedback is that I wonder whether we should put the "gil" and
>> "nogil" psuedo-context managers both in cython namespace, and sort of
>> deprecate the "global" nogil, rather than introduce yet another name that
>> can't be used safely for all kinds of variables.
>
> Hmm, good catch. Actually, 'with cython.nogil:' is already possible,
> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to
> simply be ignored).
> So I guess I'll have to fix that :)
>

I attached a patch for 'with cython.gil:'.


with_cython.gil_statement.patch
Description: Binary data
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 13:01, mark florisson  wrote:
>>> Another feedback is that I wonder whether we should put the "gil" and
>>> "nogil" psuedo-context managers both in cython namespace, and sort of
>>> deprecate the "global" nogil, rather than introduce yet another name that
>>> can't be used safely for all kinds of variables.
>>
>> Hmm, good catch. Actually, 'with cython.nogil:' is already possible,
>> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to
>> simply be ignored).
>> So I guess I'll have to fix that :)
>>
>
> I attached a patch for 'with cython.gil:'.
>

Attached a patch to disallow invalid directives in with statements. I
ran the test suite to ensure I didn't break anything.


error_on_invalid_with_directives.patch
Description: Binary data
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 13:37, Dag Sverre Seljebotn  wrote:
> On 03/16/2011 12:54 PM, mark florisson wrote:
>>
>> On 16 March 2011 11:58, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 03/16/2011 11:28 AM, mark florisson wrote:
>>>
>>> I implemented the 'with gil:' statement, and have added error checking
>>> for nested 'with gil' or 'with nogil' statements. For instance, with
>>> the patch applied Cython wil issue an error when you have e.g.
>>>
>>> with nogil:
>>>     with nogil:
>>>         ...
>>>
>>> (or the same thing for 'with gil:'). This because nested 'nogil' or
>>> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort
>>> the Python interpreter. However, if people are acquiring the GIL
>>> manually in nogil blocks and then want to release the GIL with a
>>> 'nogil' block, it will incorrectly issue an error. I do think this
>>> kind of code is uncommon, but perhaps we want to issue a warning
>>> instead?
>>>
>>> I think we should make nested nogil-s noops, i.e.
>>>
>>> with nogil:
>>>     with nogil: # =>  if True:
>>>
>>> This is because one may want to change "with nogil" to "with gil" for
>>> debugging purposes (allow printing debug information).
>>
>> Interesting, that does sound convenient, but I'm not if mere
>> convenience should move us to simply ignore what is technically most
>> likely incorrect code (unless there is intermediate manual locking).
>
> I'm not sure if I understand what you mean here. How does simply ignoring
> redundant "with [no]gil" statements cause incorrect code? Or do you mean
> this is a
>
> I'm just trying to minimize the "language getting in your way" factor. It is
> pretty useless to write
>
> if x:
>    if x:
>        ...
>
> as well, but Python does allow it.
>
> Warnings on nested "with nogil" is more the role of a "cylint" in my
> opinion.
>

Perhaps you're right. However, I just think it is important for users
to realize that in general, they cannot unblock threads recursively.
Currently the error checking code catches multiple nested 'with
(no)gil', but it doesn't catch this:

cdef void func() nogil:
with nogil:
pass

with nogil:
func()

But the problem is that it does abort the interpreter. So I thought
that perhaps emphasizing that that code is incorrect for at least the
easy-to-catch cases, we might make users somewhat more aware. Because
if the above code aborts Python, but a nested 'with nogil:' is valid
code, there might be a source for confusion.

>> In any case, I wouldn't really be against that. If you simply want to
>> allow this for debugging, we could also allow print statements in
>> nogil sections, by either rewriting it using 'with gil:', or by
>> inserting a simple printf (in which case you probably want to place a
>> few restrictions).
>
> It's not only print statements. I.e., if I think something is wrong with an
> array, I'll stick in code like
>
> print np.std(x), np.mean(x), np.any(np.isnan(x))
>
> or something more complex that may require temporaries. Or even plot the
> vector:
>
> plt.plot(x)
> plt.show() # blocks until I close plot window
>
> Or, launch a debugger:
>
> if np.any(np.isnan(x)):
>     import pdb; pdb.set_trace()
>
> so I guess I should stop saying this is only about printing. In general,
> it's nice to use Python during debugging. I find myself replacing "with
> nogil" with "if True" in such situations, to avoid reindenting.
>
> I guess I can soon start using "with gil" around the debug code though.
> Again, the restriction on nested nogil/gil is not a big problem, just an
> instance of "the language getting in your way".

I understand. If your code doesn't introduce Cython-level blocks, you
could at least put it on one line so you can easily comment it out.

> Dag
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 13:55, mark florisson  wrote:
> On 16 March 2011 13:37, Dag Sverre Seljebotn  
> wrote:
>> On 03/16/2011 12:54 PM, mark florisson wrote:
>>>
>>> On 16 March 2011 11:58, Dag Sverre Seljebotn
>>>  wrote:
>>>>
>>>> On 03/16/2011 11:28 AM, mark florisson wrote:
>>>>
>>>> I implemented the 'with gil:' statement, and have added error checking
>>>> for nested 'with gil' or 'with nogil' statements. For instance, with
>>>> the patch applied Cython wil issue an error when you have e.g.
>>>>
>>>> with nogil:
>>>>     with nogil:
>>>>         ...
>>>>
>>>> (or the same thing for 'with gil:'). This because nested 'nogil' or
>>>> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort
>>>> the Python interpreter. However, if people are acquiring the GIL
>>>> manually in nogil blocks and then want to release the GIL with a
>>>> 'nogil' block, it will incorrectly issue an error. I do think this
>>>> kind of code is uncommon, but perhaps we want to issue a warning
>>>> instead?
>>>>
>>>> I think we should make nested nogil-s noops, i.e.
>>>>
>>>> with nogil:
>>>>     with nogil: # =>  if True:
>>>>
>>>> This is because one may want to change "with nogil" to "with gil" for
>>>> debugging purposes (allow printing debug information).
>>>
>>> Interesting, that does sound convenient, but I'm not if mere
>>> convenience should move us to simply ignore what is technically most
>>> likely incorrect code (unless there is intermediate manual locking).
>>
>> I'm not sure if I understand what you mean here. How does simply ignoring
>> redundant "with [no]gil" statements cause incorrect code? Or do you mean
>> this is a
>>
>> I'm just trying to minimize the "language getting in your way" factor. It is
>> pretty useless to write
>>
>> if x:
>>    if x:
>>        ...
>>
>> as well, but Python does allow it.
>>
>> Warnings on nested "with nogil" is more the role of a "cylint" in my
>> opinion.
>>
>
> Perhaps you're right. However, I just think it is important for users
> to realize that in general, they cannot unblock threads recursively.
> Currently the error checking code catches multiple nested 'with
> (no)gil', but it doesn't catch this:
>
> cdef void func() nogil:
>    with nogil:
>        pass
>
> with nogil:
>    func()
>
> But the problem is that it does abort the interpreter. So I thought
> that perhaps emphasizing that that code is incorrect for at least the
> easy-to-catch cases, we might make users somewhat more aware. Because
> if the above code aborts Python, but a nested 'with nogil:' is valid
> code, there might be a source for confusion.
>

I have to mention, though, that the converse is not true. The gil may
be acquired recursively, i.e., 'with gil:' blocks may be nested.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] PyCharm

[mark florisson]

On 16 March 2011 15:26, Stefan Behnel  wrote:
> mark florisson, 16.03.2011 11:33:
>>
>> The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the
>> Cython project a free Open Source license, which means that anyone
>> developing Cython may freely use PyCharm to develop Cython.
>
> Looks like they don't support Cython code, though:
>
> http://youtrack.jetbrains.net/issue/PY-1046
>
> (although, maybe that's the reason for the offer ;)
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Indeed, that is rather unfortunate. I only use it for plain Python
code, so I work with two editors, which is somewhat inconvenient.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 15:07, Stefan Behnel  wrote:
> mark florisson, 16.03.2011 13:28:
>>
>> On 16 March 2011 13:01, mark florisson wrote:
>>>>>
>>>>> Another feedback is that I wonder whether we should put the "gil" and
>>>>> "nogil" psuedo-context managers both in cython namespace, and sort of
>>>>> deprecate the "global" nogil, rather than introduce yet another name
>>>>> that
>>>>> can't be used safely for all kinds of variables.
>>>>
>>>> Hmm, good catch. Actually, 'with cython.nogil:' is already possible,
>>>> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to
>>>> simply be ignored).
>>>> So I guess I'll have to fix that :)
>>>>
>>>
>>> I attached a patch for 'with cython.gil:'.
>
> Looks ok.

Does that mean I should push the changes, or do you want some more
elaborate reviewing first?

>
>> Attached a patch to disallow invalid directives in with statements.
>
> +1, except that I'd print the directive name as part of the error message.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 16:14, Stefan Behnel  wrote:
> mark florisson, 16.03.2011 11:28:
>>
>> I implemented the 'with gil:' statement
>> [...]
>> The 'with gil:' statement can now be used in the same way as 'with
>> nogil:'. Exceptions raised from GIL blocks will be propagated if
>> possible (i.e., if the return type is 'object'). Otherwise it will
>> jump to the end of the function and use the usual
>> __Pyx_WriteUnraisable, there's not really anything new there.
>
> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
> handle exceptions, so simply jumping out of them because an inner 'with gil'
> block raised an exception can have unexpected side effects.
>
> Would you do the same when calling a cdef function that uses "with gil" in
> its signature? We don't currently do that, but it feels like it's the same
> situation.
>

I beg to differ. Consider this code:

with nogil:
with gil:
raise SomeError
printf("something\n")

With normal 'with' statements the printf (in the snippet above) would
be skipped, and so it is with 'with gil:' blocks, it follows the exact
semantics of the Python behaviour as you expect it. With statements
are like try/finally, and the 'with (no)gil:' blocks are implemented
as such. The above would translate to:

try:
   release gil
   try:
   acquire gil
   raise SomeError
   finally:
   release gil
   printf("something\n")
finally:
acquire gil

If users need to do (C-level) cleanup, they need to wrap the code in
the 'with gil:' in something like try/finally, or perhaps try/except.

In fact, if you declare a function 'with gil', you will want to either
handle any exception in that function, or you want your caller to
check for errors while returning an error indicator. So in that case,
you should still do error checking, if you don't, exceptions will
simply be ignored.

Another related issue, you cannot declare your function with both
'with gil' and  'except NULL' at the same time, as calling such a
function without having the GIL will segfault your program. In fact, I
think we should add a check for this, as e.g. the following code will
segfault in func() when it calls __Pyx_WriteUnraisable, as it will try
to get the exception from a NULL thread-state.

from cpython.ref cimport PyObject
from libc.stdio cimport printf

cdef PyObject *func2() except NULL with gil:
raise Exception("blah")

cdef void func() nogil:
cdef PyObject *p = func2()
printf("This is not printed!\n")

func()

Of course, you cannot really declare the return type of func2 as
'object', because you wouldn't be able to call it from 'nogil'
sections that way.

I guess my point is, if the user isn't very careful about the way
exceptions are handled, undesired behaviour may occur. With both
functions and with gil blocks, the user will have to override the
default behaviour if needed. I think "python semantics" most
accurately reflect the syntax for 'with gil:' blocks.

>> For functions declared 'nogil' that contain 'with gil:' statements, it
>> will safely lock around around the initialization of any Python
>> objects and set up the refnanny context (with appropriate preprocessor
>> guards). At the end of the function it will safely lock around the
>> teardown of the refnanny context. With 'safely' I mean that it will
>> create a thread state if it was not already created and may be called
>> even while the GIL is already held (using PyGILState_Ensure()). This
>> means variables are declared and initialized in the same way as in
>> normal GIL-holding functions (except that there is additional
>> locking), and of course the GIL-checking code ensures that errors are
>> issued if those variables are attempted to be used outside any GIL
>> blocks.
>
> I find that surprising semantics. So the GIL will always be acquired at
> least twice in the following example, regardless of the input?
>
>    cdef int callme(bint flag) nogil:
>         if flag:
>             with gil: x = object()
>

Yes. It should be noted to users that 'with gil:' does not come
without its repercussions.

>> Could someone review the patch (which is attached)? Maybe check if I
>> haven't missed any side cases and such?
>
> From a first look, the test file you added seems far too short. I would
> expect that this feature requires a lot more testing in combination with
> declared and undeclared local variables, type inference, several exception
> raising and catching situations (e.g. raise 

Re: [Cython] 'with gil:' statement

[mark florisson]

>>> Could someone review the patch (which is attached)? Maybe check if I
>>> haven't missed any side cases and such?
>>
>> From a first look, the test file you added seems far too short. I would
>> expect that this feature requires a lot more testing in combination with
>> declared and undeclared local variables, type inference, several exception
>> raising and catching situations (e.g. raise in one block, catch in an outer
>> block, try-finally, ...) or looping. It may also have an impact on Vitja's
>> control flow analysis branch that's worth considering.
>
> I agree. I think I'll start a branch and work on some more tests.

I also have to adjust the function teardown locking. I'll start a
branch and report back if I get positive results.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 17 March 2011 10:08, Dag Sverre Seljebotn  wrote:
> On 03/17/2011 09:27 AM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 17.03.2011 08:38:
>>>
>>> On 03/17/2011 12:24 AM, Greg Ewing wrote:

 Stefan Behnel wrote:

> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
> handle exceptions, so simply jumping out of them because an inner 'with
> gil' block raised an exception can have unexpected side effects.

 Seems to me that the __Pyx_WriteUnraisable should be done at
 the end of the 'with gil' block, and execution should then
 continue from there.

 In other words, the effect on exception handling should be
 the same as if the 'with gil' block had been factored out into
 a separate function having no exception return value.

>>>
>>> -1.
>>>
>>> I consider the fact that exceptions don't propagate from some functions a
>>> "currently unfixable bug". We should plan for it being fixed some day.
>>
>> It can't be fixed in general, because there are cases where exceptions
>> simply cannot be propagated. Think of C callbacks, for example. C doesn't
>> have a "normal" way of dealing with exceptions, so if an exception that
>> originated from a callback simply leads to returning from the function, it
>> may mean that the outer C code will simply continue to execute normally.
>> Nothing's won in that case.
>
> Yes, that's a good point. (This is what I used setjmp/longjmp to work around
> BTW, to longjmp across the calling Fortran code. I knew it wasn't doing any
> mallocs/frees, let alone any file handling etc., so this was safe.)
>
> I'll admit that I'm mostly focused on code like
>
> def f():
>    with nogil:
>        for ...:
>            A
>            if something_exceptional:
>                with gil:
>                    raise Exception(...)
>            B
>        C
>
> where I'd say it's up to me to make sure that B and C can safely be skipped.
> It would be a major pain to have my raised exception here be "trapped" -- in
> fact, it would make the "with gil" statement unusable for my purposes.
>
>
>
>>
>> In code:
>>
>>    cdef void c_callback(...) nogil:
>>        ... do some C stuff ...
>>        with gil:
>>            ... do some Python stuff ...
>>        ... do some more C stuff ...
>>
>> So far, there are two proposed ways of doing this.
>>
>> 1) acquire the GIL on entry and exit, handling unraisable exceptions right
>> before exiting.
>>
>> 2) keep all GIL requiring code inside of the "with gil" block, including
>> unraisable exceptions.
>>
>> I find (2) a *lot* more intuitive, as well as much safer. We can't know
>> what effects the surrounding "do C stuff" code has. It may contain
>> thread-safe C level cleanup code for the "with gil" block, for example, or
>> preparation code that enables returning into the calling C code. Simply
>> jumping out of the GIL block without executing the trailing code may simply
>> not work at all.
>
> I think you find (2) more intuitive because you have a very detailed
> knowledge of Cython and CPython, but that somebody new to Cython would
> expect a "with" statement to have the same control flow logic as the Python
> with statement. Of course, I don't have any data for that.
>
> How about this compromise: We balk on the code you wrote with:
>
> Error line 345: Exceptions propagating from "with gil" block cannot be
> propagated out of function, please insert try/except and handle exception
>
> So that we require this:
>
> with gil:
>    try:
>        ...
>    except:
>        warnings.warning(...) # or even cython.unraisable(e)
>
> This keeps me happy about not abusing the with statement for strange control
> flow, and makes the "with gil" useful for raising exceptions inside regular
> def functions with nogil blocks.
>

I agree with your previous statement, but not with your compromise :).
We have to differentiate between two cases, similar to Stefan's cases,
but different in a very important way that matter for nested GIL
blocks.

1) Exceptions can propagate to some outer GIL section (in or outside
the current function)
2) Exceptions can't propagate, because there is no outer GIL section
and the function has a non-object return type

With your compromise, with 1) exceptions cannot propagate, but with 2)
you win forcing the user to be explicit. But then you still need to
write to some variable indicating that an exception occurred and
adjust control flow accordingly in your nogil section (unless you want
to clean up and return immediately).

If you have Python with-statement semantics, you can do the following,
for instance:

cdef void func() nogil:
with gil:
try:

with nogil:
with gil:
code that may raise an exception

this is not executed

except ExceptionRaisedFromInnerWithGilBlock:
handle exception here

The point is, if you have case 2), and you want to use GIL code, you
need to h

Re: [Cython] 'with gil:' statement

[mark florisson]

On 17 March 2011 09:27, Stefan Behnel  wrote:
> Dag Sverre Seljebotn, 17.03.2011 08:38:
>>
>> On 03/17/2011 12:24 AM, Greg Ewing wrote:
>>>
>>> Stefan Behnel wrote:
>>>
 I'm not sure if this is a good idea. "nogil" blocks don't have a way to
 handle exceptions, so simply jumping out of them because an inner 'with
 gil' block raised an exception can have unexpected side effects.
>>>
>>> Seems to me that the __Pyx_WriteUnraisable should be done at
>>> the end of the 'with gil' block, and execution should then
>>> continue from there.
>>>
>>> In other words, the effect on exception handling should be
>>> the same as if the 'with gil' block had been factored out into
>>> a separate function having no exception return value.
>>>
>>
>> -1.
>>
>> I consider the fact that exceptions don't propagate from some functions a
>> "currently unfixable bug". We should plan for it being fixed some day.
>
> It can't be fixed in general, because there are cases where exceptions
> simply cannot be propagated. Think of C callbacks, for example. C doesn't
> have a "normal" way of dealing with exceptions, so if an exception that
> originated from a callback simply leads to returning from the function, it
> may mean that the outer C code will simply continue to execute normally.
> Nothing's won in that case.
>
> In code:
>
>    cdef void c_callback(...) nogil:
>        ... do some C stuff ...
>        with gil:
>            ... do some Python stuff ...
>        ... do some more C stuff ...
>
> So far, there are two proposed ways of doing this.
>
> 1) acquire the GIL on entry and exit, handling unraisable exceptions right
> before exiting.
>
> 2) keep all GIL requiring code inside of the "with gil" block, including
> unraisable exceptions.
>
> I find (2) a *lot* more intuitive, as well as much safer. We can't know what
> effects the surrounding "do C stuff" code has. It may contain thread-safe C
> level cleanup code for the "with gil" block, for example, or preparation
> code that enables returning into the calling C code. Simply jumping out of
> the GIL block without executing the trailing code may simply not work at
> all.

Which is exactly why users will have to handle exceptions if they want
their C code to execute.

>
>> We could perhaps fix exception propagation from nogil functions by using
>> some conventions + setjmp/longjmp. Mono does this when calling into native
>> code, and I recently did it manually in Cython to propagate exceptions
>> through the Fortran wrappers in SciPy.
>
> Regardless of the topic of this thread, it would be nice to have longjmp
> support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work
> around several quirks in that area.
>
>
>> Also, the GIL may not be around
>> forever even in CPython? (All arguments I've seen for keeping it has been
>> along the lines of "it slows down serial code", not that it is considered
>> a
>> good thing.)
>
> If it ever gets removed, there will surely have to be an emulation layer for
> C modules. Many of them simply use it as thread-lock, and that's totally
> reasonable IMHO.
>
>
>> Designing a language around the GIL feels like a dead-end to me.
>
> We keep having diverging opinions about the GIL. I like it, and I keep
> repeating myself by saying that "threading should be explicit". Having a way
> to lock the whole interpreter and to keep parallel execution and reentry
> points to well defined places in your code is a great feature.
>
>
>> I'm OK
>> with being practical in the face of the limitations of today; but let's
>> keep "with gil" and "with nogil" something that can become noops in the
>> future without too much pain. Yes, I know that if the GIL goes it will
>> break Stefan's lxml code, and I'm sure other code -- I'm just saying that
>> we shouldn't make the language design even more GIL-centric than it
>> already
>> is.
>
> It's not. Even a removal of the GIL won't remove the fact that C can't
> propagate exceptions.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 17 March 2011 11:35, Dag Sverre Seljebotn  wrote:
> On 03/17/2011 11:16 AM, mark florisson wrote:
>>
>> On 17 March 2011 10:08, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> How about this compromise: We balk on the code you wrote with:
>>>
>>> Error line 345: Exceptions propagating from "with gil" block cannot be
>>> propagated out of function, please insert try/except and handle exception
>>>
>>> So that we require this:
>>>
>>> with gil:
>>>    try:
>>>        ...
>>>    except:
>>>        warnings.warning(...) # or even cython.unraisable(e)
>>>
>>> This keeps me happy about not abusing the with statement for strange
>>> control
>>> flow, and makes the "with gil" useful for raising exceptions inside
>>> regular
>>> def functions with nogil blocks.
>>>
>> I agree with your previous statement, but not with your compromise :).
>> We have to differentiate between two cases, similar to Stefan's cases,
>> but different in a very important way that matter for nested GIL
>> blocks.
>>
>> 1) Exceptions can propagate to some outer GIL section (in or outside
>> the current function)
>> 2) Exceptions can't propagate, because there is no outer GIL section
>> and the function has a non-object return type
>>
>> With your compromise, with 1) exceptions cannot propagate, but with 2)
>> you win forcing the user to be explicit. But then you still need to
>> write to some variable indicating that an exception occurred and
>> adjust control flow accordingly in your nogil section (unless you want
>> to clean up and return immediately).
>>
>> If you have Python with-statement semantics, you can do the following,
>> for instance:
>>
>> cdef void func() nogil:
>>     with gil:
>>         try:
>>
>>             with nogil:
>>                 with gil:
>>                     code that may raise an exception
>>
>>                 this is not executed
>>
>>         except ExceptionRaisedFromInnerWithGilBlock:
>>             handle exception here
>>
>> The point is, if you have case 2), and you want to use GIL code, you
>> need to handle exceptions in some way. Forcing the user to not
>> propagate anything doesn't sound right, unless this holds only for the
>> outermost 'with gil' block. I would be OK with that, although it would
>> be inconsistent with how exceptions in normal cdef functions with
>> non-object return work, so I would say that we'd have to force it in
>> the same manner there.
>
> I think we should perhaps look at enforcing explicit exception-ignoring
> everywhere.. there's a lot of details to hash out, and there's the issue of
> backwards compatability, but it could be dealt with with a couple of
> releases where we only raise a warning and so on.
>
> It could involve a *very* limited subset of exception handling for use in
> nogil mode (i.e., only a bare "except:" statement allowed, where one can
> call either "cython.unraisable()", "pass", or set a flag).

I don't think we should allow the forced exception handling in nogil
sections, but in gil sections (and you'd have to hold the GIL as
exceptions are stored on the thread state). So forced exception
handling for functions declared 'with gil' and for outermost 'with
gil' blocks in cdef functions with non-object return.

> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 18 March 2011 00:32, Greg Ewing  wrote:
> Dag Sverre Seljebotn wrote:
>
>> def f():
>>    with nogil:
>>        for ...:
>>            A
>>            if something_exceptional:
>>                with gil:
>>                    raise Exception(...)
>>            B
>>        C
>
> If that's to be supported, the following really ought to be
> supported as well:
>
>  def f():
>     with nogil:
>        try:
>           ...
>           with gil:
>              raise Exception()
>        finally:
>           ...do some cleanup...
>

Why? I assume in his example the for loop was a Cython C for loop, not
one that deals with Python objects. If you want to do cleanup you can
catch the exception in the 'with gil:' block, or use try/finally in
the 'with gil:' block or outside the 'with nogil:' block.

Special-casing try/finally in nogil sections for only 'with gil:'
sounds somewhat weird. On the other hand, it may be somewhat more
convenient, and I think we could support it without having to acquire
the GIL in the finally clause. I wouldn't be particularly opposed to
that.

> Greg
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 16 March 2011 20:16, mark florisson  wrote:
>>>> Could someone review the patch (which is attached)? Maybe check if I
>>>> haven't missed any side cases and such?
>>>
>>> From a first look, the test file you added seems far too short. I would
>>> expect that this feature requires a lot more testing in combination with
>>> declared and undeclared local variables, type inference, several exception
>>> raising and catching situations (e.g. raise in one block, catch in an outer
>>> block, try-finally, ...) or looping. It may also have an impact on Vitja's
>>> control flow analysis branch that's worth considering.
>>
>> I agree. I think I'll start a branch and work on some more tests.
>
> I also have to adjust the function teardown locking. I'll start a
> branch and report back if I get positive results.
>

I added more tests, the code can be found in this fork:
https://github.com/markflorisson88/cython .
There is currently no compile-time checking for exceptions that might
be swallowed, it still works in the same way as normal cdef functions
with a non-object return type. Should we issue warnings for such cases
instead of relying on the implicit swallow-and-print, as Dag
suggested?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 18 March 2011 07:07, Stefan Behnel  wrote:
> Greg Ewing, 18.03.2011 01:18:
>>
>> mark florisson wrote:
>>>
>>> I think we could support it without having to acquire
>>> the GIL in the finally clause.
>>
>> That was the intention -- the code in the finally clause would
>> be subject to the same nogil restrictions as the rest of
>> the nogil block.
>>
>> My point is that as long as you're allowing exceptions to be
>> tunnelled through nogil blocks, they should respect any finally
>> clauses that they pass through on the way.
>
> +1

Ok, I will give it a go and try to allow it when they surround with
gil blocks. I would however like to reiterate that it is a
special-case, inconsistent with previous behaviour, and basically
extends the language and won't work for functions that are called and
declared 'with gil'. But it is convenient, so I can't help but like it
at the same time :]

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[mark florisson]

On 18 March 2011 13:36, Stefan Behnel  wrote:
> Dag Sverre Seljebotn, 18.03.2011 12:07:
>>
>> On 03/18/2011 11:10 AM, Stefan Behnel wrote:
>>>
>>> Actually, I think I still find it more convenient to not provide *any*
>>> special exception paths through nogil code, i.e. to not let exceptions in
>>> "with gil" blocks exit from outer "nogil" blocks. That would avoid all of
>>> the semantic difficulties above.
>>
>> Well, of course not supporting something is easier.
>
> That's a bit exaggerated, IMHO. I would expect that there'd be many cases
> where the right thing to do in case of an exception is to return from a
> function, with a bit of explicit cleanup. So many exceptions won't leave the
> "with gil" block anyways. And few users will actually use nested "with gil"
> and "nogil" blocks.

This also holds for cdef nogil functions with 'with gil' blocks.

> But still, there's the question how code that re-acquires the GIL should
> behave (still thinking of a callback case here). Should it check for a
> raised exception and jump into the exception handling path? Or should it
> ignore it and continue until something tests for the exception explicitly?

I'm not sure what you mean, are you talking about exceptions raised in
the with gil block, or about exceptions that were pending before the
function was called? In the latter, case, my answer would be, "in the
same way that functions declared 'with gil' work": you assume there is
no pending exception. It doesn't make sense to call Python-related
code with an exception set. And indeed, the finally clause works by
first clearing and storing the exception on the stack, and then
restoring it afterwards if no other exception propagated. This is not
a case we should strife to handle, it's incorrect code.

> And what if intermediately running nogil code changed some state somewhere
> that breaks the exception handling path?

If users call into C code that raises an exception, it is their task
to take care of any exceptions manually, or by using the right Cython
declarations (such as the 'except' declaration clause for cdef
functions). This is not any different from any other semantics we
currently have for gil sections.

> Or what if the code block that
> temporarily re-acquires the GIL (e.g. a logging handler) is completely
> unrelated to the one that actually raised the exception or the one that
> called into the whole machinery (and that would be the right one to handle
> it)? These things are not trivial, neither for Cython's language semantics
> nor for users.

If this happens in a with gil section in the finally clause of a nogil
section, then I think they should be chained (in python 3). In python
2 the exception should simply be replaced, as per the semantics of
finally.

> We shouldn't forget that basically all Python operations can at least raise
> a MemoryError or a KeyboardInterrupt etc. Most users won't think of these
> cases. I think it would help users in writing safer code if the need to
> handle exceptions in nogil blocks was always made explicit by the compiler.
> That's different from "not supporting something".
>
>> Relying on boolean flags to signal errors states is a real pain
>> when one is used to using exceptions.
>
> Well, if those exception do not take safe and obvious paths, it may still be
> better to use explicit boolean flags instead.

Again, I think "Python semantics" are quite obvious for exception
paths from with gil statements. If the user really needs to do
cleanup, exceptions may be caught in the with gil block and an error
flag used, or the cleanup can be moved to the finally clause in the
gil section. Without compiler enforcement, the user will forget the
try/finally around the gil section as easily as the try/except or
try/finally around the body of the with gil section. I don't find
enforcing it useful at all, though. The only thing that would be
useful is enforcing the user to handle exceptions when they might be
unraisable due to the function's return type.

> Specifically, I'm fine with letting exceptions flow through nogil code iff
> there is a way to react on exceptional exit paths inside of nogil blocks.
> I'm against enabling "except" clauses in nogil blocks, but I think a
> try-finally would catch the most important use cases iff we can figure out
> clean semantics for this, including the cases I mentioned previously. That's
> the condition I see.

To summarize, I think the semantics should be like this:
- propagate exceptions from with gil blocks through nogil sections
- if there is a finally clause in a nogil section corresponding to a
try surrounding a with gil block, execute it
- if the finally clause has with gil blocks, allow them to raise
exceptions, thereby overriding any pending exception, using the
semantics just mentioned (applied recursively)

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] PyCharm

[mark florisson]

On 22 March 2011 15:33, Jason Grout  wrote:
> On 3/16/11 5:33 AM, mark florisson wrote:
>>
>> The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the
>> Cython project a free Open Source license, which means that anyone
>> developing Cython may freely use PyCharm to develop Cython. They
>> prefer to license to remain unpublic, so if you develop Cython and
>> want a free PyCharm license, send me an email and I will send you the
>> license key. It is valid until 13 March 2012.
>
>
> Is it the offer that is valid for a year, or is it the license that is valid
> for only a year?

The license, after that we have to renew it.

> Thanks,
>
> Jason
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] cygdb gdb script problem

[mark florisson]

On 29 March 2011 21:11, Vitja Makarov  wrote:
> Running cygdb with Cython installed in the system leads to the
> following problem:
>
> vitja@vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py
> GNU gdb (GDB) 7.2-ubuntu
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> .
> Traceback (most recent call last):
>  File "", line 1, in 
> ImportError: No module named Debugger
> /tmp/tmp1ZvOf9:3: Error in sourced command file:
> Error while executing Python code.
> (gdb)
>
> So may be it's better to explicitly specify correct path to Cython
> package in gdb script?
>
> Small example patch is attached.
>
> --
> vitja.
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

Hmm, when using cygdb it expects Cython to be in the path. I believe
your traceback means that it can find Cython, but not Cython.Debugger,
right? Are you sure you're using Python 2.6+? It doesn't work with any
version below 2.6 as they don't use the TP_FLAGS_*_SUBCLASS tp_flags.

In your patch you probably want to join() the dirname() with
os.pardir. However, I don't think it would really solve anything,
because if you install Cython, cygdb will be installed in e.g.
/usr/bin, so you'd be adding / to the path. If you want to run cygdb
from the cython/bin directory, then cython should be listed in
PYTHONPATH.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] cygdb gdb script problem

[mark florisson]

On 29 March 2011 22:20, Vitja Makarov  wrote:
> 2011/3/29 mark florisson :
>> On 29 March 2011 21:11, Vitja Makarov  wrote:
>>> Running cygdb with Cython installed in the system leads to the
>>> following problem:
>>>
>>> vitja@vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py
>>> GNU gdb (GDB) 7.2-ubuntu
>>> Copyright (C) 2010 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later 
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>.
>>> Traceback (most recent call last):
>>>  File "", line 1, in 
>>> ImportError: No module named Debugger
>>> /tmp/tmp1ZvOf9:3: Error in sourced command file:
>>> Error while executing Python code.
>>> (gdb)
>>>
>>> So may be it's better to explicitly specify correct path to Cython
>>> package in gdb script?
>>>
>>> Small example patch is attached.
>>>
>>> --
>>> vitja.
>>>
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>
>> Hmm, when using cygdb it expects Cython to be in the path. I believe
>> your traceback means that it can find Cython, but not Cython.Debugger,
>> right? Are you sure you're using Python 2.6+? It doesn't work with any
>> version below 2.6 as they don't use the TP_FLAGS_*_SUBCLASS tp_flags.
>>
>
> That's correct when cygdb is executed it's executed with right cython version,
> and gdb import system version of cython
>
>> In your patch you probably want to join() the dirname() with
>> os.pardir. However, I don't think it would really solve anything,
>> because if you install Cython, cygdb will be installed in e.g.
>> /usr/bin, so you'd be adding / to the path.
>
> No it will add /usr/lib to path, path is taken from Cython.Debugger.Cygdb
Ah indeed, that code was moved.

>> If you want to run cygdb
>> from the cython/bin directory, then cython should be listed in
>> PYTHONPATH.
>
> Ok. But you may get in trouble importing system version instead of
> development one.

Hmm, PYTHONPATH should come before the default search path. But it is
true that those eggs can do anything, like insert themselves at the
front of the path by putting python code in those .pth files (a gross
hack). So indeed, it sounds like a good idea to insert that path at
the front of sys.path.

> --
> vitja.
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP: prange for parallel loops

[mark florisson]

On 4 April 2011 13:53, Dag Sverre Seljebotn  wrote:
> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>
>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>
>>> CEP up at http://wiki.cython.org/enhancements/prange
>>
>> """
>> Variable handling
>>
>> Rather than explicit declaration of shared/private variables we rely on
>> conventions:
>>
>>    * Thread-shared: Variables that are only read and not written in the
>> loop body are shared across threads. Variables that are only used in the
>> else block are considered shared as well.
>>
>>    * Thread-private: Variables that are assigned to in the loop body are
>> thread-private. Obviously, the iteration counter is thread-private as well.
>>
>>    * Reduction: Variables that only used on the LHS of an inplace
>> operator, such as s above, are marked as targets for reduction. If the
>> variable is also used in other ways (LHS of assignment or in an expression)
>> it does instead turn into a thread-private variable. Note: This means that
>> if one, e.g., inserts printf(... s) above, s is turned into a thread-local
>> variable. OTOH, there is simply no way to correctly emulate the effect
>> printf(... s) would have in a sequential loop, so such code must be
>> discouraged anyway.
>> """
>>
>> What about simply (ab-)using Python semantics and creating a new inner
>> scope for the prange loop body? That would basically make the loop behave
>> like a closure function, but with the looping header at the 'right' place
>> rather than after the closure.
>
> I'm not quite sure what the concrete changes to the CEP this would lead to
> (assuming you mean this as a proposal for alternative semantics, and not an
> implementation detail).
>
> How would we treat reduction variables? They need to be supported, and
> there's nothing in Python semantics to support reduction variables, they are
> a rather special case everywhere. I suppose keeping the reduction clause
> above, or use the "nonlocal" keyword in the loop body...
>
> Also there's the else:-block, although we could make that part of the scope.
> And the "lastprivate" functionality, although that could be dropped without
> much loss.
>
>>
>> Also, in the example, the local variable declaration of "tmp" outside of
>> the loop looks somewhat misplaced, although it's precedented by
>> comprehensions (which also have their own local scope in Cython).
>
> Well, depending on the decision of lastprivate, the declaration would need
> to be outside; I really like the idea of moving "cdef", and am prepared to
> drop lastprivate for this.
>
> Being explicit about thread-local variables does make things a lot safer to
> use.
>
> (One problem is that switching between serial and parallel one needs to move
> variable declarations. But that only happens once, and one can use
> "nthreads=1" to disable parallel after that.)
>
> An example would then be:
>
> def f(np.ndarray[double] x, double alpha):
>    cdef double s = 0, globtmp
>    with nogil:
>        for i in prange(x.shape[0]):
>            cdef double tmp # thread-private
>            tmp = alpha * i # alpha available from global scope
>            s += x[i] * tmp # still automatic reduction for inplace operators
>            # printf(...s) -> now leads to error, since s is not declared
> thread-private but is read
>        else:
>            # tmp still available here...looks a bit strange, but useful
>            s += tmp * 10
>            globtmp = tmp # we save tmp for later
>        # tmp not available here, globtmp is
>    return s
>
> Or, we just drop support for the else block on these loops.

I think since we are disallowing break (yet) we shouldn't support the
else clause. Basically, I think we can make the CEP a tad more simple.

I think we could declare everything outside of the prange body. Then,
in the prange loop body:

if a variable is assigned to anywhere -> make it lastprivate
- if a variable is read before assigned to -> make it
firstprivate in addition to lastprivate (raise compiler error if the
variable is not initialized outside of the loop body)

if a variable is only ever read -> make it shared (the default for OpenMP)

if a variable has an inplace operator -> make it a reduction

There is really no reason to disallow reading of the reduction
variable (in e.g. a printf). The reduction should also be initialized
outside of the prange body.

Then prange() could be implemented in pure mode as simply the
sequential version, i.e. range() which some more arguments.

For any scratch space buffers etc, I'd prefer something like


with cython.parallel:
cdef char *buf = malloc(100)

for i in prange(n):
use buf

free(buf)

At least it fits my brain pretty well :) (this code does however
assume that malloc is thread-safe).

Anyway, I'm not sure I just covered all cases, but what do you think?

> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/

Re: [Cython] CEP: prange for parallel loops

[mark florisson]

On 4 April 2011 19:18, Dag Sverre Seljebotn  wrote:
> On 04/04/2011 05:22 PM, mark florisson wrote:
>>
>> On 4 April 2011 13:53, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 04/04/2011 01:23 PM, Stefan Behnel wrote:
>>>>
>>>> Dag Sverre Seljebotn, 04.04.2011 12:17:
>>>>>
>>>>> CEP up at http://wiki.cython.org/enhancements/prange
>>>>
>>>> """
>>>> Variable handling
>>>>
>>>> Rather than explicit declaration of shared/private variables we rely on
>>>> conventions:
>>>>
>>>>    * Thread-shared: Variables that are only read and not written in the
>>>> loop body are shared across threads. Variables that are only used in the
>>>> else block are considered shared as well.
>>>>
>>>>    * Thread-private: Variables that are assigned to in the loop body are
>>>> thread-private. Obviously, the iteration counter is thread-private as
>>>> well.
>>>>
>>>>    * Reduction: Variables that only used on the LHS of an inplace
>>>> operator, such as s above, are marked as targets for reduction. If the
>>>> variable is also used in other ways (LHS of assignment or in an
>>>> expression)
>>>> it does instead turn into a thread-private variable. Note: This means
>>>> that
>>>> if one, e.g., inserts printf(... s) above, s is turned into a
>>>> thread-local
>>>> variable. OTOH, there is simply no way to correctly emulate the effect
>>>> printf(... s) would have in a sequential loop, so such code must be
>>>> discouraged anyway.
>>>> """
>>>>
>>>> What about simply (ab-)using Python semantics and creating a new inner
>>>> scope for the prange loop body? That would basically make the loop
>>>> behave
>>>> like a closure function, but with the looping header at the 'right'
>>>> place
>>>> rather than after the closure.
>>>
>>> I'm not quite sure what the concrete changes to the CEP this would lead
>>> to
>>> (assuming you mean this as a proposal for alternative semantics, and not
>>> an
>>> implementation detail).
>>>
>>> How would we treat reduction variables? They need to be supported, and
>>> there's nothing in Python semantics to support reduction variables, they
>>> are
>>> a rather special case everywhere. I suppose keeping the reduction clause
>>> above, or use the "nonlocal" keyword in the loop body...
>>>
>>> Also there's the else:-block, although we could make that part of the
>>> scope.
>>> And the "lastprivate" functionality, although that could be dropped
>>> without
>>> much loss.
>>>
>>>> Also, in the example, the local variable declaration of "tmp" outside of
>>>> the loop looks somewhat misplaced, although it's precedented by
>>>> comprehensions (which also have their own local scope in Cython).
>>>
>>> Well, depending on the decision of lastprivate, the declaration would
>>> need
>>> to be outside; I really like the idea of moving "cdef", and am prepared
>>> to
>>> drop lastprivate for this.
>>>
>>> Being explicit about thread-local variables does make things a lot safer
>>> to
>>> use.
>>>
>>> (One problem is that switching between serial and parallel one needs to
>>> move
>>> variable declarations. But that only happens once, and one can use
>>> "nthreads=1" to disable parallel after that.)
>>>
>>> An example would then be:
>>>
>>> def f(np.ndarray[double] x, double alpha):
>>>    cdef double s = 0, globtmp
>>>    with nogil:
>>>        for i in prange(x.shape[0]):
>>>            cdef double tmp # thread-private
>>>            tmp = alpha * i # alpha available from global scope
>>>            s += x[i] * tmp # still automatic reduction for inplace
>>> operators
>>>            # printf(...s) ->  now leads to error, since s is not declared
>>> thread-private but is read
>>>        else:
>>>            # tmp still available here...looks a bit strange, but useful
>>>            s += tmp * 10
>>>            globtmp = tmp # we save tmp for later
>>>        # tmp not available here, globtmp is
>>>    return s

Re: [Cython] Another CEP: Parallel block

[mark florisson]

On 5 April 2011 09:21, Dag Sverre Seljebotn  wrote:
> There's a (much shorter) proposal for a more explicit parallelism construct
> at
>
> http://wiki.cython.org/enhancements/parallelblock
>
> This is a little more verbose for the simplest case, but makes the
> medium-cases that needs work buffers much simpler, and is also more explicit
> and difficult to get wrong.

I actually think your else block really complicates matters. In this
example even your index variable is not well-defined right after the
loop, because it's not "declared lastprivate through the else block".
There is really no reason to make variables private instead of
lastprivate (and additionally firstprivate if needed) by default.

I think we should allow at least both options, so if the variables are
declared in the parallel nogil block they can only be used inside that
block (but are still lastprivate, as the first loop may be followed by
other code). But the user will also still be able to declare and
define stuff outside of the block and omit the with parallel block
entirely.

And again, you will want something like cython.parallel.range instead
of just range, as you will want to pass scheduling parameters to the
range(), and not the parallel.

So e.g. you can still write something like this:

cdef Py_ssize_t i
for i in cython.parallel.range(..., schedule='dynamic', nogil=True):
do something

print i # i is well-defined here

My point is, implicit first- and lastprivate can be implicit because
it works the exact same way as the sequential python version does. The
only remaining pitfall is the in-place operator which declares a
reduction.

> I am not sure myself which one I prefer of this and prange.
>
> Justification for Cython-specific syntax: This is something that is really
> only useful if you can release the GIL *outside* of the loop. So I feel this
> is an area where a custom Cython solution is natural, sort of like "cdef
> extern", and the buffer access.
>
> Since a similar pure-Python solution is rather useless, I also think there's
> less incentive for making something that works well in pure-Python mode.

Which feature is Cython specific here? The 'with a, b as c:' thing?

> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Another CEP: Parallel block

[mark florisson]

On 5 April 2011 10:34, Stefan Behnel  wrote:
> mark florisson, 05.04.2011 10:26:
>>
>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>
>>> Justification for Cython-specific syntax: This is something that is
>>> really
>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>> this
>>> is an area where a custom Cython solution is natural, sort of like "cdef
>>> extern", and the buffer access.
>>>
>>> Since a similar pure-Python solution is rather useless, I also think
>>> there's
>>> less incentive for making something that works well in pure-Python mode.
>>
>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>
> No, the syntax is just Python. It's the scoping that's Cython specific,
> including the local variable declarations inside of the "with" block.

Hmm, but you can use cython.declare() for that, no?

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Another CEP: Parallel block

[mark florisson]

On 5 April 2011 10:44, mark florisson  wrote:
> On 5 April 2011 10:34, Stefan Behnel  wrote:
>> mark florisson, 05.04.2011 10:26:
>>>
>>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>>
>>>> Justification for Cython-specific syntax: This is something that is
>>>> really
>>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>>> this
>>>> is an area where a custom Cython solution is natural, sort of like "cdef
>>>> extern", and the buffer access.
>>>>
>>>> Since a similar pure-Python solution is rather useless, I also think
>>>> there's
>>>> less incentive for making something that works well in pure-Python mode.
>>>
>>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>>
>> No, the syntax is just Python. It's the scoping that's Cython specific,
>> including the local variable declarations inside of the "with" block.
>
> Hmm, but you can use cython.declare() for that, no?

(disregarding the malloc() and pointer arithmetic, of course :)

>> Stefan
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Another CEP: Parallel block

[mark florisson]

On 5 April 2011 11:01, Stefan Behnel  wrote:
> mark florisson, 05.04.2011 10:44:
>>
>> On 5 April 2011 10:34, Stefan Behnel wrote:
>>>
>>> mark florisson, 05.04.2011 10:26:
>>>>
>>>> On 5 April 2011 09:21, Dag Sverre Seljebotn wrote:
>>>>>
>>>>> Justification for Cython-specific syntax: This is something that is
>>>>> really
>>>>> only useful if you can release the GIL *outside* of the loop. So I feel
>>>>> this
>>>>> is an area where a custom Cython solution is natural, sort of like
>>>>> "cdef
>>>>> extern", and the buffer access.
>>>>>
>>>>> Since a similar pure-Python solution is rather useless, I also think
>>>>> there's
>>>>> less incentive for making something that works well in pure-Python
>>>>> mode.
>>>>
>>>> Which feature is Cython specific here? The 'with a, b as c:' thing?
>>>
>>> No, the syntax is just Python. It's the scoping that's Cython specific,
>>> including the local variable declarations inside of the "with" block.
>>
>> Hmm, but you can use cython.declare() for that, no?
>
> cython.declare() is a no-op (or just a plain assignment) in Python. But the
> thread-local scoping of these variables cannot be emulated in Python. So
> this would be a feature that cannot be used in pure Python mode, unlike
> closures.

Sure, but the Python version would just be serial, it wouldn't use
threads at all. That's the great thing about OpenMP's philosophy is
that it can be either serial or parallel, the only difference is
speed. If you want speed, use Cython.

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP: prange for parallel loops

[mark florisson]

On 5 April 2011 12:51, Stefan Behnel  wrote:
> mark florisson, 04.04.2011 21:26:
>>
>> For clarity, I'll add an example:
>>
>> def f(np.ndarray[double] x, double alpha):
>>     cdef double s = 0
>>     cdef double tmp = 2
>>     cdef double other = 6.6
>>
>>     with nogil:
>>         for i in prange(x.shape[0]):
>>             # reading 'tmp' makes it firstprivate in addition to
>> lastprivate
>>             # 'other' is only ever read, so it's shared
>>             printf("%lf %lf %lf\n", tmp, s, other)
>
> So, adding a printf() to your code can change the semantics of your
> variables? That sounds like a really bad design to me.

I agree, I think we should refrain from the firstprivate() entirely,
as it wouldn't have the same semantics as serial execution (as 'tmp'
would have the original value with parallel execution and the value
from previous iterations with serial execution). So basically we
should allow reading of private variables only after they are assigned
to in the loop body.

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP: prange for parallel loops

[mark florisson]

On 5 April 2011 14:55, Pauli Virtanen  wrote:
>
> Mon, 04 Apr 2011 21:26:34 +0200, mark florisson wrote:
> [clip]
> > For clarity, I'll add an example:
> [clip]
>
> How about making all the special declarations explicit? The automatic
> inference of variables has a problem in that a small change in a part of
> the code can have somewhat unintuitive non-local effects, as the private/
> shared/reduction status of the variable changes in the whole function
> scope (if Python scoping is retained).
>
> Like so with explicit declarations:
>
> def f(np.ndarray[double] x, double alpha):
>    cdef double alpha = 6.6
>    cdef char *ptr = something()
>
>    # Parallel variables are declared beforehand;
>    # the exact syntax could also be something else
>    cdef cython.parallel.private[int] tmp = 2, tmp2
>    cdef cython.parallel.reduction[int] s = 0
>
>    # Act like ordinary cdef outside prange(); in the prange they are
>    # firstprivate if initialized or written to outside the loop anywhere
>    # in the scope. Or, they could be firstprivate always, if this
>    # has a negligible performance impact.
>    tmp = 3

The problem with firstprivate() is that it doesn't give you the same
semantics as in the sequential version. That's why I think it would be
best to forget about firstprivate entirely and allow reading of
private variables only after they are assigned to in the loop body.

>
>    with nogil:
>        s = 9
>
>        for i in prange(x.shape[0]):
>            if cython.parallel.first_iteration(i):
>                # whatever initialization; Cython is in principle allowed
>                # to move this outside the loop, at least if it is
>                # the first thing here
>                pass

For this I prefer the aforementioned 'with cython.parallel:' block.

>
>            # tmp2 is not firstprivate, as it's not written to outside
>            # the loop body; also, it's also not lastprivate as it's not
>            # read outside the loop
>            tmp2 = 99
>
>            # Increment a private variable
>            tmp += 2*tmp
>
>            # Add stuff to reduction
>            s += alpha*i
>
>            # The following raise a compilation error -- the reduction
>            # variable cannot be assigned to, and can be only operated on
>            # with only a single reduction operation inside prange
>            s *= 9
>            s = 8

I think OpenMP allows arbitrary assignments and expressions to the
reduction variable, all the spec says "usually it will be of the form
'x = ...'".

>
>            # It can be read, however, provided openmp supports this
>            tmp = s
>
>            # Assignment to non-private variables causes a compile-time
>            # error; this avoids common mistakes, such as forgetting to
>            # declare the reduction variable.
>            alpha += 42
>            alpha123 = 9
>            ptr = 94
>
>            # These, however, need to be allowed:
>            # the users are on their own to make sure they don't clobber
>            # non-local variables
>            x[i] = 123
>            (ptr + i)[0] = 123
>            some_routine(x, ptr, i)

Indeed. They could be either shared or firstprivate (as the pointer
would be firstprivate, and not the entire array, unless it was
declared as a C array of certain size).

>        else:
>            # private variables are lastprivate if read outside the loop
>            foo = tmp
>
>        # The else: block can be added, but actually has no effect
>        # as it is always executed --- the code here could as well
>        # be written after the for loop
>        foo = tmp  # <- same result
>
>    with nogil:
>        # Suppose Cython allowed cdef inside blocks with usual scoping
>        # rules
>        cdef cython.parallel.reduction[double] r = 0
>
>        # the same variables can be used again in a second parallel loop
>        for i in prange(x.shape[0]):
>            r += 1.5
>            s -= i
>            tmp = 9
>
>        # also the iteration variable is available after the loop
>        count = i
>
>    # As per usual Cython scoping rules
>    return r, s
>
> What did I miss here? As far as I see, the above would have the same
> semantics and scoping as a single-threaded Python implementation.
>
> The only change required to make things parallel is replacing range() by
> prange() and adding the variable declarations.

Basically, I like your approach. It's only slightly more verbose as
the implicit way, as you need to declare the type of each variable
anyway.

I also still like the implicit way, 

Re: [Cython] CEP: prange for parallel loops

[mark florisson]

On 5 April 2011 15:10, Pauli Virtanen  wrote:
> Tue, 05 Apr 2011 12:55:36 +, Pauli Virtanen wrote:
> [clip]
>>             # Assignment to non-private variables causes a compile-time
>>             # error; this avoids common mistakes, such as forgetting to
>>             # declare the reduction variable.
>>             alpha += 42
>>             alpha123 = 9
>>             ptr = 94
>
> Actually, I'm not sure this is absolutely necessary -- life is tough,
> especially if you are programming in parallel, and there are limits to
> hand-holding.
>
> However, an explicit declaration could be added for turning the error off
> for the (rare) cases where this makes sense (e.g. setting a shared flag)
>
>        cdef cython.parallel.shared[double] some_flag

I think that unless we add support for critical, single or master
sections, or the atomic construct, we should also disallow assigning
to shared variables entirely.

> --
> Pauli Virtanen
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP: prange for parallel loops

[mark florisson]

On 5 April 2011 18:32, Dag Sverre Seljebotn  wrote:
> On 04/05/2011 05:26 PM, Robert Bradshaw wrote:
>>
>> On Tue, Apr 5, 2011 at 8:02 AM, Dag Sverre Seljebotn
>>   wrote:
>>>
>>> On 04/05/2011 04:58 PM, Dag Sverre Seljebotn wrote:
>>>>
>>>> On 04/05/2011 04:53 PM, Robert Bradshaw wrote:
>>>>>
>>>>> On Tue, Apr 5, 2011 at 3:51 AM, Stefan Behnel
>>>>>  wrote:
>>>>>>
>>>>>> mark florisson, 04.04.2011 21:26:
>>>>>>>
>>>>>>> For clarity, I'll add an example:
>>>>>>>
>>>>>>> def f(np.ndarray[double] x, double alpha):
>>>>>>>     cdef double s = 0
>>>>>>>     cdef double tmp = 2
>>>>>>>     cdef double other = 6.6
>>>>>>>
>>>>>>>     with nogil:
>>>>>>>         for i in prange(x.shape[0]):
>>>>>>>             # reading 'tmp' makes it firstprivate in addition to
>>>>>>> lastprivate
>>>>>>>             # 'other' is only ever read, so it's shared
>>>>>>>             printf("%lf %lf %lf\n", tmp, s, other)
>>>>>>
>>>>>> So, adding a printf() to your code can change the semantics of your
>>>>>> variables? That sounds like a really bad design to me.
>>>>>
>>>>> That's what I was thinking. Basically, if you do an inlace operation,
>>>>> then it's a reduction variable, no matter what else you do to it
>>>>> (including possibly a direct assignment, though we could make that a
>>>>> compile-time error).
>>>>
>>>> -1, I think that's too obscure. Not being able to use inplace operators
>>>> for certain variables will be at the very least be nagging.
>>
>> You could still use inplace operators to your hearts content--just
>> don't bother using the reduced variable outside the loop. (I guess I'm
>> assuming reducing a variable has negligible performance overhead,
>> which it should.) For the rare cases that you want the non-aggregated
>> private, make an assignment to another variable, or use non-inplace
>> operations.
>
> Ahh! Of course! With some control flow analysis we could even eliminate the
> reduction if the variable isn't used after the loop, although I agree the
> cost should be trivial.
>
>
>> Not being able to mix inplace operators might be an annoyance. We
>> could also allow explicit declarations, as per Pauli's suggestion, but
>> not require them. Essentially, as long as we have
>
> I think you should be able to mix them, but if you do a reduction doesn't
> happen. This is slightly uncomfortable, but I believe control flow analysis
> and disabling firstprivate can solve it, see below.
>
> I believe I'm back in the implicit-camp. And the CEP can probably be
> simplified a bit too, I'll try to do that tomorrow.
>
> Two things:
>
>  * It'd still be nice with something like a parallel block for thread
> setup/teardown rather than "if firstthreaditeration():". So, a prange for
> the 50% simplest cases, followed by a parallel-block for the next 30%.

Definitely, I think it could also make way for things such as sections
etc, but I'll bring that up later :)

>  * Control flow analysis can help us tight it up a bit: For loops where you
> actually depend on values of thread-private variables computed in the
> previous iteration (beyond reduction), it'd be nice to raise a warning
> unless the variable is explicitly declared thread-local or similar. There
> are uses for such variables but they'd be rather rare, and such a hint could
> be very helpful.
>
> I'm still not sure if we want firstprivate, even if we can do it. It'd be
> good to see a usecase for it. I'd rather have NaN and 0x7FFF personally,
> as relying on the firstprivate value is likely a bug -- yes, it makes the
> sequential case work, but that is exactly in the case where parallelizing
> the sequential case would be wrong!!

Yeah, I think if we go the implicit route then firstprivate might be
quite a surprise for users.

> Grepping through 3 lines of heavily OpenMP-ified Fortran code here
> there's no mention of firstprivate or lastprivate (although we certainly
> want lastprivate to align with the sequential case).
>
> Dag Sverre
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Basically I'm fine with either implicit or explicit, although I think
the explicit case would be easier to understand for people that have
used OpenMP. In either case it would be nice to give prange a 'nogil'
option.

So to be clear, when we assign to a variable it will be lastprivate,
and when we assign to the subscript of a variable we make that
variable shared (unless it is declared inside the parallel with
block), right?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.

[mark florisson]

On 8 April 2011 19:59, Arthur de Souza Ribeiro
 wrote:
> The moduels suggested for the first two milestones you think are ok?
> Best Regards..
> []s
> Arthur

You mention the 'dis' module, but isn't that one (and 'opcode' too)
entirely written in Python?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] GSoC Proposal for Supporting Parallelism, Fused Types and Typed Views on Memory

[mark florisson]

Hey,

My GSoC proposal can be found here:
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/markflorisson88/1
. It's about implementing the prange CEP (524), Fused Types (522) and
Typed Memory Views (517).

I really hope to participate this year, but due to time constraints I
may not find the required time to actually complete the GSoC, so next
week I will decide whether I'll chicken out or not.

Cheers,

Mark
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 5 April 2011 22:29, Dag Sverre Seljebotn  wrote:
> I've done a pretty major revision to the prange CEP, bringing in a lot of
> the feedback.
>
> Thread-private variables are now split in two cases:
>
>  i) The safe cases, which really require very little technical knowledge ->
> automatically inferred
>
>  ii) As an advanced feature, unsafe cases that requires some knowledge of
> threading -> must be explicitly declared
>
> I think this split simplifies things a great deal.

Can't we obsolete the declaration entirely by assigning to variables
that need to have firstprivate behaviour inside the with parallel
block? Basically in the same way the scratch space is used. The only
problem with that is that it won't be lastprivate, so the value will
be undefined after the parallel block (but not after the worksharing
loop).

cdef int myvariable

with nogil, parallel:
myvariable = 2
for i in prange(...):
use myvariable
maybe assign to myvariable

# myvariable is well-defined here

# myvariable is not well-defined here

If you still desperately want lastprivate behaviour you can simply
assign myvariable to another variable in the loop body.

> I'm rather excited over this now; this could turn out to be a really
> user-friendly and safe feature that would not only allow us to support
> OpenMP-like threading, but be more convenient to use in a range of common
> cases.
>
> http://wiki.cython.org/enhancements/prange
>
> Dag Sverre
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 11 April 2011 11:10, Dag Sverre Seljebotn  wrote:
> On 04/11/2011 10:45 AM, mark florisson wrote:
>>
>> On 5 April 2011 22:29, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>> the feedback.
>>>
>>> Thread-private variables are now split in two cases:
>>>
>>>  i) The safe cases, which really require very little technical knowledge
>>> ->
>>> automatically inferred
>>>
>>>  ii) As an advanced feature, unsafe cases that requires some knowledge of
>>> threading ->  must be explicitly declared
>>>
>>> I think this split simplifies things a great deal.
>>
>> Can't we obsolete the declaration entirely by assigning to variables
>> that need to have firstprivate behaviour inside the with parallel
>> block? Basically in the same way the scratch space is used. The only
>> problem with that is that it won't be lastprivate, so the value will
>> be undefined after the parallel block (but not after the worksharing
>> loop).
>>
>> cdef int myvariable
>>
>> with nogil, parallel:
>>     myvariable = 2
>>     for i in prange(...):
>>         use myvariable
>>         maybe assign to myvariable
>>
>>     # myvariable is well-defined here
>>
>> # myvariable is not well-defined here
>>
>> If you still desperately want lastprivate behaviour you can simply
>> assign myvariable to another variable in the loop body.
>
> I don't care about lastprivate, I don't think that is an issue, as you say.
>
> My problem with this is that it means going into an area where possibly
> tricky things are implicit rather than explicit. I also see this as a rather
> special case that will be seldomly used, and implicit behaviour is more
> difficult to justify because of that.

Indeed, I actually considered if we should support firstprivate at
all, as it's really about "being firstprivate and lastprivate".
Without any declaration, you can have firstprivate or lastprivate, but
not both :) So I agree that supporting such a (probably) uncommon case
is better left explicit. On the other hand it seems silly to have
support for such a weird case.

> (The other instance of thread-local variables I feel is still explicit: You
> use prange instead of range, which means that you declare that values
> created in the iteration does not leak to the next iteration. The rest is
> just optimization from there.)
>
> As Robert said in his recent talk: A lot of languages are easy to write. The
> advantage of Python is that it is easy to *read*. That's what I feel is
> wrong with the proposal above: An assignment to a variable changes the
> semantics of it. Granted, it happens in a way so that it will almost always
> be correct, but I feel that reading the code, I'd spend some extra cycles to
> go "ah, so this variable is thread-local and therefore its values survive
> across a loop iteration".
>
> If I even knew about the feature in the first place. In seeing
> "threadprivate" spelled out, it is either obvious what it means, or obvious
> that I should look up the docs.
>
> There's *a lot* of things that can be made implicit in a programming
> language; Python/Cython simply usually leans towards the explicit side.
>
> Oh, and we may want to support writable shared variables (and flush)
> eventually too, and the above doesn't easily differentiate there?

Right, everything is implicit. So I guess it'll be good to introduce
it anyway as you say, so we can later declare stuff shared with
similar syntax. I suppose that's the point where I'm convinced.

> That's just my opinion, I'm happy to be overruled here.
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 11 April 2011 12:08, Dag Sverre Seljebotn  wrote:
> On 04/11/2011 11:41 AM, mark florisson wrote:
>>
>> On 11 April 2011 11:10, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>
>>>> On 5 April 2011 22:29, Dag Sverre Seljebotn
>>>>  wrote:
>>>>>
>>>>> I've done a pretty major revision to the prange CEP, bringing in a lot
>>>>> of
>>>>> the feedback.
>>>>>
>>>>> Thread-private variables are now split in two cases:
>>>>>
>>>>>  i) The safe cases, which really require very little technical
>>>>> knowledge
>>>>> ->
>>>>> automatically inferred
>>>>>
>>>>>  ii) As an advanced feature, unsafe cases that requires some knowledge
>>>>> of
>>>>> threading ->    must be explicitly declared
>>>>>
>>>>> I think this split simplifies things a great deal.
>>>>
>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>> that need to have firstprivate behaviour inside the with parallel
>>>> block? Basically in the same way the scratch space is used. The only
>>>> problem with that is that it won't be lastprivate, so the value will
>>>> be undefined after the parallel block (but not after the worksharing
>>>> loop).
>>>>
>>>> cdef int myvariable
>>>>
>>>> with nogil, parallel:
>>>>     myvariable = 2
>>>>     for i in prange(...):
>>>>         use myvariable
>>>>         maybe assign to myvariable
>>>>
>>>>     # myvariable is well-defined here
>>>>
>>>> # myvariable is not well-defined here
>>>>
>>>> If you still desperately want lastprivate behaviour you can simply
>>>> assign myvariable to another variable in the loop body.
>>>
>>> I don't care about lastprivate, I don't think that is an issue, as you
>>> say.
>>>
>>> My problem with this is that it means going into an area where possibly
>>> tricky things are implicit rather than explicit. I also see this as a
>>> rather
>>> special case that will be seldomly used, and implicit behaviour is more
>>> difficult to justify because of that.
>>
>> Indeed, I actually considered if we should support firstprivate at
>> all, as it's really about "being firstprivate and lastprivate".
>> Without any declaration, you can have firstprivate or lastprivate, but
>> not both :) So I agree that supporting such a (probably) uncommon case
>> is better left explicit. On the other hand it seems silly to have
>> support for such a weird case.
>
> Well, I actually need to do the per-thread cache thing I described in the
> CEP in my own codes, so it's not *that* special; it'd be nice to support it.

You need 'old_ell' and 'alpha' after the loop?

> OTOH I *could* work around it by having an array of scalars
>
> cdef int[:] old_ell = int[:numthreads]()
>
> ...
>    if old_ell[threadid()] != ell: ...
>
>
> So I guess, it's at least on the bottom of list of priorities in that CEP.
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Test runner

[mark florisson]

Can we select tests in the tests directory selectively? I see the -T
or --ticket option, but it doens't seem to find the test tagged with #
ticket: .

I can select unit tests using python runtests.py
Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
thing for tests in the tests directory. Running the entire suite takes
rather long.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Test runner

[mark florisson]

On 11 April 2011 12:45, Stefan Behnel  wrote:
> mark florisson, 11.04.2011 12:26:
>>
>> Can we select tests in the tests directory selectively? I see the -T
>> or --ticket option, but it doens't seem to find the test tagged with #
>> ticket:.
>>
>> I can select unit tests using python runtests.py
>> Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
>> thing for tests in the tests directory. Running the entire suite takes
>> rather long.
>
> You can still select them by name using a regex, e.g.
>
>   runtests.py 'run\.empty_builtin_constructors'
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Great, thanks! I'll update the hackerguide wiki.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Test runner

[mark florisson]

On 11 April 2011 12:53, mark florisson  wrote:
> On 11 April 2011 12:45, Stefan Behnel  wrote:
>> mark florisson, 11.04.2011 12:26:
>>>
>>> Can we select tests in the tests directory selectively? I see the -T
>>> or --ticket option, but it doens't seem to find the test tagged with #
>>> ticket:.
>>>
>>> I can select unit tests using python runtests.py
>>> Cython.SubPackage.Tests.SomeTest, but I can't seem to do the same
>>> thing for tests in the tests directory. Running the entire suite takes
>>> rather long.
>>
>> You can still select them by name using a regex, e.g.
>>
>>   runtests.py 'run\.empty_builtin_constructors'
>>
>> Stefan
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> Great, thanks! I'll update the hackerguide wiki.
>
I see now that it is briefly mentioned there, apologies.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 11 April 2011 13:03, Dag Sverre Seljebotn  wrote:
> On 04/11/2011 01:02 PM, Dag Sverre Seljebotn wrote:
>>
>> On 04/11/2011 12:14 PM, mark florisson wrote:
>>>
>>> On 11 April 2011 12:08, Dag Sverre
>>> Seljebotn wrote:
>>>>
>>>> On 04/11/2011 11:41 AM, mark florisson wrote:
>>>>>
>>>>> On 11 April 2011 11:10, Dag Sverre
>>>>> Seljebotn
>>>>> wrote:
>>>>>>
>>>>>> On 04/11/2011 10:45 AM, mark florisson wrote:
>>>>>>>
>>>>>>> On 5 April 2011 22:29, Dag Sverre
>>>>>>> Seljebotn
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I've done a pretty major revision to the prange CEP, bringing in
>>>>>>>> a lot
>>>>>>>> of
>>>>>>>> the feedback.
>>>>>>>>
>>>>>>>> Thread-private variables are now split in two cases:
>>>>>>>>
>>>>>>>> i) The safe cases, which really require very little technical
>>>>>>>> knowledge
>>>>>>>> ->
>>>>>>>> automatically inferred
>>>>>>>>
>>>>>>>> ii) As an advanced feature, unsafe cases that requires some
>>>>>>>> knowledge
>>>>>>>> of
>>>>>>>> threading -> must be explicitly declared
>>>>>>>>
>>>>>>>> I think this split simplifies things a great deal.
>>>>>>>
>>>>>>> Can't we obsolete the declaration entirely by assigning to variables
>>>>>>> that need to have firstprivate behaviour inside the with parallel
>>>>>>> block? Basically in the same way the scratch space is used. The only
>>>>>>> problem with that is that it won't be lastprivate, so the value will
>>>>>>> be undefined after the parallel block (but not after the worksharing
>>>>>>> loop).
>>>>>>>
>>>>>>> cdef int myvariable
>>>>>>>
>>>>>>> with nogil, parallel:
>>>>>>> myvariable = 2
>>>>>>> for i in prange(...):
>>>>>>> use myvariable
>>>>>>> maybe assign to myvariable
>>>>>>>
>>>>>>> # myvariable is well-defined here
>>>>>>>
>>>>>>> # myvariable is not well-defined here
>>>>>>>
>>>>>>> If you still desperately want lastprivate behaviour you can simply
>>>>>>> assign myvariable to another variable in the loop body.
>>>>>>
>>>>>> I don't care about lastprivate, I don't think that is an issue, as you
>>>>>> say.
>>>>>>
>>>>>> My problem with this is that it means going into an area where
>>>>>> possibly
>>>>>> tricky things are implicit rather than explicit. I also see this as a
>>>>>> rather
>>>>>> special case that will be seldomly used, and implicit behaviour is
>>>>>> more
>>>>>> difficult to justify because of that.
>>>>>
>>>>> Indeed, I actually considered if we should support firstprivate at
>>>>> all, as it's really about "being firstprivate and lastprivate".
>>>>> Without any declaration, you can have firstprivate or lastprivate, but
>>>>> not both :) So I agree that supporting such a (probably) uncommon case
>>>>> is better left explicit. On the other hand it seems silly to have
>>>>> support for such a weird case.
>>>>
>>>> Well, I actually need to do the per-thread cache thing I described in
>>>> the
>>>> CEP in my own codes, so it's not *that* special; it'd be nice to
>>>> support it.
>>>
>>> You need 'old_ell' and 'alpha' after the loop?
>>
>>
>> No...but I need the values to not be blanked out at the beginning of
>> each loop iteration!
>
> Sorry, I now realize that re-reading your email I may have misunderstood
> you. Anyway, no, I don't need lastprivate at all anywhere.

Right, so basically you can rewrite your example by introducing the
parallel block (which doesn't add an indentation level as you're
already using nogil) and assigning to your variables that need to be
firstprivate there. The only thing you miss out on is lastprivate
behaviour. So basically, the question is, do we want explicit syntax
for such a rare case (firstprivate + lastprivate)?

I must say, I found your previous argument of future shared
declarations persuasive enough to introduce explicit syntax.

> Dag Sverre
>
>>
>> Note that in the CEP, the implicitly thread-local variables are *not
>> available* before the first assignment in the loop. That is, code such
>> as this is NOT allowed:
>>
>> cdef double x
>> ...
>> for i in prange(10):
>> print x
>> x = f(x)
>>
>> We raise a compiler error in such cases if we can: The code above is
>> violating the contract that the order of execution of loop bodies should
>> not matter.
>>
>> In cases where we can't raise an error (because we didn't bother or
>> because it is not possible with a proof), we still initialize the
>> variables to invalid values (NaN for double) at the beginning of the
>> for-loop just to be sure the contract is satisfied.
>>
>> This was added to answer Stefan's objection to new types of implicit
>> scopes (and I agree with his concern).
>>
>> Dag Sverre
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Test runner

[mark florisson]

Another, different but related issue: how can we get useful output
from the test runner? e.g. I'm running my test with a
'@cython.test_assert_path_exists("...")' and I get this error output:

==
ERROR: runTest (__main__.CythonRunTestCase)
compiling (c) and running parallel
--
Traceback (most recent call last):
  File "runtests.py", line 555, in run
self.runCompileTest()
  File "runtests.py", line 386, in runCompileTest
self.test_directory, self.expect_errors, self.annotate)
  File "runtests.py", line 532, in compile
self.assertEquals(None, unexpected_error)
AssertionError: None != u'9:0: Compiler crash in TreeAssertVisitor'

So I'm seeing a traceback from the test runner (which I'm not really
interested in :), but the actual traceback is not displayed.

Can I also specify special link and compiler flags for a certain test,
like in http://wiki.cython.org/enhancements/distutils_preprocessing ?
Or do I have to export LDFLAGS and CFLAGS in my environment?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 5 April 2011 22:29, Dag Sverre Seljebotn  wrote:
> I've done a pretty major revision to the prange CEP, bringing in a lot of
> the feedback.
>
> Thread-private variables are now split in two cases:
>
>  i) The safe cases, which really require very little technical knowledge ->
> automatically inferred
>
>  ii) As an advanced feature, unsafe cases that requires some knowledge of
> threading -> must be explicitly declared
>
> I think this split simplifies things a great deal.
>
> I'm rather excited over this now; this could turn out to be a really
> user-friendly and safe feature that would not only allow us to support
> OpenMP-like threading, but be more convenient to use in a range of common
> cases.
>
> http://wiki.cython.org/enhancements/prange
>
> Dag Sverre
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>

If we want to support cython.parallel.threadsavailable outside of
parallel regions (which does not depend on the schedule used for
worksharing constructs!), then we have to disable dynamic scheduling.
For instance, if OpenMP sees some OpenMP threads are already busy,
then with dynamic scheduling it dynamically establishes how many
threads to use for any parallel region.
So basically, if you put omp_get_num_threads() in a parallel region,
you have a race when you depend on that result in a subsequent
parallel region, because the number of busy OpenMP threads may have
changed.

So basically, to make threadsavailable() work outside parallel
regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
Of course, when OpenMP cannot request the amount of threads desired
(because they are bounded by a configurable thread limit (and the OS
of course)), the behaviour will be implementation defined. So then we
could just put a warning in the docs for that, and users can check for
this in the parallel region using threadsavailable() if it's really
important.

Does that sound like a good idea? And should I update the CEP?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 13 April 2011 21:57, Dag Sverre Seljebotn  wrote:
> On 04/13/2011 09:31 PM, mark florisson wrote:
>>
>> On 5 April 2011 22:29, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>> the feedback.
>>>
>>> Thread-private variables are now split in two cases:
>>>
>>>  i) The safe cases, which really require very little technical knowledge
>>> ->
>>> automatically inferred
>>>
>>>  ii) As an advanced feature, unsafe cases that requires some knowledge of
>>> threading ->  must be explicitly declared
>>>
>>> I think this split simplifies things a great deal.
>>>
>>> I'm rather excited over this now; this could turn out to be a really
>>> user-friendly and safe feature that would not only allow us to support
>>> OpenMP-like threading, but be more convenient to use in a range of common
>>> cases.
>>>
>>> http://wiki.cython.org/enhancements/prange
>>>
>>> Dag Sverre
>>>
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>
>> If we want to support cython.parallel.threadsavailable outside of
>> parallel regions (which does not depend on the schedule used for
>> worksharing constructs!), then we have to disable dynamic scheduling.
>> For instance, if OpenMP sees some OpenMP threads are already busy,
>> then with dynamic scheduling it dynamically establishes how many
>> threads to use for any parallel region.
>> So basically, if you put omp_get_num_threads() in a parallel region,
>> you have a race when you depend on that result in a subsequent
>> parallel region, because the number of busy OpenMP threads may have
>> changed.
>
> Ah, I don't know why I thought there wouldn't be a race condition. I wonder
> if the whole threadsavailable() idea should just be ditched and that we
> should think of something else. It's not a very common usecase. Starting to
> disable some forms of scheduling just to, essentially, shoehorn in one
> particular syntax, doesn't seem like the way to go.
>
> Perhaps this calls for support for the critical(?) block then, after all.
> I'm at least +1 on dropping threadsavailable() and instead require that you
> call numthreads() in a critical block:
>
> with parallel:
>    with critical:
>        # call numthreads() and allocate global buffer
>        # calling threadid() not allowed, if we can manage that
>    # get buffer slice for each thread

In that case I think you'd want single + a barrier. 'critical' means
that all threads execute the section, but exclusively. I think you
usually want to allocate either a shared worksharing buffer, or a
private thread-local buffer. In the former case you can allocate your
buffer outside any parallel section, in the latter case within the
parallel section. It the latter case the buffer will just not be
available outside of the parallel section.

We can still support any write-back to shared variables that are
explicitly declared later on (supposing we'd also support single and
barriers. Then the code would read as follows

cdef shared(void *) buf
cdef void *localbuf

with nogil, parallel:
with single:
buf = malloc(n * numthreads())

barrier()

localbuf = buf + n * threadid()


# localbuf undefined here
# buf is well-defined here

However, I don't believe it's very common to want to use private
buffers after the loop. If you have a buffer in terms of your loop
size, you want it shared, but I can't imagine a case where you want to
examine buffers that were allocated specifically for each thread after
the parallel section. So I'm +1 on dropping threadsavailable outside
parallel sections, but currently -1 on supporting this case, because
we can solve it later on with support for explicitly declared
variables + single + barriers.

>> So basically, to make threadsavailable() work outside parallel
>> regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
>> Of course, when OpenMP cannot request the amount of threads desired
>> (because they are bounded by a configurable thread limit (and the OS
>> of course)), the behaviour will be implementation defined. So then we
>> could just put a warning in the docs for that, and users can check for
>> this in the parallel region using threadsavailable() if it's really
>> important.
>
> Do you have any experience with what actually happen with, say, GNU O

Re: [Cython] prange CEP updated

[mark florisson]

On 13 April 2011 22:53, mark florisson  wrote:
> On 13 April 2011 21:57, Dag Sverre Seljebotn  
> wrote:
>> On 04/13/2011 09:31 PM, mark florisson wrote:
>>>
>>> On 5 April 2011 22:29, Dag Sverre Seljebotn
>>>  wrote:
>>>>
>>>> I've done a pretty major revision to the prange CEP, bringing in a lot of
>>>> the feedback.
>>>>
>>>> Thread-private variables are now split in two cases:
>>>>
>>>>  i) The safe cases, which really require very little technical knowledge
>>>> ->
>>>> automatically inferred
>>>>
>>>>  ii) As an advanced feature, unsafe cases that requires some knowledge of
>>>> threading ->  must be explicitly declared
>>>>
>>>> I think this split simplifies things a great deal.
>>>>
>>>> I'm rather excited over this now; this could turn out to be a really
>>>> user-friendly and safe feature that would not only allow us to support
>>>> OpenMP-like threading, but be more convenient to use in a range of common
>>>> cases.
>>>>
>>>> http://wiki.cython.org/enhancements/prange
>>>>
>>>> Dag Sverre
>>>>
>>>> ___
>>>> cython-devel mailing list
>>>> cython-devel@python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>>
>>>
>>> If we want to support cython.parallel.threadsavailable outside of
>>> parallel regions (which does not depend on the schedule used for
>>> worksharing constructs!), then we have to disable dynamic scheduling.
>>> For instance, if OpenMP sees some OpenMP threads are already busy,
>>> then with dynamic scheduling it dynamically establishes how many
>>> threads to use for any parallel region.
>>> So basically, if you put omp_get_num_threads() in a parallel region,
>>> you have a race when you depend on that result in a subsequent
>>> parallel region, because the number of busy OpenMP threads may have
>>> changed.
>>
>> Ah, I don't know why I thought there wouldn't be a race condition. I wonder
>> if the whole threadsavailable() idea should just be ditched and that we
>> should think of something else. It's not a very common usecase. Starting to
>> disable some forms of scheduling just to, essentially, shoehorn in one
>> particular syntax, doesn't seem like the way to go.
>>
>> Perhaps this calls for support for the critical(?) block then, after all.
>> I'm at least +1 on dropping threadsavailable() and instead require that you
>> call numthreads() in a critical block:
>>
>> with parallel:
>>    with critical:
>>        # call numthreads() and allocate global buffer
>>        # calling threadid() not allowed, if we can manage that
>>    # get buffer slice for each thread
>
> In that case I think you'd want single + a barrier. 'critical' means
> that all threads execute the section, but exclusively. I think you
> usually want to allocate either a shared worksharing buffer, or a
> private thread-local buffer. In the former case you can allocate your
> buffer outside any parallel section, in the latter case within the
> parallel section. It the latter case the buffer will just not be
> available outside of the parallel section.
>
> We can still support any write-back to shared variables that are
> explicitly declared later on (supposing we'd also support single and
> barriers. Then the code would read as follows
>
> cdef shared(void *) buf
> cdef void *localbuf
>
> with nogil, parallel:
>    with single:
>        buf = malloc(n * numthreads())
>
>    barrier()
>
>    localbuf = buf + n * threadid()
>    
>
> # localbuf undefined here
> # buf is well-defined here
>
> However, I don't believe it's very common to want to use private
> buffers after the loop. If you have a buffer in terms of your loop
> size, you want it shared, but I can't imagine a case where you want to
> examine buffers that were allocated specifically for each thread after
> the parallel section. So I'm +1 on dropping threadsavailable outside
> parallel sections, but currently -1 on supporting this case, because
> we can solve it later on with support for explicitly declared
> variables + single + barriers.
>
>>> So basically, to make threadsavailable() work outside parallel
>>> regions, we'd have to disable dynamic scheduling (omp_set_dynamic(0)).
>>> Of c

Re: [Cython] prange CEP updated

[mark florisson]

On 14 April 2011 20:29, Dag Sverre Seljebotn  wrote:
> On 04/13/2011 11:13 PM, mark florisson wrote:
>>
>> Although there is omp_get_max_threads():
>>
>> "The omp_get_max_threads routine returns an upper bound on the number
>> of threads that could be used to form a new team if a parallel region
>> without a num_threads clause were encountered after execution returns
>> from this routine."
>>
>> So we could have threadsvailable() evaluate to that if encountered
>> outside a parallel region. Inside, it would evaluate to
>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>
> Well, over-allocating could well mean 1 GB, which could well mean getting an
> unecesarry MemoryError (or, like in my case, if I'm not careful to set
> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
> cluster patrol process...)

The upper bound is not "however many threads you think you can start",
but rather "how many threads are considered useful for your machine".
So if you use omp_set_num_threads(), it will return the value you set
there. Otherwise, if you have e.g. a quadcore, it will return 4. The
spec says:

"Note – The return value of the omp_get_max_threads routine can be
used to dynamically allocate sufficient storage for all threads in the
team formed at the subsequent active parallel region."

So this sounds like a viable option.

> But even ignoring this, we also have to plan for people misusing the
> feature. If we put it in there, somebody somewhere *will* write code like
> this:
>
> nthreads = threadsavailable()
> with parallel:
>    for i in prange(nthreads):
>        for j in range(100*i, 100*(i+1)): [...]
>
> (Yes, they shouldn't. Yes, they will.)
>
> Combined with a race condition that will only very seldomly trigger, this
> starts to sound like a very bad idea indeed.
>
> So I agree with you that we should just leave it for now, and do
> single/barrier later.
>
> DS
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 14 April 2011 20:29, Dag Sverre Seljebotn  wrote:
> On 04/13/2011 11:13 PM, mark florisson wrote:
>>
>> Although there is omp_get_max_threads():
>>
>> "The omp_get_max_threads routine returns an upper bound on the number
>> of threads that could be used to form a new team if a parallel region
>> without a num_threads clause were encountered after execution returns
>> from this routine."
>>
>> So we could have threadsvailable() evaluate to that if encountered
>> outside a parallel region. Inside, it would evaluate to
>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>
> Well, over-allocating could well mean 1 GB, which could well mean getting an
> unecesarry MemoryError (or, like in my case, if I'm not careful to set
> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
> cluster patrol process...)
>
> But even ignoring this, we also have to plan for people misusing the
> feature. If we put it in there, somebody somewhere *will* write code like
> this:
>
> nthreads = threadsavailable()
> with parallel:
>    for i in prange(nthreads):
>        for j in range(100*i, 100*(i+1)): [...]
>
> (Yes, they shouldn't. Yes, they will.)
>
> Combined with a race condition that will only very seldomly trigger, this
> starts to sound like a very bad idea indeed.
>
> So I agree with you that we should just leave it for now, and do
> single/barrier later.

omp_get_max_threads() doesn't have a race, as it returns the upper
bound. So e.g. if between your call and your parallel section less
OpenMP threads become available, then you might get less threads, but
never more.

> DS
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 14 April 2011 20:58, Dag Sverre Seljebotn  wrote:
> On 04/14/2011 08:42 PM, mark florisson wrote:
>>
>> On 14 April 2011 20:29, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>>
>>>> Although there is omp_get_max_threads():
>>>>
>>>> "The omp_get_max_threads routine returns an upper bound on the number
>>>> of threads that could be used to form a new team if a parallel region
>>>> without a num_threads clause were encountered after execution returns
>>>> from this routine."
>>>>
>>>> So we could have threadsvailable() evaluate to that if encountered
>>>> outside a parallel region. Inside, it would evaluate to
>>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>>
>>> Well, over-allocating could well mean 1 GB, which could well mean getting
>>> an
>>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>>> cluster patrol process...)
>>>
>>> But even ignoring this, we also have to plan for people misusing the
>>> feature. If we put it in there, somebody somewhere *will* write code like
>>> this:
>>>
>>> nthreads = threadsavailable()
>>> with parallel:
>>>    for i in prange(nthreads):
>>>        for j in range(100*i, 100*(i+1)): [...]
>>>
>>> (Yes, they shouldn't. Yes, they will.)
>>>
>>> Combined with a race condition that will only very seldomly trigger, this
>>> starts to sound like a very bad idea indeed.
>>>
>>> So I agree with you that we should just leave it for now, and do
>>> single/barrier later.
>>
>> omp_get_max_threads() doesn't have a race, as it returns the upper
>> bound. So e.g. if between your call and your parallel section less
>> OpenMP threads become available, then you might get less threads, but
>> never more.
>
> Oh, now I'm following you.
>
> Well, my argument was that I think erroring in that direction is pretty bad
> as well.
>
> Also, even if we're not making it available in cython.parallel, we're not
> stopping people from calling omp_get_max_threads directly themselves, which
> should be OK for the people who know enough to do this safely...

True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any
event, we could just put a warning in the docs stating that using
threadsavailable outside parallel sections returns an upper bound on
the actual number of threads in a subsequent parallel section.

> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 14 April 2011 21:37, Dag Sverre Seljebotn  wrote:
> On 04/14/2011 09:08 PM, mark florisson wrote:
>>
>> On 14 April 2011 20:58, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 04/14/2011 08:42 PM, mark florisson wrote:
>>>>
>>>> On 14 April 2011 20:29, Dag Sverre Seljebotn
>>>>  wrote:
>>>>>
>>>>> On 04/13/2011 11:13 PM, mark florisson wrote:
>>>>>>
>>>>>> Although there is omp_get_max_threads():
>>>>>>
>>>>>> "The omp_get_max_threads routine returns an upper bound on the number
>>>>>> of threads that could be used to form a new team if a parallel region
>>>>>> without a num_threads clause were encountered after execution returns
>>>>>> from this routine."
>>>>>>
>>>>>> So we could have threadsvailable() evaluate to that if encountered
>>>>>> outside a parallel region. Inside, it would evaluate to
>>>>>> omp_get_num_threads(). At worst, people would over-allocate a bit.
>>>>>
>>>>> Well, over-allocating could well mean 1 GB, which could well mean
>>>>> getting
>>>>> an
>>>>> unecesarry MemoryError (or, like in my case, if I'm not careful to set
>>>>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the
>>>>> cluster patrol process...)
>>>>>
>>>>> But even ignoring this, we also have to plan for people misusing the
>>>>> feature. If we put it in there, somebody somewhere *will* write code
>>>>> like
>>>>> this:
>>>>>
>>>>> nthreads = threadsavailable()
>>>>> with parallel:
>>>>>    for i in prange(nthreads):
>>>>>        for j in range(100*i, 100*(i+1)): [...]
>>>>>
>>>>> (Yes, they shouldn't. Yes, they will.)
>>>>>
>>>>> Combined with a race condition that will only very seldomly trigger,
>>>>> this
>>>>> starts to sound like a very bad idea indeed.
>>>>>
>>>>> So I agree with you that we should just leave it for now, and do
>>>>> single/barrier later.
>>>>
>>>> omp_get_max_threads() doesn't have a race, as it returns the upper
>>>> bound. So e.g. if between your call and your parallel section less
>>>> OpenMP threads become available, then you might get less threads, but
>>>> never more.
>>>
>>> Oh, now I'm following you.
>>>
>>> Well, my argument was that I think erroring in that direction is pretty
>>> bad
>>> as well.
>>>
>>> Also, even if we're not making it available in cython.parallel, we're not
>>> stopping people from calling omp_get_max_threads directly themselves,
>>> which
>>> should be OK for the people who know enough to do this safely...
>>
>> True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any
>> event, we could just put a warning in the docs stating that using
>> threadsavailable outside parallel sections returns an upper bound on
>> the actual number of threads in a subsequent parallel section.
>
> I don't think outside or within makes a difference -- what about nested
> parallel sections? At least my intention in the CEP was that
> threadsavailable was always for the next section (so often it would be 1
> after entering the section).
>
> Perhaps just calling it "maxthreads" instead solves the issue.
>
> (Still, I favour just dropping threadsavailable/maxthreads for the time
> being. It is much simpler to add something later, when we've had some time
> to use it and reflect about it, than to remove something that shouldn't have
> been added.)
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Definitely true, I'll disable it for now.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 16 April 2011 18:42, Dag Sverre Seljebotn  wrote:
> (Moving discussion from http://markflorisson.wordpress.com/, where Mark
> said:)

Ok, sure, it was just an issue I was wondering about at that moment,
but it's a tricky issue, so thanks.

> """
> Started a new branch https://github.com/markflorisson88/cython/tree/openmp .
>
> Now the question is whether sharing attributes should be propagated
> outwards. e.g. if you do
>
> for i in prange(m):
>    for j in prange(n):
>        sum += i * j
>
> then ‘sum’ is a reduction for the inner parallel loop, but not for the outer
> one. So the user would currently have to rewrite this to
>
> for i in prange(m):
>    for j in prange(n):
>        sum += i * j
>    sum += 0
>
> which seems a bit silly  . Of course, we could just disable nested
> parallelism, or tell the users to use a prange and a ‘for from’ in such
> cases.
> """
>
> Dag: Interesting. The first one is definitely the behaviour we want, as long
> as it doesn't cause unintended consequences.
>
> I don't really think it will -- the important thing is that that the order
> of loop iteration evaluation must be unimportant. And that is still true
> (for the outer loop, as well as for the inner) in your first example.
>
> Question: When you have nested pranges, what will happen is that two nested
> OpenMP parallel blocks are used, right? And do you know if there is complete
> freedom/"reentrancy" in that variables that are thread-private in an outer
> parallel block and be shared in an inner one, and vice versa?

An implementation may or may not support it, and if it is supported
the behaviour can be configured through omp_set_nested(). So we should
consider the case where it is supported and enabled.

If you have a lastprivate or reduction, and after the loop these are
(reduced and) assigned to the original variable. So if that happens
inside a parallel construct which does not declare the variable
private to the construct, you actually have a race. So e.g. the nested
prange currently races in the outer parallel range.

> If so I'd think that this algorithm should work and feel natural:
>
>  - In each prange, for the purposes of variable private/shared/reduction
> inference, consider all internal "prange" just as if they had been "range";
> no special treatment.
>
>  - Recurse to children pranges.

Right, that is most natural. Algorithmically, reductions and
lastprivates (as those can have races if placed in inner parallel
constructs) propagate outwards towards the outermost parallel block,
or up to the first parallel with block, or up to the first construct
that already determined the sharing attribute.

e.g.

with parallel:
 with parallel:
for i in prange(n):
for j in prange(n):
sum += i * j
 # sum is well-defined here
# sum is undefined here

Here 'sum' is a reduction for the two innermost loops. 'sum' is not
private for the inner parallel with block, as a prange in a parallel
with block is a worksharing loop that binds to that parallel with
block. However, the outermost parallel with block declares sum (and i
and j) private, so after that block all those variables become
undefined.

However, in the outermost parallel with block, sum will have to be
initialized to 0 before anything else, or be declared firstprivate,
otherwise 'sum' is undefined to begin with. Do you think declaring it
firstprivate would be the way to go, or should we make it private and
issue a warning or perhaps even an error?

> DS
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 18 April 2011 13:06, mark florisson  wrote:
> On 16 April 2011 18:42, Dag Sverre Seljebotn  
> wrote:
>> (Moving discussion from http://markflorisson.wordpress.com/, where Mark
>> said:)
>
> Ok, sure, it was just an issue I was wondering about at that moment,
> but it's a tricky issue, so thanks.
>
>> """
>> Started a new branch https://github.com/markflorisson88/cython/tree/openmp .
>>
>> Now the question is whether sharing attributes should be propagated
>> outwards. e.g. if you do
>>
>> for i in prange(m):
>>    for j in prange(n):
>>        sum += i * j
>>
>> then ‘sum’ is a reduction for the inner parallel loop, but not for the outer
>> one. So the user would currently have to rewrite this to
>>
>> for i in prange(m):
>>    for j in prange(n):
>>        sum += i * j
>>    sum += 0
>>
>> which seems a bit silly  . Of course, we could just disable nested
>> parallelism, or tell the users to use a prange and a ‘for from’ in such
>> cases.
>> """
>>
>> Dag: Interesting. The first one is definitely the behaviour we want, as long
>> as it doesn't cause unintended consequences.
>>
>> I don't really think it will -- the important thing is that that the order
>> of loop iteration evaluation must be unimportant. And that is still true
>> (for the outer loop, as well as for the inner) in your first example.
>>
>> Question: When you have nested pranges, what will happen is that two nested
>> OpenMP parallel blocks are used, right? And do you know if there is complete
>> freedom/"reentrancy" in that variables that are thread-private in an outer
>> parallel block and be shared in an inner one, and vice versa?
>
> An implementation may or may not support it, and if it is supported
> the behaviour can be configured through omp_set_nested(). So we should
> consider the case where it is supported and enabled.
>
> If you have a lastprivate or reduction, and after the loop these are
> (reduced and) assigned to the original variable. So if that happens
> inside a parallel construct which does not declare the variable
> private to the construct, you actually have a race. So e.g. the nested
> prange currently races in the outer parallel range.
>
>> If so I'd think that this algorithm should work and feel natural:
>>
>>  - In each prange, for the purposes of variable private/shared/reduction
>> inference, consider all internal "prange" just as if they had been "range";
>> no special treatment.
>>
>>  - Recurse to children pranges.
>
> Right, that is most natural. Algorithmically, reductions and
> lastprivates (as those can have races if placed in inner parallel
> constructs) propagate outwards towards the outermost parallel block,
> or up to the first parallel with block, or up to the first construct
> that already determined the sharing attribute.
>
> e.g.
>
> with parallel:
>     with parallel:
>        for i in prange(n):
>            for j in prange(n):
>                sum += i * j
>     # sum is well-defined here
> # sum is undefined here
>
> Here 'sum' is a reduction for the two innermost loops. 'sum' is not
> private for the inner parallel with block, as a prange in a parallel
> with block is a worksharing loop that binds to that parallel with
> block. However, the outermost parallel with block declares sum (and i
> and j) private, so after that block all those variables become
> undefined.
>
> However, in the outermost parallel with block, sum will have to be
> initialized to 0 before anything else, or be declared firstprivate,
> otherwise 'sum' is undefined to begin with. Do you think declaring it
> firstprivate would be the way to go, or should we make it private and
> issue a warning or perhaps even an error?
>
>> DS
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>

Everything seems to be working, although now the user has to be
careful with nested parallel blocks as variables can be private there
(and not firstprivate), i.e., the user has to do initialization at the
right place (e.g. in the outermost parallel block that determines it
private). I'm thinking of adding a warning, as the C compiler does.

Two issues are remaining:

1) explicit declarations of firstprivates

Do we still want those?

2) buffer auxiliary vars

When unpacking numpy buffers and using typed numpy arrays, can
reassignment or updates of a buffer-related variable ever occur in
nogil code sections? I'm thinking this is not possible and therefore
all buffer variables may be shared in parallel (for) sections?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 18 April 2011 16:01, Dag Sverre Seljebotn  wrote:
> (apologies for top post)

No problem, it means I have to scroll less :)

> This all seems to scream 'disallow' to me, in particular since some openmp
> implementations may not support it etc.
>
> At any rate I feel 'parallel/parallel/prange/prange' is going to far; so
> next step could be to only allowing 'parallel/prange/parallel/prange'.
>
> But really, my feeling is that if you really do need this then you can
> always write a seperate function for the inner loop (I honestly can't think
> of a usecase anyway...). So I'd really drop it; at least until the rest of
> the gsoc project is completed :)

Ok, sure, I'll disallow it. Then the user won't be able to make
mistakes and I don't have to detect the case and issue a warning for
inner reductions or lastprivates :).

> DS
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
> mark florisson  wrote:
>>
>> On 16 April 2011 18:42, Dag Sverre Seljebotn 
>> wrote: > (Moving discussion from http://markflorisson.wordpress.com/, where
>> Mark > said:) Ok, sure, it was just an issue I was wondering about at that
>> moment, but it's a tricky issue, so thanks. > """ > Started a new branch
>> https://github.com/markflorisson88/cython/tree/openmp . > > Now the question
>> is whether sharing attributes should be propagated > outwards. e.g. if you
>> do > > for i in prange(m): >    for j in prange(n): >        sum += i * j >
>> > then ‘sum’ is a reduction for the inner parallel loop, but not for the
>> outer > one. So the user would currently have to rewrite this to > > for i
>> in prange(m): >    for j in prange(n): >        sum += i * j >    sum += 0 >
>> > which seems a bit silly  . Of course, we could just disable nested >
>> parallelism, or tell the users to use a prange and a ‘for from’ in such >
>> cases. > """ > > Dag: Interesting. The first one is definitely the behaviour
>> we want, as long > as it doesn't cause unintended consequences. > > I don't
>> really think it will -- the important thing is that that the order > of loop
>> iteration evaluation must be unimportant. And that is still true > (for the
>> outer loop, as well as for the inner) in your first example. > > Question:
>> When you have nested pranges, what will happen is that two nested > OpenMP
>> parallel blocks are used, right? And do you know if there is complete >
>> freedom/"reentrancy" in that variables that are thread-private in an outer >
>> parallel block and be shared in an inner one, and vice versa? An
>> implementation may or may not support it, and if it is supported the
>> behaviour can be configured through omp_set_nested(). So we should consider
>> the case where it is supported and enabled. If you have a lastprivate or
>> reduction, and after the loop these are (reduced and) assigned to the
>> original variable. So if that happens inside a parallel construct which does
>> not declare the variable private to the construct, you actually have a race.
>> So e.g. the nested prange currently races in the outer parallel range. > If
>> so I'd think that this algorithm should work and feel natural: > >  - In
>> each prange, for the purposes of variable private/shared/reduction >
>> inference, consider all internal "prange" just as if they had been "range";
>> > no special treatment. > >  - Recurse to children pranges. Right, that is
>> most natural. Algorithmically, reductions and lastprivates (as those can
>> have races if placed in inner parallel constructs) propagate outwards
>> towards the outermost parallel block, or up to the first parallel with
>> block, or up to the first construct that already determined the sharing
>> attribute. e.g. with parallel: with parallel: for i in prange(n): for j in
>> prange(n): sum += i * j # sum is well-defined here # sum is undefined here
>> Here 'sum' is a reduction for the two innermost loops. 'sum' is not private
>> for the inner parallel with block, as a prange in a parallel with block is a
>> worksharing loop that binds to that parallel with block. However, the
>> outermost parallel with block declares sum (and i and j) private, so after
>> that block all those variables become undefined. However, in the outermost
>> parallel with block, sum will have to be initialized to 0 before anything
>> else, or be declared firstprivate, otherwise 'sum' is undefined to be

Re: [Cython] prange CEP updated

[mark florisson]

On 18 April 2011 16:41, Dag Sverre Seljebotn  wrote:
> Excellent! Sounds great! (as I won't have my laptop for some days I can't
> have a look yet but I will later)
>
> You're right about (the current) buffers and the gil. A testcase explicitly
> for them would be good.
>
> Firstprivate etc: i think it'd be nice myself, but it is probably better to
> take a break from it at this point so that we can think more about that and
> not do anything rash; perhaps open up a specific thread on them and ask for
> more general input. Perhaps you want to take a break or task-switch to
> something else (fused types?) until I can get around to review and merge
> what you have so far? You'll know best what works for you though. If you
> decide to implement explicit threadprivate variables because you've got the
> flow I certainly wom't object myself.
>
 Ok, cool, I'll move on :) I already included a test with a prange and
a numpy buffer with indexing.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] gilnanny

[mark florisson]

Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
for every refnanny inc- and decref, so that it will issue a fatal
error whenever reference counting is done without the gil, to make
sure we never do any illegal things in nogil code blocks. While I'm at
it, I also want to add a pystate.pxd to the cpython includes.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] gilnanny

[mark florisson]

On 18 April 2011 22:26, Robert Bradshaw  wrote:
> On Mon, Apr 18, 2011 at 12:08 PM, mark florisson
>  wrote:
>> Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get()
>> for every refnanny inc- and decref, so that it will issue a fatal
>> error whenever reference counting is done without the gil, to make
>> sure we never do any illegal things in nogil code blocks.
>
> Sounds like a good idea to me.

Ok, cool :)

>> While I'm at it, I also want to add a pystate.pxd to the cpython includes.
>
> Sure, corresponding to pystate.h? Have you looked into how stable this
> API is (e.g. Py2 vs. Py3)?

I haven't looked at all the functions, but most of them are documented
in both Python 2 and Python 3. I'll be vigilant.

> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Hudson account

[mark florisson]

Can I get a Hudson account so I can setup my branches there? I can't
seem to login using my trac credentials.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson account

[mark florisson]

On 19 April 2011 12:19, Stefan Behnel  wrote:
> mark florisson, 19.04.2011 11:14:
>>
>> Can I get a Hudson account so I can setup my branches there?
>
> I've created one for you. Please tell me when you have changed your
> password, so that I can give you write access.

Done.

> Please be careful not to break too much, and note that it's easier to work
> at the file system level if you want to do mass duplication/adaptation of
> jobs. If you tell me what you need, I can set it up for you.

All I want is to run the master branch (it has the 'with gil'
statement') and the openmp branch. I also want the fusedtypes branch,
which I'll create in a minute or two.

> It may actually be best to start from Viteks configured jobs, copy them and
> adapt their github URL and branch name.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson account

[mark florisson]

On 19 April 2011 12:31, mark florisson  wrote:
> On 19 April 2011 12:19, Stefan Behnel  wrote:
>> mark florisson, 19.04.2011 11:14:
>>>
>>> Can I get a Hudson account so I can setup my branches there?
>>
>> I've created one for you. Please tell me when you have changed your
>> password, so that I can give you write access.
>
> Done.
>
>> Please be careful not to break too much, and note that it's easier to work
>> at the file system level if you want to do mass duplication/adaptation of
>> jobs. If you tell me what you need, I can set it up for you.
>
> All I want is to run the master branch (it has the 'with gil'
> statement') and the openmp branch. I also want the fusedtypes branch,
> which I'll create in a minute or two.

Oh, as for the versions, I think at least 2.3, 2.7 and py3k would be a
good idea.
Does the hudson server have numpy installed?

>> It may actually be best to start from Viteks configured jobs, copy them and
>> adapt their github URL and branch name.
>>
>> Stefan
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson account

[mark florisson]

On 19 April 2011 12:53, Stefan Behnel  wrote:
> mark florisson, 19.04.2011 12:31:
>>
>> On 19 April 2011 12:19, Stefan Behnel wrote:
>>>
>>> mark florisson, 19.04.2011 11:14:
>>>>
>>>> Can I get a Hudson account so I can setup my branches there?
>>>
>>> I've created one for you. Please tell me when you have changed your
>>> password, so that I can give you write access.
>>
>> Done.
>
> Ok, you should have write rights now.
>
>
>>> Please be careful not to break too much, and note that it's easier to
>>> work
>>> at the file system level if you want to do mass duplication/adaptation of
>>> jobs. If you tell me what you need, I can set it up for you.
>>
>> All I want is to run the master branch (it has the 'with gil'
>> statement') and the openmp branch. I also want the fusedtypes branch,
>> which I'll create in a minute or two.
>
> That's three branches in total, plus the builds in three CPython versions
> (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30 new
> jobs that will compete with the others.
>
> I would expect that the three branches would rarely be changed all at the
> same time (except when you update them from the main branch), but that may
> still result in quite some additional delays due to the longer build queue.

Switching the branches would be fine with me, so 3 Python versions x 1
branch x 2xC/C++ = 6 jobs. Is it necessary to separate C from C++
testing?

> Vitek currently has one set of build jobs for himself and can change the
> branches at need. Question to the others: should we continue to do this, or
> set up a full set of build jobs for each git branch we want to test?
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson account

[mark florisson]

On 19 April 2011 13:36, Stefan Behnel  wrote:
> mark florisson, 19.04.2011 13:09:
>>
>> On 19 April 2011 12:53, Stefan Behnel wrote:
>>>
>>> mark florisson, 19.04.2011 12:31:
>>>>
>>>> All I want is to run the master branch (it has the 'with gil'
>>>> statement') and the openmp branch. I also want the fusedtypes branch,
>>>> which I'll create in a minute or two.
>>>
>>> That's three branches in total, plus the builds in three CPython versions
>>> (2.4, 2.7 and Py3k), plus 2xC/C++ testing each. So, at least 3+9+18=30
>>> new
>>> jobs that will compete with the others.
>>>
>>> I would expect that the three branches would rarely be changed all at the
>>> same time (except when you update them from the main branch), but that
>>> may
>>> still result in quite some additional delays due to the longer build
>>> queue.
>>
>> Switching the branches would be fine with me, so 3 Python versions x 1
>> branch x 2xC/C++ = 6 jobs.
>
> 10 actually (1+3+6), but only the (6) test jobs are long running.
>
> The job tree first builds an sdist from github, then builds bdists from that
> in all CPython versions, then tests the result in the test jobs.
>
I see, ok.

>> Is it necessary to separate C from C++ testing?
>
> Not necessary, it just gives quicker feedback, right after the first test
> job is finished. If you test both in one job, it just runs twice as long.
>
> The problem is not so much the sheer number of jobs (we have several hundred
> Hudson jobs up at work). It's the time it takes to run the ones that get
> triggered, and especially long running jobs that fill up the pipeline.
>
> If you can live with a longer response time (usually just a couple of
> minutes), I'd suggest merging the test jobs, so that only one of them fills
> up the pipeline per build, instead of two.

Ok. So are you setting it up, or should I do it?

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson account

[mark florisson]

On 19 April 2011 18:27, Stefan Behnel  wrote:
> mark florisson, 19.04.2011 13:46:
>>
>> So are you setting it up
>
> Done. You now have your first red Hudson jobs. ;)

Thanks a lot for setting it up! Red means good right? :)

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Build module script

[mark florisson]

On 21 April 2011 08:57, Vitja Makarov  wrote:
> Now we have cythonrun build script, may be it's time to create script
> for easy module building?
>
> --
> vitja.
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

We have cythonize():
http://wiki.cython.org/enhancements/distutils_preprocessing . Is
something like that what you mean? Unfortunately I believe it doesn't
parse sys.argv (yet). Perhaps a recursive option would also be useful.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Build module script

[mark florisson]

On 21 April 2011 10:23, Vitja Makarov  wrote:
> 2011/4/21 mark florisson :
>> On 21 April 2011 08:57, Vitja Makarov  wrote:
>>> Now we have cythonrun build script, may be it's time to create script
>>> for easy module building?
>>>
>>> --
>>> vitja.
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> We have cythonize():
>> http://wiki.cython.org/enhancements/distutils_preprocessing . Is
>> something like that what you mean? Unfortunately I believe it doesn't
>> parse sys.argv (yet). Perhaps a recursive option would also be useful.
>
> Not exactly cythonize is for distutils and friends.
>
> I mean simple script that run cython compiler and then build shared
> python module.
> It's much better then manually run cython then gcc or write makefile for this.

Ah, I see. Yeah that might be convenient. Although there is always pyximport.

> --
> vitja.
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 21 April 2011 10:37, Robert Bradshaw  wrote:
> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>  wrote:
>> On 18 April 2011 16:41, Dag Sverre Seljebotn  
>> wrote:
>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>> have a look yet but I will later)
>>>
>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>> for them would be good.
>>>
>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>> take a break from it at this point so that we can think more about that and
>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>> more general input. Perhaps you want to take a break or task-switch to
>>> something else (fused types?) until I can get around to review and merge
>>> what you have so far? You'll know best what works for you though. If you
>>> decide to implement explicit threadprivate variables because you've got the
>>> flow I certainly wom't object myself.
>>>
>>  Ok, cool, I'll move on :) I already included a test with a prange and
>> a numpy buffer with indexing.
>
> Wow, you're just plowing away at this. Very cool.
>
> +1 to disallowing nested prange, that seems to get really messy with
> little benefit.
>
> In terms of the CEP, I'm still unconvinced that firstprivate is not
> safe to infer, but lets leave the initial values undefined rather than
> specifying them to be NaNs (we can do that as an implementation if you
> want), which will give us flexibility to change later once we've had a
> chance to play around with it.

Yes, they are currently undefined (and not initialized to NaN etc).
The thing is that without the control flow analysis (or perhaps not
until runtime) you won't know whether a variable is initialized at all
before the parallel section, so making it firstprivate might actually
copy an undefined value (perhaps with a trap representation!) into the
thread-private copy, which might invalidate valid code. e.g. consider

x_is_initialized = False
if condition:
x = 1
x_is_initialized = True

for i in prange(10, schedule='static'):
if x_is_initialized:
printf("%d\n", x)
x = i

> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
> We also probably want some way of explicitly marking a variable as
> shared and still be able to assign to/flush/sync it. Perhaps the
> parallel context could be used for these declarations, i.e.
>
>    with parallel(threadlocal=a, shared=(b,c)):
>        ...
>
> which would be considered an "expert" usecase.

Indeed, assigning to elements in an array instead doesn't seem very
convenient :)

> For all the discussion of threadsavailable/threadid, the most common
> usecase I see is for allocating a large shared buffer and partitioning
> it. This seems better handled by allocating separate thread-local
> buffers, no? I still like the context idea, but everything in a
> parallel block before and after the loop(s) also seems like a natural
> place to put any setup/teardown code (though the context has the
> advantage that __exit__ is always called, even if exceptions are
> raised, which makes cleanup a lot easier to handle).

Currently 'with gil' isn't merged into that branch, and if it will, it
will be disallowed, as I'm not yet sure how (if at all) it could be
handled with regard to exceptions. It seems a lot easier to disallow
it and have the user write a 'with gil' function, from which nothing
can propagate.

> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 21 April 2011 10:59, mark florisson  wrote:
> On 21 April 2011 10:37, Robert Bradshaw  wrote:
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>  wrote:
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn  
>>> wrote:
>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>> have a look yet but I will later)
>>>>
>>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>>> for them would be good.
>>>>
>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>>> take a break from it at this point so that we can think more about that and
>>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>>> more general input. Perhaps you want to take a break or task-switch to
>>>> something else (fused types?) until I can get around to review and merge
>>>> what you have so far? You'll know best what works for you though. If you
>>>> decide to implement explicit threadprivate variables because you've got the
>>>> flow I certainly wom't object myself.
>>>>
>>>  Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> Yes, they are currently undefined (and not initialized to NaN etc).
> The thing is that without the control flow analysis (or perhaps not
> until runtime) you won't know whether a variable is initialized at all
> before the parallel section, so making it firstprivate might actually
> copy an undefined value (perhaps with a trap representation!) into the
> thread-private copy, which might invalidate valid code. e.g. consider
>
> x_is_initialized = False
> if condition:
>    x = 1
>    x_is_initialized = True
>
> for i in prange(10, schedule='static'):
>    if x_is_initialized:
>        printf("%d\n", x)
>    x = i

Erm, that snippet I posted is invalid in any case, as x will be
private. So guess initializing things to NaN in such would have to
occur in the parallel section that should enclose the for. So e.g.
we'd have to do

#pragma omp parallel private(x)
{
x = INT_MAX;
#pragma omp for lastprivate(i)
for (...)
...
}

Which would then mean that 'x' cannot be lastprivate anymore :). So
it's either "uninitialized and undefined" or "firstprivate". I
personally prefer the former for the implicit route.

I do like the threadlocal=a stuff to parallel, it's basically what I
proposed a while back except that you don't make them strings, but
better because most of your variables can be inferred, so the
messiness is gone.

>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>>    with parallel(threadlocal=a, shared=(b,c)):
>>        ...
>>
>> which would be considered an "expert" usecase.
>
> Indeed, assigning to elements in an array instead doesn't seem very
> convenient :)
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> Currently 'with gil' isn't merged into that branch, and if it will, it
> will be disallowed, as I'm not yet sure how (if at all) it could be
> handled with regard to exceptions. It seems a lot easier to disallow
> it and have the user write a 'with gil' function, from which nothing
> can propagate.
>
>> - Robert
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 21 April 2011 11:37, Robert Bradshaw  wrote:
> On Thu, Apr 21, 2011 at 2:21 AM, mark florisson
>  wrote:
>> On 21 April 2011 10:59, mark florisson  wrote:
>>> On 21 April 2011 10:37, Robert Bradshaw  
>>> wrote:
>>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>>>  wrote:
>>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn  
>>>>> wrote:
>>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>>>> have a look yet but I will later)
>>>>>>
>>>>>> You're right about (the current) buffers and the gil. A testcase 
>>>>>> explicitly
>>>>>> for them would be good.
>>>>>>
>>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better 
>>>>>> to
>>>>>> take a break from it at this point so that we can think more about that 
>>>>>> and
>>>>>> not do anything rash; perhaps open up a specific thread on them and ask 
>>>>>> for
>>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>>> something else (fused types?) until I can get around to review and merge
>>>>>> what you have so far? You'll know best what works for you though. If you
>>>>>> decide to implement explicit threadprivate variables because you've got 
>>>>>> the
>>>>>> flow I certainly wom't object myself.
>>>>>>
>>>>>  Ok, cool, I'll move on :) I already included a test with a prange and
>>>>> a numpy buffer with indexing.
>>>>
>>>> Wow, you're just plowing away at this. Very cool.
>>>>
>>>> +1 to disallowing nested prange, that seems to get really messy with
>>>> little benefit.
>>>>
>>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>>> safe to infer, but lets leave the initial values undefined rather than
>>>> specifying them to be NaNs (we can do that as an implementation if you
>>>> want), which will give us flexibility to change later once we've had a
>>>> chance to play around with it.
>>>
>>> Yes, they are currently undefined (and not initialized to NaN etc).
>>> The thing is that without the control flow analysis (or perhaps not
>>> until runtime) you won't know whether a variable is initialized at all
>>> before the parallel section, so making it firstprivate might actually
>>> copy an undefined value (perhaps with a trap representation!) into the
>>> thread-private copy, which might invalidate valid code. e.g. consider
>>>
>>> x_is_initialized = False
>>> if condition:
>>>    x = 1
>>>    x_is_initialized = True
>>>
>>> for i in prange(10, schedule='static'):
>>>    if x_is_initialized:
>>>        printf("%d\n", x)
>>>    x = i
>>
>> Erm, that snippet I posted is invalid in any case, as x will be
>> private. So guess initializing things to NaN in such would have to
>> occur in the parallel section that should enclose the for. So e.g.
>> we'd have to do
>>
>> #pragma omp parallel private(x)
>> {
>>    x = INT_MAX;
>>    #pragma omp for lastprivate(i)
>>    for (...)
>>        ...
>> }
>>
>> Which would then mean that 'x' cannot be lastprivate anymore :). So
>> it's either "uninitialized and undefined" or "firstprivate". I
>> personally prefer the former for the implicit route.
>
> A variable can't be both first and last private? In any case, as long
> as we don't promise anything about them now, we can decide later.

It can be, but not if the binding parallel region declares it private.
So we wouldn't actually need the snippet above, we could just do

x = INT_MAX;
#pragma omp parallel for firstprivate(x) lastprivate(i, x)
for (...)
...

Yeah, that would work.

>> I do like the threadlocal=a stuff to parallel, it's basically what I
>> proposed a while back except that you don't make them strings, but
>> better because most of your variables can be inferred, so the
>> messiness is gone.
>
> Yep.
>
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 21 April 2011 11:18, Robert Bradshaw  wrote:
> On Thu, Apr 21, 2011 at 1:59 AM, mark florisson
>  wrote:
>> On 21 April 2011 10:37, Robert Bradshaw  wrote:
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>>  wrote:
>>>> On 18 April 2011 16:41, Dag Sverre Seljebotn  
>>>> wrote:
>>>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>>>> have a look yet but I will later)
>>>>>
>>>>> You're right about (the current) buffers and the gil. A testcase 
>>>>> explicitly
>>>>> for them would be good.
>>>>>
>>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better 
>>>>> to
>>>>> take a break from it at this point so that we can think more about that 
>>>>> and
>>>>> not do anything rash; perhaps open up a specific thread on them and ask 
>>>>> for
>>>>> more general input. Perhaps you want to take a break or task-switch to
>>>>> something else (fused types?) until I can get around to review and merge
>>>>> what you have so far? You'll know best what works for you though. If you
>>>>> decide to implement explicit threadprivate variables because you've got 
>>>>> the
>>>>> flow I certainly wom't object myself.
>>>>>
>>>>  Ok, cool, I'll move on :) I already included a test with a prange and
>>>> a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> Yes, they are currently undefined (and not initialized to NaN etc).
>> The thing is that without the control flow analysis (or perhaps not
>> until runtime) you won't know whether a variable is initialized at all
>> before the parallel section, so making it firstprivate might actually
>> copy an undefined value (perhaps with a trap representation!) into the
>> thread-private copy, which might invalidate valid code. e.g. consider
>>
>> x_is_initialized = False
>> if condition:
>>    x = 1
>>    x_is_initialized = True
>>
>> for i in prange(10, schedule='static'):
>>    if x_is_initialized:
>>        printf("%d\n", x)
>>    x = i
>
> I'm still failing to see how this is a problem (or anything new, as
> opposed to this same example with an ordinary range).
>>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>>> We also probably want some way of explicitly marking a variable as
>>> shared and still be able to assign to/flush/sync it. Perhaps the
>>> parallel context could be used for these declarations, i.e.
>>>
>>>    with parallel(threadlocal=a, shared=(b,c)):
>>>        ...
>>>
>>> which would be considered an "expert" usecase.
>>
>> Indeed, assigning to elements in an array instead doesn't seem very
>> convenient :)
>>
>>> For all the discussion of threadsavailable/threadid, the most common
>>> usecase I see is for allocating a large shared buffer and partitioning
>>> it. This seems better handled by allocating separate thread-local
>>> buffers, no? I still like the context idea, but everything in a
>>> parallel block before and after the loop(s) also seems like a natural
>>> place to put any setup/teardown code (though the context has the
>>> advantage that __exit__ is always called, even if exceptions are
>>> raised, which makes cleanup a lot easier to handle).
>>
>> Currently 'with gil' isn't merged into that branch, and if it will, it
>> will be disallowed, as I'm not yet sure how (if at all) it could be
>> handled with regard to exceptions. It seems a lot easier to disallow
>> it and have the user write a 'with gil' function, from which nothing
>> can propagate.
>
> Not being able to propagate exceptions is a pretty strong
> constraint--even if the implementation doesn't yet support it, it'd be
> nice to have an API that makes it possible as a future feature.

It would be possible, with some modifications to try/finally. I think
it'd be best to stabilize and merge with gil first.

> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Fused Types

[mark florisson]

Hey,

I've been working a bit on fused types
(http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
generate code for every permutation of specific types. Currently it
only works for cdef functions. Now in SimpleCallNode I'm trying to use
PyrexTypes.best_match() to find the function with the best signature,
but it doesn't seem to take care of coercions, e.g. it calls
'assignable_from' on the dst_type (e.g. char *), but for
BuiltinObjectType (a str) this returns false. Why is this the case,
don't calls to overloaded C++ methods need to dispatch properly here
also?

Other issues are public and api declarations. So should we support
that at all? We could define a macro that calls a function that does
the dispatch. So e.g. for this

ctypedef cython.fused_type(typeA, typeB) dtype

cdef func(dtype x):
...

we would get two generated functions, say, __pyx_typeA_func and
__pyx_typeB_func. So we could have a macro get_func(dtype) or
something that then substitutes __pyx_get_func(#dtype), where
__pyx_get_func returns the pointer to the right function based on the
type names. I'm not sure we should support it, right now I just put
the mangled names in the header. At least the cdef functions will be
sharable between Cython implementation files.

I also noticed that for cdef functions with optional argument it gets
a struct as argument, but this struct doesn't seem to end up in the
header file when the function is declared public. I believe that also
the typedefs for ctypedef-ed things in the .pyx file don't make it
there when used to type a cdef function's arguments. Should that be
fixed?

Cheers,

Mark
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] prange CEP updated

[mark florisson]

On 21 April 2011 20:13, Dag Sverre Seljebotn  wrote:
> On 04/21/2011 10:37 AM, Robert Bradshaw wrote:
>>
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>   wrote:
>>>
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn
>>>  wrote:
>>>>
>>>> Excellent! Sounds great! (as I won't have my laptop for some days I
>>>> can't
>>>> have a look yet but I will later)
>>>>
>>>> You're right about (the current) buffers and the gil. A testcase
>>>> explicitly
>>>> for them would be good.
>>>>
>>>> Firstprivate etc: i think it'd be nice myself, but it is probably better
>>>> to
>>>> take a break from it at this point so that we can think more about that
>>>> and
>>>> not do anything rash; perhaps open up a specific thread on them and ask
>>>> for
>>>> more general input. Perhaps you want to take a break or task-switch to
>>>> something else (fused types?) until I can get around to review and merge
>>>> what you have so far? You'll know best what works for you though. If you
>>>> decide to implement explicit threadprivate variables because you've got
>>>> the
>>>> flow I certainly wom't object myself.
>>>>
>>>  Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> I don't see any technical issues with inferring firstprivate, the question
> is whether we want to. I suggest not inferring it in order to make this
> safer: One should be able to just try to change a loop from "range" to
> "prange", and either a) have things fail very hard, or b) just work
> correctly and be able to trust the results.
>
> Note that when I suggest using NaN, it is as initial values for EACH
> ITERATION, not per-thread initialization. It is not about "firstprivate" or
> not, but about disabling thread-private variables entirely in favor of
> "per-iteration" variables.
>
> I believe that by talking about "readonly" and "per-iteration" variables,
> rather than "thread-shared" and "thread-private" variables, this can be used
> much more safely and with virtually no knowledge of the details of
> threading. Again, what's in my mind are scientific programmers with (too)
> little training.
>
> In the end it's a matter of taste and what is most convenient to more users.
> But I believe the case of needing real thread-private variables that
> preserves per-thread values across iterations (and thus also can possibly
> benefit from firstprivate) is seldomly enough used that an explicit
> declaration is OK, in particular when it buys us so much in safety in the
> common case.
>
> To be very precise,
>
> cdef double x, z
> for i in prange(n):
>    x = f(x)
>    z = f(i)
>    ...
>
> goes to
>
> cdef double x, z
> for i in prange(n):
>    x = z = nan
>    x = f(x)
>    z = f(i)
>    ...
>
> and we leave it to the C compiler to (trivially) optimize away "z = nan".
> And, yes, it is a stopgap solution until we've got control flow analysis so
> that we can outright disallow such uses of x (without threadprivate
> declaration, which also gives firstprivate behaviour).
>
Ah, I see, sure, that sounds sensible. I'm currently working on fused
types, so when I finish that up I'll return to that.
>
>>
>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>>     with parallel(threadlocal=a, shared=(b,c)):
>>         ...
>>
>> which would be considered an "expert" usecase.
>
> I'm not set on the syntax for threadlocal variables; although your proposal
> feels funny/very unpythonic, almost like a C macro. For some inspiration,
> 

Re: [Cython] Fused Types

[mark florisson]

On 26 April 2011 16:43, Stefan Behnel  wrote:
> mark florisson, 26.04.2011 16:23:
>>
>> I've been working a bit on fused types
>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>> generate code for every permutation of specific types. Currently it
>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>> PyrexTypes.best_match() to find the function with the best signature,
>> but it doesn't seem to take care of coercions, e.g. it calls
>> 'assignable_from' on the dst_type (e.g. char *), but for
>> BuiltinObjectType (a str) this returns false.
>
> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
> "bytes", though.
>
> http://wiki.cython.org/enhancements/stringliterals

Right, I see, so the thing is that I was using string literals which
should be inferred as the bytes type when they are assigned to a char
*. The thing is that because the type is fused, the argument may be
anything, so I was hoping best_match would figure it out for me.
Apparently it doesn't, and indeed, this example doesn't work:

cdef extern from "Foo.h":
cdef cppclass Foo:
Foo(char *)
Foo(int)

cdef char *foo = "foo"
cdef Foo* foo = new Foo("foo") # <- this doesn't work ("no suitable
method found")
cdef Foo* bar = new Foo(foo)   # <- this works

So that's pretty lame, I think I should fix that.

>
>> Why is this the case,
>> don't calls to overloaded C++ methods need to dispatch properly here
>> also?
>
> If this doesn't work, I assume it just isn't implemented.
>
>
>> Other issues are public and api declarations. So should we support
>> that at all? We could define a macro that calls a function that does
>> the dispatch. So e.g. for this
>>
>> ctypedef cython.fused_type(typeA, typeB) dtype
>>
>> cdef func(dtype x):
>>     ...
>>
>> we would get two generated functions, say, __pyx_typeA_func and
>> __pyx_typeB_func. So we could have a macro get_func(dtype) or
>> something that then substitutes __pyx_get_func(#dtype), where
>> __pyx_get_func returns the pointer to the right function based on the
>> type names. I'm not sure we should support it, right now I just put
>> the mangled names in the header. At least the cdef functions will be
>> sharable between Cython implementation files.
>
> I'm fine with explicitly forbidding this for now. It may eventually work for
> Python object types where we can properly dispatch, but it won't easily work
> for C types. It may work in C++, though.
>

Ok, will do.

>> I also noticed that for cdef functions with optional argument it gets
>> a struct as argument, but this struct doesn't seem to end up in the
>> header file when the function is declared public. I believe that also
>> the typedefs for ctypedef-ed things in the .pyx file don't make it
>> there when used to type a cdef function's arguments. Should that be
>> fixed?
>
> No, I think this should also become a compiler error. These functions are
> not meant to be called from C code. It's a Cython convenience feature. As
> long as it's not correctly supported on both ends of the publicly exported
> C-API, it's best to keep users from using it at all.

Ok, I'll try to make Cython issue an error for these cases.

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 26 April 2011 20:05, Robert Bradshaw  wrote:
> On Tue, Apr 26, 2011 at 8:18 AM, mark florisson
>  wrote:
>> On 26 April 2011 16:43, Stefan Behnel  wrote:
>>> mark florisson, 26.04.2011 16:23:
>>>>
>>>> I've been working a bit on fused types
>>>> (http://wiki.cython.org/enhancements/fusedtypes), and I've got it to
>>>> generate code for every permutation of specific types. Currently it
>>>> only works for cdef functions. Now in SimpleCallNode I'm trying to use
>>>> PyrexTypes.best_match() to find the function with the best signature,
>>>> but it doesn't seem to take care of coercions, e.g. it calls
>>>> 'assignable_from' on the dst_type (e.g. char *), but for
>>>> BuiltinObjectType (a str) this returns false.
>>>
>>> Which is correct. "char*" cannot coerce from/to "str". It can coerce to
>>> "bytes", though.
>>>
>>> http://wiki.cython.org/enhancements/stringliterals
>>
>> Right, I see, so the thing is that I was using string literals which
>> should be inferred as the bytes type when they are assigned to a char
>> *. The thing is that because the type is fused, the argument may be
>> anything, so I was hoping best_match would figure it out for me.
>> Apparently it doesn't, and indeed, this example doesn't work:
>>
>> cdef extern from "Foo.h":
>>    cdef cppclass Foo:
>>        Foo(char *)
>>        Foo(int)
>>
>> cdef char *foo = "foo"
>> cdef Foo* foo = new Foo("foo") # <- this doesn't work ("no suitable
>> method found")
>> cdef Foo* bar = new Foo(foo)   # <- this works
>>
>> So that's pretty lame, I think I should fix that.
>
> Agreed.
>
>>>> Why is this the case,
>>>> don't calls to overloaded C++ methods need to dispatch properly here
>>>> also?
>>>
>>> If this doesn't work, I assume it just isn't implemented.
>>>
>>>
>>>> Other issues are public and api declarations. So should we support
>>>> that at all? We could define a macro that calls a function that does
>>>> the dispatch. So e.g. for this
>>>>
>>>> ctypedef cython.fused_type(typeA, typeB) dtype
>>>>
>>>> cdef func(dtype x):
>>>>     ...
>>>>
>>>> we would get two generated functions, say, __pyx_typeA_func and
>>>> __pyx_typeB_func. So we could have a macro get_func(dtype) or
>>>> something that then substitutes __pyx_get_func(#dtype), where
>>>> __pyx_get_func returns the pointer to the right function based on the
>>>> type names. I'm not sure we should support it, right now I just put
>>>> the mangled names in the header. At least the cdef functions will be
>>>> sharable between Cython implementation files.
>>>
>>> I'm fine with explicitly forbidding this for now. It may eventually work for
>>> Python object types where we can properly dispatch, but it won't easily work
>>> for C types. It may work in C++, though.
>>>
>>
>> Ok, will do.
>
> For the moment, putting mangled names in the header should be fine. A
> macro might make sense in the long term.
>
> Somewhat orthogonal, it could makes sense to do some dispatching on
> type for cpdef functions.
>
>>>> I also noticed that for cdef functions with optional argument it gets
>>>> a struct as argument, but this struct doesn't seem to end up in the
>>>> header file when the function is declared public. I believe that also
>>>> the typedefs for ctypedef-ed things in the .pyx file don't make it
>>>> there when used to type a cdef function's arguments. Should that be
>>>> fixed?
>>>
>>> No, I think this should also become a compiler error. These functions are
>>> not meant to be called from C code. It's a Cython convenience feature. As
>>> long as it's not correctly supported on both ends of the publicly exported
>>> C-API, it's best to keep users from using it at all.
>>
>> Ok, I'll try to make Cython issue an error for these cases.
>
> +1
>
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

So I fixed all that, but I'm currently wondering about the proposed
cython.typeof(). I believe it currently returns a string with the type
name, and not the type itself. So I think it would be inconsistent to
suddenly start allowing comparison with 'is' and 'isinstance' and
such.

I'm also wondering if it would be useful to allow actual type
retrieval, which could be used in declarations and casts. For instance
consider fusing two structs with the same attribute name but different
attribute types. Perhaps in your code you want to introduce a variable
compatible with such a type, e.g. consider this:

ctypdef struct A:
int attrib

ctypedef struct B:
char *attrib

ctypedef cython.fused_type(A, B) struct_t

cdef func(struct_t mystruct, int i):
cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
...

What do you think?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 28 April 2011 21:58, Stefan Behnel  wrote:
> mark florisson, 28.04.2011 21:48:
>>
>> I'm currently wondering about the proposed
>> cython.typeof(). I believe it currently returns a string with the type
>> name, and not the type itself. So I think it would be inconsistent to
>> suddenly start allowing comparison with 'is' and 'isinstance' and
>> such.
>>
>> I'm also wondering if it would be useful to allow actual type
>> retrieval, which could be used in declarations and casts. For instance
>> consider fusing two structs with the same attribute name but different
>> attribute types. Perhaps in your code you want to introduce a variable
>> compatible with such a type, e.g. consider this:
>>
>> ctypdef struct A:
>>     int attrib
>>
>> ctypedef struct B:
>>     char *attrib
>>
>> ctypedef cython.fused_type(A, B) struct_t
>>
>> cdef func(struct_t mystruct, int i):
>>     cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>
> What's wrong with type() ?
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

When you call type you kind of expect it to call the Python builtin
type(), which means it will coerce your C type to a Python object, on
which it will call type(). So if you change those semantics suddenly
for C types, I think we'd break people's code.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 28 April 2011 22:10, mark florisson  wrote:
> On 28 April 2011 21:58, Stefan Behnel  wrote:
>> mark florisson, 28.04.2011 21:48:
>>>
>>> I'm currently wondering about the proposed
>>> cython.typeof(). I believe it currently returns a string with the type
>>> name, and not the type itself. So I think it would be inconsistent to
>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>> such.
>>>
>>> I'm also wondering if it would be useful to allow actual type
>>> retrieval, which could be used in declarations and casts. For instance
>>> consider fusing two structs with the same attribute name but different
>>> attribute types. Perhaps in your code you want to introduce a variable
>>> compatible with such a type, e.g. consider this:
>>>
>>> ctypdef struct A:
>>>     int attrib
>>>
>>> ctypedef struct B:
>>>     char *attrib
>>>
>>> ctypedef cython.fused_type(A, B) struct_t
>>>
>>> cdef func(struct_t mystruct, int i):
>>>     cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>
>> What's wrong with type() ?
>>
>> Stefan
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> When you call type you kind of expect it to call the Python builtin
> type(), which means it will coerce your C type to a Python object, on
> which it will call type(). So if you change those semantics suddenly
> for C types, I think we'd break people's code.
>

Also, you'd want this to work for Python types too, for instance for
two differently typed extension type attributes
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 28 April 2011 22:12, Robert Bradshaw  wrote:
> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>  wrote:
>
>> So I fixed all that, but I'm currently wondering about the proposed
>> cython.typeof(). I believe it currently returns a string with the type
>> name, and not the type itself.
>
> Yes. This is just because there's not really anything better to return
> at this point. We should "fix" this at some point in the future.
>
>> So I think it would be inconsistent to
>> suddenly start allowing comparison with 'is' and 'isinstance' and
>> such.
>
> I'm open to other suggestions, but would like an expression that
> resolves at compile time to true/false (and we need to do branch
> pruning on it). Note that type() is not good enough, because it has
> different semantics, i.e.
>
>    cdef object o = []
>    typeof(o), type(o)
>
> so lets not touch that one.

Right, so for fused types I don't mind string comparison with
cython.typeof(), but retrieval of the actual type for casts and
declaration remains open. I'd be fine with something like
cython.gettype().

On the other hand, the user could already do this thing by manually
declaring another fused type that would perhaps be resolved based on
its usage. e.g. for my original example:

ctypedef char *string_t
ctypedef cython.fused_type(int, string_t) attrib_t

cdef func(struct_t mystruct, int i):
cdef attrib_t var = mystruct.attrib + i

Is this something that should be supported anyway? Currently fused
types can only be used when appearing as argument types (in the
function body and as return value).

>> I'm also wondering if it would be useful to allow actual type
>> retrieval, which could be used in declarations and casts. For instance
>> consider fusing two structs with the same attribute name but different
>> attribute types. Perhaps in your code you want to introduce a variable
>> compatible with such a type, e.g. consider this:
>>
>> ctypdef struct A:
>>    int attrib
>>
>> ctypedef struct B:
>>    char *attrib
>>
>> ctypedef cython.fused_type(A, B) struct_t
>>
>> cdef func(struct_t mystruct, int i):
>>    cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>    ...
>>
>> What do you think?
>
> Yes, I was thinking that would be supported as well.
>
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 28 April 2011 22:31, mark florisson  wrote:
> On 28 April 2011 22:12, Robert Bradshaw  wrote:
>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>>  wrote:
>>
>>> So I fixed all that, but I'm currently wondering about the proposed
>>> cython.typeof(). I believe it currently returns a string with the type
>>> name, and not the type itself.
>>
>> Yes. This is just because there's not really anything better to return
>> at this point. We should "fix" this at some point in the future.
>>
>>> So I think it would be inconsistent to
>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>> such.
>>
>> I'm open to other suggestions, but would like an expression that
>> resolves at compile time to true/false (and we need to do branch
>> pruning on it). Note that type() is not good enough, because it has
>> different semantics, i.e.
>>
>>    cdef object o = []
>>    typeof(o), type(o)
>>
>> so lets not touch that one.
>
> Right, so for fused types I don't mind string comparison with
> cython.typeof(), but retrieval of the actual type for casts and
> declaration remains open. I'd be fine with something like
> cython.gettype().

It seems that this isn't optimized yet, but it looks to me like it
wouldn't be very hard to do so. At least == and != could be resolved
at compile time if the other operand is a string literal.

> On the other hand, the user could already do this thing by manually
> declaring another fused type that would perhaps be resolved based on
> its usage. e.g. for my original example:
>
> ctypedef char *string_t
> ctypedef cython.fused_type(int, string_t) attrib_t
>
> cdef func(struct_t mystruct, int i):
>    cdef attrib_t var = mystruct.attrib + i
>
> Is this something that should be supported anyway? Currently fused
> types can only be used when appearing as argument types (in the
> function body and as return value).
>
>>> I'm also wondering if it would be useful to allow actual type
>>> retrieval, which could be used in declarations and casts. For instance
>>> consider fusing two structs with the same attribute name but different
>>> attribute types. Perhaps in your code you want to introduce a variable
>>> compatible with such a type, e.g. consider this:
>>>
>>> ctypdef struct A:
>>>    int attrib
>>>
>>> ctypedef struct B:
>>>    char *attrib
>>>
>>> ctypedef cython.fused_type(A, B) struct_t
>>>
>>> cdef func(struct_t mystruct, int i):
>>>    cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>>    ...
>>>
>>> What do you think?
>>
>> Yes, I was thinking that would be supported as well.
>>
>> - Robert
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 28 April 2011 23:30, Robert Bradshaw  wrote:
> On Thu, Apr 28, 2011 at 1:31 PM, mark florisson
>  wrote:
>> On 28 April 2011 22:12, Robert Bradshaw  wrote:
>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>>>  wrote:
>>>
>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>> cython.typeof(). I believe it currently returns a string with the type
>>>> name, and not the type itself.
>>>
>>> Yes. This is just because there's not really anything better to return
>>> at this point. We should "fix" this at some point in the future.
>>>
>>>> So I think it would be inconsistent to
>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>> such.
>>>
>>> I'm open to other suggestions, but would like an expression that
>>> resolves at compile time to true/false (and we need to do branch
>>> pruning on it). Note that type() is not good enough, because it has
>>> different semantics, i.e.
>>>
>>>    cdef object o = []
>>>    typeof(o), type(o)
>>>
>>> so lets not touch that one.
>>
>> Right, so for fused types I don't mind string comparison with
>> cython.typeof(), but retrieval of the actual type for casts and
>> declaration remains open. I'd be fine with something like
>> cython.gettype().
>
> I'd rather re-use typeof here than introduce yet another special
> method. On the other hand, allowing parentheses in a type declaration
> requires complicating the syntax even more.
>
> I'm not sure this is needed, couldn't one do
>
>    cdef foo(fused_type a):
>        cdef fused_type b = a + func()
>
> and all instances of fused_type get specialized in the body.

Right, that is supported already.

>> On the other hand, the user could already do this thing by manually
>> declaring another fused type that would perhaps be resolved based on
>> its usage. e.g. for my original example:
>>
>> ctypedef char *string_t
>> ctypedef cython.fused_type(int, string_t) attrib_t
>>
>> cdef func(struct_t mystruct, int i):
>>    cdef attrib_t var = mystruct.attrib + i
>>
>> Is this something that should be supported anyway?
>
> OK, I take back what I said, I was looking at the RHS, not the LHS. If
> one needs to specialize in this manner, explicitly creating two
> branches should typically be enough. The same for casting. The one
> exception (perhaps) is "my_fused_type complex." Otherwise it's
> starting to feel too much like C++ template magic and complexity for
> little additional benefit.

Ok, branching on the type sounds fine to me. It brings one problem
though: because you cannot declare the variables of your variable type
(the type of say, mystruct.attrib), you will need to do multiple
declarations outside of your branches. So in my example:

cdef func(struct_t mystruct, int i):
    cdef string_t string_var
cdef int int_var

if typeof(mystruct) is typeof(int):
int_var = mystruct.attrib + i
...
else:
   string_var = mystruct.attrib + i
   ...

But this is probably not a common case, so likely not an issue.

>> Currently fused
>> types can only be used when appearing as argument types (in the
>> function body and as return value).
>
>>>> I'm also wondering if it would be useful to allow actual type
>>>> retrieval, which could be used in declarations and casts. For instance
>>>> consider fusing two structs with the same attribute name but different
>>>> attribute types. Perhaps in your code you want to introduce a variable
>>>> compatible with such a type, e.g. consider this:
>>>>
>>>> ctypdef struct A:
>>>>    int attrib
>>>>
>>>> ctypedef struct B:
>>>>    char *attrib
>>>>
>>>> ctypedef cython.fused_type(A, B) struct_t
>>>>
>>>> cdef func(struct_t mystruct, int i):
>>>>    cdef cython.gettype(mystruct.attrib) var = mystruct.attrib + i
>>>>    ...
>>>>
>>>> What do you think?
>>>
>>> Yes, I was thinking that would be supported as well.
>>>
>>> - Robert
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 28 April 2011 23:59, Robert Bradshaw  wrote:
> On Thu, Apr 28, 2011 at 2:29 PM, mark florisson
>  wrote:
>> On 28 April 2011 22:31, mark florisson  wrote:
>>> On 28 April 2011 22:12, Robert Bradshaw  
>>> wrote:
>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson
>>>>  wrote:
>>>>
>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>> name, and not the type itself.
>>>>
>>>> Yes. This is just because there's not really anything better to return
>>>> at this point. We should "fix" this at some point in the future.
>>>>
>>>>> So I think it would be inconsistent to
>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>> such.
>>>>
>>>> I'm open to other suggestions, but would like an expression that
>>>> resolves at compile time to true/false (and we need to do branch
>>>> pruning on it). Note that type() is not good enough, because it has
>>>> different semantics, i.e.
>>>>
>>>>    cdef object o = []
>>>>    typeof(o), type(o)
>>>>
>>>> so lets not touch that one.
>>>
>>> Right, so for fused types I don't mind string comparison with
>>> cython.typeof(), but retrieval of the actual type for casts and
>>> declaration remains open. I'd be fine with something like
>>> cython.gettype().
>>
>> It seems that this isn't optimized yet, but it looks to me like it
>> wouldn't be very hard to do so. At least == and != could be resolved
>> at compile time if the other operand is a string literal.
>
> Yes. We could consider supporting "typeof(x) is typeof(double)" or
> even "typeof(int*)" etc. as well to not tie ourselves to strings. Or
> perhaps some other syntax we haven't thought of. Alternatively, it
> would be nice if it involved the literal fused_type as that's what
> we're really branching on, e.g.
>
> ctypedef cython.fused_type(A, B) AorB
>
> cdef foo(AorB x):
>    if AorB[A]:   # or .A or something
>        ...
>
> it's hard to come up with something that plays nicely with pointer types.

I like the typeof(my_type_here). So if we support listing types in
typeof(), then you could also call typeof() on AorB to specialize.

> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 29 April 2011 06:32, Robert Bradshaw  wrote:
> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel  wrote:
>> mark florisson, 28.04.2011 23:29:
>>>
>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>
>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>
>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>
>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>>> name, and not the type itself.
>>>>>
>>>>> Yes. This is just because there's not really anything better to return
>>>>> at this point. We should "fix" this at some point in the future.
>>>>>
>>>>>> So I think it would be inconsistent to
>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>> such.
>>>>>
>>>>> I'm open to other suggestions, but would like an expression that
>>>>> resolves at compile time to true/false (and we need to do branch
>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>> different semantics, i.e.
>>>>>
>>>>>    cdef object o = []
>>>>>    typeof(o), type(o)
>>>>>
>>>>> so lets not touch that one.
>>>>
>>>> Right, so for fused types I don't mind string comparison with
>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>> declaration remains open. I'd be fine with something like
>>>> cython.gettype().
>>>
>>> It seems that this isn't optimized yet, but it looks to me like it
>>> wouldn't be very hard to do so. At least == and != could be resolved
>>> at compile time if the other operand is a string literal.
>>
>> Well, the obvious place where this would happen for free would be constant
>> folding. But that runs *way* before type analysis, so all it sees is the
>> typeof() call, not the string.
>
> Actually, to support code like
>
> ctypedef cython.fused_type(object, double) my_fused_type
>
> cdef foo(my_fused_type x):
>    if my_fused_type is object:
>        print x.attr
>    else:
>        cdef my_fused_type *ptr = &x
>
> we need to resolve this branch and prune "dead code" before type
> analysis, so I'm thinking it may need to be a special phase, perhaps
> part of the fused-type-specialization phase. Here's another idea:
>
> cdef foo(numeric x):
>    if numeric in floating:
>        ...
>    elif numeric is long:
>        ...
>    else:
>        ...
>    print numeric is double   # after the specialization pass, this
> would be a literal True/False node.

Hmm, that one looks pretty ok. We could also do typeof(numeric) is
typeof(floating), and if any of the fused types are unresolved it
works like 'in' instead of 'is'. Although in that case people could
also write typeof(floating) is typeof(numeric) which would behave like
'in' with reversed operands :)

> Again, this would all be resolved before type analysis. In terms of
> syntactically supporting pointers, we could either support "fused_type
> is typeof(void*)" or require typedefs for types not representable by
> an ExprNode. (This would make declaring fused types syntactically
> simpler as well...) I prefer the latter.

Me too, cython.fused_type() currently just parses identifiers only.

> - Robert
>
>
>> Optimising this would thus basically require a second (simpler?) constant
>> folding step, or at least an additional hook during (or after) type analysis
>> that does the comparison. I wouldn't mind too much, given that type analysis
>> can end up injecting some rather handy information into the tree. So a
>> second (even complete) constant folding step *after* type analysis may still
>> introduce some nice optimisations.
>>
>> We may actually consider splitting constant folding into two steps - one
>> that calculates the results as early as possible (and maybe does only
>> safe&obvious tree replacements, e.g. boolean values, strings, etc.) and a
>> second step that replaces subtrees by constant nodes once it knows the final
>> type of the result.
>>
>> Stefan
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 29 April 2011 11:03, Pauli Virtanen  wrote:
> Fri, 29 Apr 2011 10:23:55 +0200, mark florisson wrote:
> [clip]
>> Ok, branching on the type sounds fine to me. It brings one problem
>> though: because you cannot declare the variables of your variable type
>> (the type of say, mystruct.attrib), you will need to do multiple
>> declarations outside of your branches. So in my example:
>>
>> cdef func(struct_t mystruct, int i):
>>     cdef string_t string_var
>>     cdef int int_var
>>
>>     if typeof(mystruct) is typeof(int):
>>         int_var = mystruct.attrib + i
>>         ...
>>     else:
>>        string_var = mystruct.attrib + i
>>        ...
>>
>> But this is probably not a common case, so likely not an issue.
>
> Are you planning to special-case the "real_t complex" syntax? Shooting
> from the sidelines, one more generic solution might be, e.g.,

I'm sorry, I'm not sure what syntax you are referring to. Are you
talking about actual complex numbers?

>        ctypedef cython.fused_type(A, B) struct_t
>        ctypedef cython.fused_type(float, double, paired=struct_t) real_t
>        ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>
> and just restrict the specialization to cases that make sense.

The paired means you're declaring types of attributes?

> --
> Pauli Virtanen
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 29 April 2011 12:28, Pauli Virtanen  wrote:
> Fri, 29 Apr 2011 11:30:19 +0200, mark florisson wrote:
>> On 29 April 2011 11:03, Pauli Virtanen  wrote:
> [clip]
>>> Are you planning to special-case the "real_t complex" syntax? Shooting
>>> from the sidelines, one more generic solution might be, e.g.,
>>
>> I'm sorry, I'm not sure what syntax you are referring to. Are you
>> talking about actual complex numbers?
>
> This:
>
> On 28 April 2011 23:30, Robert Bradshaw 
> wrote:
>> OK, I take back what I said, I was looking at the RHS, not the LHS. If
>> one needs to specialize in this manner, explicitly creating two
>> branches should typically be enough. The same for casting. The one
>> exception (perhaps) is "my_fused_type complex." Otherwise it's
>> starting to feel too much like C++ template magic and complexity for
>> little additional benefit.
>
> That is, declaring a complex type matching a real one.

Ah, I see what you mean now.

>>>        ctypedef cython.fused_type(A, B) struct_t
>>>        ctypedef cython.fused_type(float, double, paired=struct_t) real_t
>>>        ctypedef cython.fused_type(int_t, string_t, paired=struct_t) var_t
>>>
>>> and just restrict the specialization to cases that make sense.
>>
>> The paired means you're declaring types of attributes?
>
> No, just that real_t is specialized to float whenever struct_t is specialized
> to A and to double when B. Or a more realistic example,
>
>        ctypedef cython.fused_type(float, double) real_t
>        ctypedef cython.fused_type(float complex, double complex) complex_t
>
>        cdef real_plus_one(complex_t a):
>            real_t b = a.real
>            return b + 1
>
> which I suppose would not be a very unusual thing in numerical codes.
> This would also allow writing the case you had earlier as
>
>        cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>
>        cdef func(struct_t mystruct, int i):
>            cdef attr_t var
>
>            if typeof(mystruct) is typeof(int):
>                var = mystruct.attrib + i
>                ...
>            else:
>                var = mystruct.attrib + i
>                ...
>
> Things would need to be done explicitly instead of implicitly, though,
> but it would remove the need for any special handling of
> the "complex" keyword.

I see, so it's like a mapping. So, I didn't realize that you can't do this:

def func(arbitrary_type complex x):
...

But if we just allow that for fused types, then couldn't we simply do

ctypedef cython.fused_type(float, double) real_t

cdef real_plus_one(real_t complex a):
real_t b = a.real
return b + 1

? Then you don't need to pair anything. Perhaps we could introduce
real_t as a type, just like numeric and floating. So I guess
special-casing complex sounds fine with me. Perhaps real_t should be
builtin (not as an attribute of the Cython module), so the parser can
just recognize it immediately?

> --
> Pauli Virtanen
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 29 April 2011 13:37, Pauli Virtanen  wrote:
> Fri, 29 Apr 2011 12:53:06 +0200, mark florisson wrote:
> [clip]
>> But if we just allow that for fused types, then couldn't we simply do
>>
>> ctypedef cython.fused_type(float, double) real_t
>>
>> cdef real_plus_one(real_t complex a):
>>     real_t b = a.real
>>     return b + 1
>>
>> ? Then you don't need to pair anything. Perhaps we could introduce
>> real_t as a type, just like numeric and floating. So I guess
>> special-casing complex sounds fine with me.
>
> Special-casing for complex sounds ugly to me; "complex float"
> is a type name in the same way as "long int" is, and writing
> "real_t complex" feels a bit like writing "int_t long".
>
> But of course, if this feature is really only needed for declaring
> complex variants of real types, then how it is done doesn't matter
> so much. But, IMHO, such a special casing would be a somewhat weird
> language feature.

Hmm, indeed, it's pretty weird. I'm fine with the pairing also,
although I'm still not sure how common this case is, and if we really
want to support it. Wouldn't good old C promotion work for this? e.g.
if the type is either float or double, just declare your variable
double?

> --
> Pauli Virtanen
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 29 April 2011 06:32, Robert Bradshaw  wrote:
> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel  wrote:
>> mark florisson, 28.04.2011 23:29:
>>>
>>> On 28 April 2011 22:31, mark florisson wrote:
>>>>
>>>> On 28 April 2011 22:12, Robert Bradshaw wrote:
>>>>>
>>>>> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote:
>>>>>
>>>>>> So I fixed all that, but I'm currently wondering about the proposed
>>>>>> cython.typeof(). I believe it currently returns a string with the type
>>>>>> name, and not the type itself.
>>>>>
>>>>> Yes. This is just because there's not really anything better to return
>>>>> at this point. We should "fix" this at some point in the future.
>>>>>
>>>>>> So I think it would be inconsistent to
>>>>>> suddenly start allowing comparison with 'is' and 'isinstance' and
>>>>>> such.
>>>>>
>>>>> I'm open to other suggestions, but would like an expression that
>>>>> resolves at compile time to true/false (and we need to do branch
>>>>> pruning on it). Note that type() is not good enough, because it has
>>>>> different semantics, i.e.
>>>>>
>>>>>    cdef object o = []
>>>>>    typeof(o), type(o)
>>>>>
>>>>> so lets not touch that one.
>>>>
>>>> Right, so for fused types I don't mind string comparison with
>>>> cython.typeof(), but retrieval of the actual type for casts and
>>>> declaration remains open. I'd be fine with something like
>>>> cython.gettype().
>>>
>>> It seems that this isn't optimized yet, but it looks to me like it
>>> wouldn't be very hard to do so. At least == and != could be resolved
>>> at compile time if the other operand is a string literal.
>>
>> Well, the obvious place where this would happen for free would be constant
>> folding. But that runs *way* before type analysis, so all it sees is the
>> typeof() call, not the string.
>
> Actually, to support code like
>
> ctypedef cython.fused_type(object, double) my_fused_type
>
> cdef foo(my_fused_type x):
>    if my_fused_type is object:
>        print x.attr
>    else:
>        cdef my_fused_type *ptr = &x
>
> we need to resolve this branch and prune "dead code" before type
> analysis, so I'm thinking it may need to be a special phase, perhaps
> part of the fused-type-specialization phase. Here's another idea:
>
> cdef foo(numeric x):
>    if numeric in floating:
>        ...
>    elif numeric is long:
>        ...
>    else:
>        ...
>    print numeric is double   # after the specialization pass, this
> would be a literal True/False node.
>
> Again, this would all be resolved before type analysis. In terms of
> syntactically supporting pointers, we could either support "fused_type
> is typeof(void*)" or require typedefs for types not representable by
> an ExprNode. (This would make declaring fused types syntactically
> simpler as well...) I prefer the latter.
>
> - Robert
>

I made both things work, you can use both typeof() on expressions and
you can compare types using 'is' and 'in'. However, with typeof() it
doesn't remove branches, so if you're doing incompatible stuff you
need to do the type matching. It could be supported, but currently it
isn't because TransformBuiltinMethods runs after
AnalyseDeclarationsTransform (for which the reason is not immediately
apparent).

With the type matching it matches on exactly 'if src_type is
dst_type:' so you can't use 'and' and such... perhaps I should turn
these expression into a node with the constant value first and then
see if the result of the entire expression is known at compile time?
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 30 April 2011 09:51, Dag Sverre Seljebotn  wrote:
> On 04/30/2011 08:39 AM, Robert Bradshaw wrote:
>>
>> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson
>>   wrote:
>>>
>>> On 29 April 2011 12:28, Pauli Virtanen  wrote:
>>>>
>>>> No, just that real_t is specialized to float whenever struct_t is
>>>> specialized
>>>> to A and to double when B. Or a more realistic example,
>>>>
>>>>        ctypedef cython.fused_type(float, double) real_t
>>>>        ctypedef cython.fused_type(float complex, double complex)
>>>> complex_t
>>>>
>>>>        cdef real_plus_one(complex_t a):
>>>>            real_t b = a.real
>>>>            return b + 1
>>>>
>>>> which I suppose would not be a very unusual thing in numerical codes.
>>>> This would also allow writing the case you had earlier as
>>>>
>>>>        cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>>>
>>>>        cdef func(struct_t mystruct, int i):
>>>>            cdef attr_t var
>>>>
>>>>            if typeof(mystruct) is typeof(int):
>>>>                var = mystruct.attrib + i
>>>>                ...
>>>>            else:
>>>>                var = mystruct.attrib + i
>>>>                ...
>>>>
>>>> Things would need to be done explicitly instead of implicitly, though,
>>>> but it would remove the need for any special handling of
>>>> the "complex" keyword.
>>
>> If we're going to introduce pairing, another option would be
>>
>>     ctypedef fused_type((double complex, double), (float complex,
>> float)) (complex_t, real_t)
>>
>> though I'm not sure I like that either. We're not trying to create the
>> all-powerful templating system here, and anything that can be done
>> with pairing can be done (though less elegantly) via branching on the
>> types, or, as Pauli mentions, using a wider type is often (but not
>> always) a viable option.
>
> Keeping the right balance is difficult. But, at least there's some cases of
> needing this in various codebases when interfacing with LAPACK.
>
> Most uses of templating with Cython code I've seen so far does a similar
> kind of "zip" as what you have above (as we discussed on the workshop). So
> at least the usage pattern you write above is very common.
>
> float32 is not about to disappear, it really is twice as fast when you're
> memory IO bound.
>
> Using a wider type is actually quite often not possible; any time the type
> is involved as the base type of an array it is not possible, and that's a
> pretty common case.

Well, if the array is passed into the function directly (and not e.g.
as an attribute of something passed in), then you can just write
'my_fused_type *array' or 'my_fused_type array[]', and the base type
will be available as 'my_fused_type'.

> (With LAPACK you take the address of the variable and
> pass it to Fortran, so using a wider type is not possible there either,
> although I'll agree that's a more remote case.)
>
> My proposal: Don't support either "real_t complex" or paired fused types for
> the time being. Then see.

Ok, sounds good.

> But my vote is for paired fused types instead of "real_t complex".
>
> Dag Sverre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

A remaining issue which I'm not quite certain about is the
specialization through subscripts, e.g. func[double]. How should this
work from Python space (assuming cpdef functions)? Would we want to
pass in cython.double etc? Because it would only work for builtin
types, so what about types that aren't exposed to Python but can still
be coerced to and from Python? Perhaps it would be better to pass in
strings instead. I also think e.g. "int *" reads better than
cython.pointer(cython.int).

It also sounds bad to rely on objects from the Shadow module, as
people using Cython modules may not have Cython (or the Shadow module
shipped as cython) installed. And what happens if they import it under
a different name and we import another cython module? Perhaps the
specializations of the signature could also be exposed on the function
object, so users can see which ones are available.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 2 May 2011 11:08, Stefan Behnel  wrote:
> Robert Bradshaw, 30.04.2011 08:16:
>>
>> On Fri, Apr 29, 2011 at 8:04 AM, mark florisson
>>>
>>> With the type matching it matches on exactly 'if src_type is
>>> dst_type:' so you can't use 'and' and such... perhaps I should turn
>>> these expression into a node with the constant value first and then
>>> see if the result of the entire expression is known at compile time?
>>
>> Yes, that's exactly what I was thinking you should do.
>
> For now, you could simply (try to) re-run the ConstantFolding transform
> after the type splitting. It will evaluate constant expressions and also
> drop unreachable branches from conditionals.

Right thanks! I actually ran it on the condition only, as I overlooked
its branch pruning capability (sometimes I see 'if (1)' generated
code, so that's probably because that happens after ConstantFolding).
So wouldn't it be a good idea to just insert another ConstantFolding
transform at the end of the pipeline also, and have it recalculate
constants in case it was previously determined not_a_constant?

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 2 May 2011 12:32, Stefan Behnel  wrote:
> mark florisson, 02.05.2011 11:22:
>>
>> On 2 May 2011 11:08, Stefan Behnel wrote:
>>>
>>> Robert Bradshaw, 30.04.2011 08:16:
>>>>
>>>> On Fri, Apr 29, 2011 at 8:04 AM, mark florisson
>>>>>
>>>>> With the type matching it matches on exactly 'if src_type is
>>>>> dst_type:' so you can't use 'and' and such... perhaps I should turn
>>>>> these expression into a node with the constant value first and then
>>>>> see if the result of the entire expression is known at compile time?
>>>>
>>>> Yes, that's exactly what I was thinking you should do.
>>>
>>> For now, you could simply (try to) re-run the ConstantFolding transform
>>> after the type splitting. It will evaluate constant expressions and also
>>> drop unreachable branches from conditionals.
>>
>> Right thanks! I actually ran it on the condition only, as I overlooked
>> its branch pruning capability (sometimes I see 'if (1)' generated
>> code, so that's probably because that happens after ConstantFolding).
>
> ... or because something generates that explicitly (e.g. to simplify some
> specific code generation), but I couldn't find this with a quick grep right
> now. I once put a C for-loop into straight sequential argument unpacking
> code, so that I could "break" out of it at any point, rather than using
> gotos and labels for that. But that was only half-way hackish because that
> particular code mimics an unrolled loop anyway.
>
> In the future, Vitja's dead code removal branch will handle branch pruning
> anyway. I just put it into the ConstantFolding transform at the time as it
> fit in there quite well, and because I thought it would help also during the
> constant calculation to get rid of dead code while we're at it, e.g. in
> conditional expressions (x if c else y) or later on in combination with
> things like the FlattenInList/SwitchTransform.
>
>
>> So wouldn't it be a good idea to just insert another ConstantFolding
>> transform at the end of the pipeline also, and have it recalculate
>> constants in case it was previously determined not_a_constant?
>
> An additional run would also make sense before the optimisation steps that
> run after type analysis. Some of them can take advantage of constants,
> including those created before and during type analysis. Maybe even a third
> run right before the code generation, i.e. after the optimisations.
>
> Not sure if "not_a_constant" values can become a problem. They'll be ignored
> in the next run, only "constant_value_not_set" values will be recalculated.
> While newly created expressions will usually be virgins, new
> values/expressions that are being added to existing "not_a_constant"
> expressions may end up being ignored, even if the whole expression could
> then be condensed into a constant.

Right, in my branch I introduced a new instance variable, and if
that's true it ignores "not_a_constant" and tries to compute it
anyway, as the pruning is mandatory and at the expression was
determined "not_a_constant".

> OTOH, these cases may be rare enough to not deserve the additional
> performance penalty of looking at all expression nodes again. It may be
> better to call back into ConstantFolding explicitly when generating
> expressions at a later point, or to just calculate and set the constant
> values properly while creating the nodes.

So far (from what I've seen, I haven't actually done any benchmarks) I
think the C compiler is still like 5 times as slow as Cython.

> While you're at it, you can test if it works to reuse the same transform
> instance for all runs. That should be a tiny bit quicker, because it only
> needs to build the node dispatch table once (not that I expect it to make
> any difference, but anyway...).

Ok, sure.

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 2 May 2011 18:24, Robert Bradshaw  wrote:
> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>  wrote:
>> On 30 April 2011 09:51, Dag Sverre Seljebotn  
>> wrote:
>>> On 04/30/2011 08:39 AM, Robert Bradshaw wrote:
>>>>
>>>> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson
>>>>   wrote:
>>>>>
>>>>> On 29 April 2011 12:28, Pauli Virtanen  wrote:
>>>>>>
>>>>>> No, just that real_t is specialized to float whenever struct_t is
>>>>>> specialized
>>>>>> to A and to double when B. Or a more realistic example,
>>>>>>
>>>>>>        ctypedef cython.fused_type(float, double) real_t
>>>>>>        ctypedef cython.fused_type(float complex, double complex)
>>>>>> complex_t
>>>>>>
>>>>>>        cdef real_plus_one(complex_t a):
>>>>>>            real_t b = a.real
>>>>>>            return b + 1
>>>>>>
>>>>>> which I suppose would not be a very unusual thing in numerical codes.
>>>>>> This would also allow writing the case you had earlier as
>>>>>>
>>>>>>        cdef cython.fused_type(string_t, int, paired=struct_t) attr_t
>>>>>>
>>>>>>        cdef func(struct_t mystruct, int i):
>>>>>>            cdef attr_t var
>>>>>>
>>>>>>            if typeof(mystruct) is typeof(int):
>>>>>>                var = mystruct.attrib + i
>>>>>>                ...
>>>>>>            else:
>>>>>>                var = mystruct.attrib + i
>>>>>>                ...
>>>>>>
>>>>>> Things would need to be done explicitly instead of implicitly, though,
>>>>>> but it would remove the need for any special handling of
>>>>>> the "complex" keyword.
>>>>
>>>> If we're going to introduce pairing, another option would be
>>>>
>>>>     ctypedef fused_type((double complex, double), (float complex,
>>>> float)) (complex_t, real_t)
>>>>
>>>> though I'm not sure I like that either. We're not trying to create the
>>>> all-powerful templating system here, and anything that can be done
>>>> with pairing can be done (though less elegantly) via branching on the
>>>> types, or, as Pauli mentions, using a wider type is often (but not
>>>> always) a viable option.
>>>
>>> Keeping the right balance is difficult. But, at least there's some cases of
>>> needing this in various codebases when interfacing with LAPACK.
>>>
>>> Most uses of templating with Cython code I've seen so far does a similar
>>> kind of "zip" as what you have above (as we discussed on the workshop). So
>>> at least the usage pattern you write above is very common.
>>>
>>> float32 is not about to disappear, it really is twice as fast when you're
>>> memory IO bound.
>>>
>>> Using a wider type is actually quite often not possible; any time the type
>>> is involved as the base type of an array it is not possible, and that's a
>>> pretty common case.
>>
>> Well, if the array is passed into the function directly (and not e.g.
>> as an attribute of something passed in), then you can just write
>> 'my_fused_type *array' or 'my_fused_type array[]', and the base type
>> will be available as 'my_fused_type'.
>>
>>> (With LAPACK you take the address of the variable and
>>> pass it to Fortran, so using a wider type is not possible there either,
>>> although I'll agree that's a more remote case.)
>>>
>>> My proposal: Don't support either "real_t complex" or paired fused types for
>>> the time being. Then see.
>>
>> Ok, sounds good.
>>
>>> But my vote is for paired fused types instead of "real_t complex".
>>>
>>> Dag Sverre
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> A remaining issue which I'm not quite certain about is the
>> specialization through subscripts, e.g. func[double]. How should this
>> work from Python space (assuming cpdef functions)? Would we want to
>&

Re: [Cython] Fused Types

[mark florisson]

On 3 May 2011 00:21, Robert Bradshaw  wrote:
> On Mon, May 2, 2011 at 1:56 PM, mark florisson
>  wrote:
>> On 2 May 2011 18:24, Robert Bradshaw  wrote:
>>> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>>>  wrote:
>>>> A remaining issue which I'm not quite certain about is the
>>>> specialization through subscripts, e.g. func[double]. How should this
>>>> work from Python space (assuming cpdef functions)? Would we want to
>>>> pass in cython.double etc? Because it would only work for builtin
>>>> types, so what about types that aren't exposed to Python but can still
>>>> be coerced to and from Python? Perhaps it would be better to pass in
>>>> strings instead. I also think e.g. "int *" reads better than
>>>> cython.pointer(cython.int).
>>>
>>> That's whey we offer cython.p_int. On that note, we should support
>>> cython.astype("int *") or something like that. Generally, I don't like
>>> encoding semantic information in strings.
>>>
>>> OTHO, since it'll be a mapping of some sort, there's no reason we
>>> can't support both. Most of the time it should dispatch (at runtime or
>>> compile time) based on the type of the arguments.
>>
>> If we have an argument type that is composed of a fused type, would be
>> want the indexing to specify the composed type or the fused type? e.g.
>>
>> ctypedef floating *floating_p
>
> How should we support this? It's clear in this case, but only because
> you chose good names. Another option would be to require
> parameterization floating_p, with floating_p[floating] the
> "as-yet-unparameterized" version. Explicit but redundant. (The same
> applies to struct as classes as well as typedefs.) On the other had,
> the above is very succinct and clear in context, so I'm leaning
> towards it. Thoughts?

Well, it is already supported. floating is fused, so any composition
of floating is also fused.

>> cdef func(floating_p x):
>>    ...
>>
>> Then do we want
>>
>>    func[double](10.0)
>>
>> or
>>
>>    func[double_p](10.0)
>>
>> to specialize func?
>
> The latter.

I'm really leaning towards the former. What if you write

cdef func(floating_p x, floating_p *y):
...

Then specializing floating_p using double_p sounds slightly
nonsensical, as you're also specializing floating_p *.

>> FYI, the type checking works like 'double_p is
>> floating_p' and not 'double is floating_p'. But for functions this is
>> a little different. On the one hand specifying the full types
>> (double_p) makes sense as you're kind of specifying a signature, but
>> on the other hand you're specializing fused types and you don't care
>> how they are composed -- especially if they occur multiple times with
>> different composition. So I'm thinking we want 'func[double]'.
>
> That's what I'm thinking too. The type you're branching on is
> floating, and withing that block you can declare variables as
> floating*, ndarray[dtype=floating], etc.

What I actually meant there was "I think we want func[double] for the
func(floating_p x) signature".

Right, people can already say 'cdef func(floating *p): ...' and then
use 'floating'. However, if you do 'cdef floating_p x): ...', then
'floating' is not available, only 'floating_p'. It would be rather
trivial to also support 'floating' in the latter case, which I think
we should, unless you are adamant about prohibiting regular typedefs
of fused types.

> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 3 May 2011 10:07, Dag Sverre Seljebotn  wrote:
> On 05/03/2011 09:59 AM, mark florisson wrote:
>>
>> On 3 May 2011 00:21, Robert Bradshaw  wrote:
>>>
>>> On Mon, May 2, 2011 at 1:56 PM, mark florisson
>>>   wrote:
>>>>
>>>> On 2 May 2011 18:24, Robert Bradshaw
>>>>  wrote:
>>>>>
>>>>> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>>>>>   wrote:
>>>>>>
>>>>>> A remaining issue which I'm not quite certain about is the
>>>>>> specialization through subscripts, e.g. func[double]. How should this
>>>>>> work from Python space (assuming cpdef functions)? Would we want to
>>>>>> pass in cython.double etc? Because it would only work for builtin
>>>>>> types, so what about types that aren't exposed to Python but can still
>>>>>> be coerced to and from Python? Perhaps it would be better to pass in
>>>>>> strings instead. I also think e.g. "int *" reads better than
>>>>>> cython.pointer(cython.int).
>>>>>
>>>>> That's whey we offer cython.p_int. On that note, we should support
>>>>> cython.astype("int *") or something like that. Generally, I don't like
>>>>> encoding semantic information in strings.
>>>>>
>>>>> OTHO, since it'll be a mapping of some sort, there's no reason we
>>>>> can't support both. Most of the time it should dispatch (at runtime or
>>>>> compile time) based on the type of the arguments.
>>>>
>>>> If we have an argument type that is composed of a fused type, would be
>>>> want the indexing to specify the composed type or the fused type? e.g.
>>>>
>>>> ctypedef floating *floating_p
>>>
>>> How should we support this? It's clear in this case, but only because
>>> you chose good names. Another option would be to require
>>> parameterization floating_p, with floating_p[floating] the
>>> "as-yet-unparameterized" version. Explicit but redundant. (The same
>>> applies to struct as classes as well as typedefs.) On the other had,
>>> the above is very succinct and clear in context, so I'm leaning
>>> towards it. Thoughts?
>>
>> Well, it is already supported. floating is fused, so any composition
>> of floating is also fused.
>>
>>>> cdef func(floating_p x):
>>>>    ...
>>>>
>>>> Then do we want
>>>>
>>>>    func[double](10.0)
>>>>
>>>> or
>>>>
>>>>    func[double_p](10.0)
>>>>
>>>> to specialize func?
>>>
>>> The latter.
>>
>> I'm really leaning towards the former. What if you write
>>
>> cdef func(floating_p x, floating_p *y):
>>     ...
>>
>> Then specializing floating_p using double_p sounds slightly
>> nonsensical, as you're also specializing floating_p *.
>
> I made myself agree with both of you in turn, but in the end I think I'm
> with Robert here.
>
> Robert's approach sounds perhaps slightly simpler if you think of it this
> way:
>
> ctypedef fused_type(float, double) floating
> ctypedef floating* floating_p
>
> is really a short-hand for
>
> ctypedef fused_type(float*, double*) floating_p
>
> I.e., when using a fused_type in a typedef you simply get a new fused_type.
> This sounds in a sense simpler without extra complexity getting in the way
> ("which was my fused base type again...").
>
> Dag SVerre
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Ok, if those typedefs should be disallowed then specialization through
indexing should then definitely get the types listed in the fused_type
typedef.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 3 May 2011 10:44, Dag Sverre Seljebotn  wrote:
> On 05/03/2011 10:42 AM, mark florisson wrote:
>>
>> On 3 May 2011 10:07, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 05/03/2011 09:59 AM, mark florisson wrote:
>>>>
>>>> On 3 May 2011 00:21, Robert Bradshaw
>>>>  wrote:
>>>>>
>>>>> On Mon, May 2, 2011 at 1:56 PM, mark florisson
>>>>>     wrote:
>>>>>>
>>>>>> On 2 May 2011 18:24, Robert Bradshaw
>>>>>>  wrote:
>>>>>>>
>>>>>>> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>>>>>>>     wrote:
>>>>>>>>
>>>>>>>> A remaining issue which I'm not quite certain about is the
>>>>>>>> specialization through subscripts, e.g. func[double]. How should
>>>>>>>> this
>>>>>>>> work from Python space (assuming cpdef functions)? Would we want to
>>>>>>>> pass in cython.double etc? Because it would only work for builtin
>>>>>>>> types, so what about types that aren't exposed to Python but can
>>>>>>>> still
>>>>>>>> be coerced to and from Python? Perhaps it would be better to pass in
>>>>>>>> strings instead. I also think e.g. "int *" reads better than
>>>>>>>> cython.pointer(cython.int).
>>>>>>>
>>>>>>> That's whey we offer cython.p_int. On that note, we should support
>>>>>>> cython.astype("int *") or something like that. Generally, I don't
>>>>>>> like
>>>>>>> encoding semantic information in strings.
>>>>>>>
>>>>>>> OTHO, since it'll be a mapping of some sort, there's no reason we
>>>>>>> can't support both. Most of the time it should dispatch (at runtime
>>>>>>> or
>>>>>>> compile time) based on the type of the arguments.
>>>>>>
>>>>>> If we have an argument type that is composed of a fused type, would be
>>>>>> want the indexing to specify the composed type or the fused type? e.g.
>>>>>>
>>>>>> ctypedef floating *floating_p
>>>>>
>>>>> How should we support this? It's clear in this case, but only because
>>>>> you chose good names. Another option would be to require
>>>>> parameterization floating_p, with floating_p[floating] the
>>>>> "as-yet-unparameterized" version. Explicit but redundant. (The same
>>>>> applies to struct as classes as well as typedefs.) On the other had,
>>>>> the above is very succinct and clear in context, so I'm leaning
>>>>> towards it. Thoughts?
>>>>
>>>> Well, it is already supported. floating is fused, so any composition
>>>> of floating is also fused.
>>>>
>>>>>> cdef func(floating_p x):
>>>>>>    ...
>>>>>>
>>>>>> Then do we want
>>>>>>
>>>>>>    func[double](10.0)
>>>>>>
>>>>>> or
>>>>>>
>>>>>>    func[double_p](10.0)
>>>>>>
>>>>>> to specialize func?
>>>>>
>>>>> The latter.
>>>>
>>>> I'm really leaning towards the former. What if you write
>>>>
>>>> cdef func(floating_p x, floating_p *y):
>>>>     ...
>>>>
>>>> Then specializing floating_p using double_p sounds slightly
>>>> nonsensical, as you're also specializing floating_p *.
>>>
>>> I made myself agree with both of you in turn, but in the end I think I'm
>>> with Robert here.
>>>
>>> Robert's approach sounds perhaps slightly simpler if you think of it this
>>> way:
>>>
>>> ctypedef fused_type(float, double) floating
>>> ctypedef floating* floating_p
>>>
>>> is really a short-hand for
>>>
>>> ctypedef fused_type(float*, double*) floating_p
>>>
>>> I.e., when using a fused_type in a typedef you simply get a new
>>> fused_type.
>>> This sounds in a sense simpler without extra complexity getting in the
>>> way
>>> ("which was my fused base type again...").
>>>
>>> Dag SVerre
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>
>> Ok, if those typedefs should be disallowed then specialization through
>> indexing should then definitely get the types listed in the fused_type
>> typedef.
>
> I'm not sure what you mean here. What is disallowed exactly?

ctypedef cython.fused_type(float, double) floating
ctypedef floating *floating_p

That is what you meant right? Because prohibiting that makes it easier
to see where a type is variable (as the entire type always is, and not
some base type of it).

> DS
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[mark florisson]

On 3 May 2011 10:57, Dag Sverre Seljebotn  wrote:
> On 05/03/2011 10:49 AM, mark florisson wrote:
>>
>> On 3 May 2011 10:44, Dag Sverre Seljebotn
>>  wrote:
>>>
>>> On 05/03/2011 10:42 AM, mark florisson wrote:
>>>>
>>>> On 3 May 2011 10:07, Dag Sverre Seljebotn
>>>>  wrote:
>>>>>
>>>>> On 05/03/2011 09:59 AM, mark florisson wrote:
>>>>>>
>>>>>> On 3 May 2011 00:21, Robert Bradshaw
>>>>>>  wrote:
>>>>>>>
>>>>>>> On Mon, May 2, 2011 at 1:56 PM, mark florisson
>>>>>>>       wrote:
>>>>>>>>
>>>>>>>> On 2 May 2011 18:24, Robert Bradshaw
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> On Sun, May 1, 2011 at 2:38 AM, mark florisson
>>>>>>>>>       wrote:
>>>>>>>>>>
>>>>>>>>>> A remaining issue which I'm not quite certain about is the
>>>>>>>>>> specialization through subscripts, e.g. func[double]. How should
>>>>>>>>>> this
>>>>>>>>>> work from Python space (assuming cpdef functions)? Would we want
>>>>>>>>>> to
>>>>>>>>>> pass in cython.double etc? Because it would only work for builtin
>>>>>>>>>> types, so what about types that aren't exposed to Python but can
>>>>>>>>>> still
>>>>>>>>>> be coerced to and from Python? Perhaps it would be better to pass
>>>>>>>>>> in
>>>>>>>>>> strings instead. I also think e.g. "int *" reads better than
>>>>>>>>>> cython.pointer(cython.int).
>>>>>>>>>
>>>>>>>>> That's whey we offer cython.p_int. On that note, we should support
>>>>>>>>> cython.astype("int *") or something like that. Generally, I don't
>>>>>>>>> like
>>>>>>>>> encoding semantic information in strings.
>>>>>>>>>
>>>>>>>>> OTHO, since it'll be a mapping of some sort, there's no reason we
>>>>>>>>> can't support both. Most of the time it should dispatch (at runtime
>>>>>>>>> or
>>>>>>>>> compile time) based on the type of the arguments.
>>>>>>>>
>>>>>>>> If we have an argument type that is composed of a fused type, would
>>>>>>>> be
>>>>>>>> want the indexing to specify the composed type or the fused type?
>>>>>>>> e.g.
>>>>>>>>
>>>>>>>> ctypedef floating *floating_p
>>>>>>>
>>>>>>> How should we support this? It's clear in this case, but only because
>>>>>>> you chose good names. Another option would be to require
>>>>>>> parameterization floating_p, with floating_p[floating] the
>>>>>>> "as-yet-unparameterized" version. Explicit but redundant. (The same
>>>>>>> applies to struct as classes as well as typedefs.) On the other had,
>>>>>>> the above is very succinct and clear in context, so I'm leaning
>>>>>>> towards it. Thoughts?
>>>>>>
>>>>>> Well, it is already supported. floating is fused, so any composition
>>>>>> of floating is also fused.
>>>>>>
>>>>>>>> cdef func(floating_p x):
>>>>>>>>    ...
>>>>>>>>
>>>>>>>> Then do we want
>>>>>>>>
>>>>>>>>    func[double](10.0)
>>>>>>>>
>>>>>>>> or
>>>>>>>>
>>>>>>>>    func[double_p](10.0)
>>>>>>>>
>>>>>>>> to specialize func?
>>>>>>>
>>>>>>> The latter.
>>>>>>
>>>>>> I'm really leaning towards the former. What if you write
>>>>>>
>>>>>> cdef func(floating_p x, floating_p *y):
>>>>>>     ...
>>>>>>
>>>>>> Then specializing floating_p u

Re: [Cython] Fused Types

[mark florisson]

On 3 May 2011 07:47, Greg Ewing  wrote:
> I'm a bit confused about how fused types combine to
> create further fused types. If you have something
> like
>
>  ctypedef struct Vector:
>    floating x
>    floating y
>    floating z
>
> then is it going to generate code for all possible
> combinations of types for x, y and z?
>
> That's probably not what the user intended -- it's
> more likely that he wants *two* versions of type
> Vector, one with all floats and one with all doubles.
> But without explicit type parameters, there's no way
> to express that.
>
> I worry that you're introducing a parameterised type
> system in a half-baked way here without properly
> thinking through all the implications.
>
> --
> Greg
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>

Currently you cannot use them at all in structs, only as cdef function
argument types. Then if you have

cdef func(floating x, floating y):
...

you get a "float, float" version, and a "double, double" version, but
not "float, double" or "double, float". When and if support for
structs is there, it will work the same way as with functions.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel