Re: [Cython] Build module script
On 21 April 2011 08:57, Vitja Makarov wrote: > Now we have cythonrun build script, may be it's time to create script > for easy module building? > > -- > vitja. > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > We have cythonize(): http://wiki.cython.org/enhancements/distutils_preprocessing . Is something like that what you mean? Unfortunately I believe it doesn't parse sys.argv (yet). Perhaps a recursive option would also be useful. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Build module script
2011/4/21 mark florisson : > On 21 April 2011 08:57, Vitja Makarov wrote: >> Now we have cythonrun build script, may be it's time to create script >> for easy module building? >> >> -- >> vitja. >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > We have cythonize(): > http://wiki.cython.org/enhancements/distutils_preprocessing . Is > something like that what you mean? Unfortunately I believe it doesn't > parse sys.argv (yet). Perhaps a recursive option would also be useful. Not exactly cythonize is for distutils and friends. I mean simple script that run cython compiler and then build shared python module. It's much better then manually run cython then gcc or write makefile for this. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Mon, Apr 18, 2011 at 7:51 AM, mark florisson wrote: > On 18 April 2011 16:41, Dag Sverre Seljebotn > wrote: >> Excellent! Sounds great! (as I won't have my laptop for some days I can't >> have a look yet but I will later) >> >> You're right about (the current) buffers and the gil. A testcase explicitly >> for them would be good. >> >> Firstprivate etc: i think it'd be nice myself, but it is probably better to >> take a break from it at this point so that we can think more about that and >> not do anything rash; perhaps open up a specific thread on them and ask for >> more general input. Perhaps you want to take a break or task-switch to >> something else (fused types?) until I can get around to review and merge >> what you have so far? You'll know best what works for you though. If you >> decide to implement explicit threadprivate variables because you've got the >> flow I certainly wom't object myself. >> > Ok, cool, I'll move on :) I already included a test with a prange and > a numpy buffer with indexing. Wow, you're just plowing away at this. Very cool. +1 to disallowing nested prange, that seems to get really messy with little benefit. In terms of the CEP, I'm still unconvinced that firstprivate is not safe to infer, but lets leave the initial values undefined rather than specifying them to be NaNs (we can do that as an implementation if you want), which will give us flexibility to change later once we've had a chance to play around with it. The "cdef threadlocal(int) foo" declaration syntax feels odd to me... We also probably want some way of explicitly marking a variable as shared and still be able to assign to/flush/sync it. Perhaps the parallel context could be used for these declarations, i.e. with parallel(threadlocal=a, shared=(b,c)): ... which would be considered an "expert" usecase. For all the discussion of threadsavailable/threadid, the most common usecase I see is for allocating a large shared buffer and partitioning it. This seems better handled by allocating separate thread-local buffers, no? I still like the context idea, but everything in a parallel block before and after the loop(s) also seems like a natural place to put any setup/teardown code (though the context has the advantage that __exit__ is always called, even if exceptions are raised, which makes cleanup a lot easier to handle). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Build module script
On 21 April 2011 10:23, Vitja Makarov wrote: > 2011/4/21 mark florisson : >> On 21 April 2011 08:57, Vitja Makarov wrote: >>> Now we have cythonrun build script, may be it's time to create script >>> for easy module building? >>> >>> -- >>> vitja. >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >> >> We have cythonize(): >> http://wiki.cython.org/enhancements/distutils_preprocessing . Is >> something like that what you mean? Unfortunately I believe it doesn't >> parse sys.argv (yet). Perhaps a recursive option would also be useful. > > Not exactly cythonize is for distutils and friends. > > I mean simple script that run cython compiler and then build shared > python module. > It's much better then manually run cython then gcc or write makefile for this. Ah, I see. Yeah that might be convenient. Although there is always pyximport. > -- > vitja. > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 21 April 2011 10:37, Robert Bradshaw wrote:
> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
> wrote:
>> On 18 April 2011 16:41, Dag Sverre Seljebotn
>> wrote:
>>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>>> have a look yet but I will later)
>>>
>>> You're right about (the current) buffers and the gil. A testcase explicitly
>>> for them would be good.
>>>
>>> Firstprivate etc: i think it'd be nice myself, but it is probably better to
>>> take a break from it at this point so that we can think more about that and
>>> not do anything rash; perhaps open up a specific thread on them and ask for
>>> more general input. Perhaps you want to take a break or task-switch to
>>> something else (fused types?) until I can get around to review and merge
>>> what you have so far? You'll know best what works for you though. If you
>>> decide to implement explicit threadprivate variables because you've got the
>>> flow I certainly wom't object myself.
>>>
>> Ok, cool, I'll move on :) I already included a test with a prange and
>> a numpy buffer with indexing.
>
> Wow, you're just plowing away at this. Very cool.
>
> +1 to disallowing nested prange, that seems to get really messy with
> little benefit.
>
> In terms of the CEP, I'm still unconvinced that firstprivate is not
> safe to infer, but lets leave the initial values undefined rather than
> specifying them to be NaNs (we can do that as an implementation if you
> want), which will give us flexibility to change later once we've had a
> chance to play around with it.
Yes, they are currently undefined (and not initialized to NaN etc).
The thing is that without the control flow analysis (or perhaps not
until runtime) you won't know whether a variable is initialized at all
before the parallel section, so making it firstprivate might actually
copy an undefined value (perhaps with a trap representation!) into the
thread-private copy, which might invalidate valid code. e.g. consider
x_is_initialized = False
if condition:
x = 1
x_is_initialized = True
for i in prange(10, schedule='static'):
if x_is_initialized:
printf("%d\n", x)
x = i
> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
> We also probably want some way of explicitly marking a variable as
> shared and still be able to assign to/flush/sync it. Perhaps the
> parallel context could be used for these declarations, i.e.
>
> with parallel(threadlocal=a, shared=(b,c)):
> ...
>
> which would be considered an "expert" usecase.
Indeed, assigning to elements in an array instead doesn't seem very
convenient :)
> For all the discussion of threadsavailable/threadid, the most common
> usecase I see is for allocating a large shared buffer and partitioning
> it. This seems better handled by allocating separate thread-local
> buffers, no? I still like the context idea, but everything in a
> parallel block before and after the loop(s) also seems like a natural
> place to put any setup/teardown code (though the context has the
> advantage that __exit__ is always called, even if exceptions are
> raised, which makes cleanup a lot easier to handle).
Currently 'with gil' isn't merged into that branch, and if it will, it
will be disallowed, as I'm not yet sure how (if at all) it could be
handled with regard to exceptions. It seems a lot easier to disallow
it and have the user write a 'with gil' function, from which nothing
can propagate.
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 21 April 2011 10:59, mark florisson wrote:
> On 21 April 2011 10:37, Robert Bradshaw wrote:
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>> wrote:
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn
>>> wrote:
Excellent! Sounds great! (as I won't have my laptop for some days I can't
have a look yet but I will later)
You're right about (the current) buffers and the gil. A testcase explicitly
for them would be good.
Firstprivate etc: i think it'd be nice myself, but it is probably better to
take a break from it at this point so that we can think more about that and
not do anything rash; perhaps open up a specific thread on them and ask for
more general input. Perhaps you want to take a break or task-switch to
something else (fused types?) until I can get around to review and merge
what you have so far? You'll know best what works for you though. If you
decide to implement explicit threadprivate variables because you've got the
flow I certainly wom't object myself.
>>> Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> Yes, they are currently undefined (and not initialized to NaN etc).
> The thing is that without the control flow analysis (or perhaps not
> until runtime) you won't know whether a variable is initialized at all
> before the parallel section, so making it firstprivate might actually
> copy an undefined value (perhaps with a trap representation!) into the
> thread-private copy, which might invalidate valid code. e.g. consider
>
> x_is_initialized = False
> if condition:
> x = 1
> x_is_initialized = True
>
> for i in prange(10, schedule='static'):
> if x_is_initialized:
> printf("%d\n", x)
> x = i
Erm, that snippet I posted is invalid in any case, as x will be
private. So guess initializing things to NaN in such would have to
occur in the parallel section that should enclose the for. So e.g.
we'd have to do
#pragma omp parallel private(x)
{
x = INT_MAX;
#pragma omp for lastprivate(i)
for (...)
...
}
Which would then mean that 'x' cannot be lastprivate anymore :). So
it's either "uninitialized and undefined" or "firstprivate". I
personally prefer the former for the implicit route.
I do like the threadlocal=a stuff to parallel, it's basically what I
proposed a while back except that you don't make them strings, but
better because most of your variables can be inferred, so the
messiness is gone.
>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> with parallel(threadlocal=a, shared=(b,c)):
>> ...
>>
>> which would be considered an "expert" usecase.
>
> Indeed, assigning to elements in an array instead doesn't seem very
> convenient :)
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> Currently 'with gil' isn't merged into that branch, and if it will, it
> will be disallowed, as I'm not yet sure how (if at all) it could be
> handled with regard to exceptions. It seems a lot easier to disallow
> it and have the user write a 'with gil' function, from which nothing
> can propagate.
>
>> - Robert
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Thu, Apr 21, 2011 at 1:59 AM, mark florisson
wrote:
> On 21 April 2011 10:37, Robert Bradshaw wrote:
>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>> wrote:
>>> On 18 April 2011 16:41, Dag Sverre Seljebotn
>>> wrote:
Excellent! Sounds great! (as I won't have my laptop for some days I can't
have a look yet but I will later)
You're right about (the current) buffers and the gil. A testcase explicitly
for them would be good.
Firstprivate etc: i think it'd be nice myself, but it is probably better to
take a break from it at this point so that we can think more about that and
not do anything rash; perhaps open up a specific thread on them and ask for
more general input. Perhaps you want to take a break or task-switch to
something else (fused types?) until I can get around to review and merge
what you have so far? You'll know best what works for you though. If you
decide to implement explicit threadprivate variables because you've got the
flow I certainly wom't object myself.
>>> Ok, cool, I'll move on :) I already included a test with a prange and
>>> a numpy buffer with indexing.
>>
>> Wow, you're just plowing away at this. Very cool.
>>
>> +1 to disallowing nested prange, that seems to get really messy with
>> little benefit.
>>
>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>> safe to infer, but lets leave the initial values undefined rather than
>> specifying them to be NaNs (we can do that as an implementation if you
>> want), which will give us flexibility to change later once we've had a
>> chance to play around with it.
>
> Yes, they are currently undefined (and not initialized to NaN etc).
> The thing is that without the control flow analysis (or perhaps not
> until runtime) you won't know whether a variable is initialized at all
> before the parallel section, so making it firstprivate might actually
> copy an undefined value (perhaps with a trap representation!) into the
> thread-private copy, which might invalidate valid code. e.g. consider
>
> x_is_initialized = False
> if condition:
> x = 1
> x_is_initialized = True
>
> for i in prange(10, schedule='static'):
> if x_is_initialized:
> printf("%d\n", x)
> x = i
I'm still failing to see how this is a problem (or anything new, as
opposed to this same example with an ordinary range).
>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>> We also probably want some way of explicitly marking a variable as
>> shared and still be able to assign to/flush/sync it. Perhaps the
>> parallel context could be used for these declarations, i.e.
>>
>> with parallel(threadlocal=a, shared=(b,c)):
>> ...
>>
>> which would be considered an "expert" usecase.
>
> Indeed, assigning to elements in an array instead doesn't seem very
> convenient :)
>
>> For all the discussion of threadsavailable/threadid, the most common
>> usecase I see is for allocating a large shared buffer and partitioning
>> it. This seems better handled by allocating separate thread-local
>> buffers, no? I still like the context idea, but everything in a
>> parallel block before and after the loop(s) also seems like a natural
>> place to put any setup/teardown code (though the context has the
>> advantage that __exit__ is always called, even if exceptions are
>> raised, which makes cleanup a lot easier to handle).
>
> Currently 'with gil' isn't merged into that branch, and if it will, it
> will be disallowed, as I'm not yet sure how (if at all) it could be
> handled with regard to exceptions. It seems a lot easier to disallow
> it and have the user write a 'with gil' function, from which nothing
> can propagate.
Not being able to propagate exceptions is a pretty strong
constraint--even if the implementation doesn't yet support it, it'd be
nice to have an API that makes it possible as a future feature.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Thu, Apr 21, 2011 at 2:21 AM, mark florisson
wrote:
> On 21 April 2011 10:59, mark florisson wrote:
>> On 21 April 2011 10:37, Robert Bradshaw wrote:
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>> wrote:
On 18 April 2011 16:41, Dag Sverre Seljebotn
wrote:
> Excellent! Sounds great! (as I won't have my laptop for some days I can't
> have a look yet but I will later)
>
> You're right about (the current) buffers and the gil. A testcase
> explicitly
> for them would be good.
>
> Firstprivate etc: i think it'd be nice myself, but it is probably better
> to
> take a break from it at this point so that we can think more about that
> and
> not do anything rash; perhaps open up a specific thread on them and ask
> for
> more general input. Perhaps you want to take a break or task-switch to
> something else (fused types?) until I can get around to review and merge
> what you have so far? You'll know best what works for you though. If you
> decide to implement explicit threadprivate variables because you've got
> the
> flow I certainly wom't object myself.
>
Ok, cool, I'll move on :) I already included a test with a prange and
a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> Yes, they are currently undefined (and not initialized to NaN etc).
>> The thing is that without the control flow analysis (or perhaps not
>> until runtime) you won't know whether a variable is initialized at all
>> before the parallel section, so making it firstprivate might actually
>> copy an undefined value (perhaps with a trap representation!) into the
>> thread-private copy, which might invalidate valid code. e.g. consider
>>
>> x_is_initialized = False
>> if condition:
>> x = 1
>> x_is_initialized = True
>>
>> for i in prange(10, schedule='static'):
>> if x_is_initialized:
>> printf("%d\n", x)
>> x = i
>
> Erm, that snippet I posted is invalid in any case, as x will be
> private. So guess initializing things to NaN in such would have to
> occur in the parallel section that should enclose the for. So e.g.
> we'd have to do
>
> #pragma omp parallel private(x)
> {
> x = INT_MAX;
> #pragma omp for lastprivate(i)
> for (...)
> ...
> }
>
> Which would then mean that 'x' cannot be lastprivate anymore :). So
> it's either "uninitialized and undefined" or "firstprivate". I
> personally prefer the former for the implicit route.
A variable can't be both first and last private? In any case, as long
as we don't promise anything about them now, we can decide later.
> I do like the threadlocal=a stuff to parallel, it's basically what I
> proposed a while back except that you don't make them strings, but
> better because most of your variables can be inferred, so the
> messiness is gone.
Yep.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 21 April 2011 11:37, Robert Bradshaw wrote:
> On Thu, Apr 21, 2011 at 2:21 AM, mark florisson
> wrote:
>> On 21 April 2011 10:59, mark florisson wrote:
>>> On 21 April 2011 10:37, Robert Bradshaw
>>> wrote:
On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
wrote:
> On 18 April 2011 16:41, Dag Sverre Seljebotn
> wrote:
>> Excellent! Sounds great! (as I won't have my laptop for some days I can't
>> have a look yet but I will later)
>>
>> You're right about (the current) buffers and the gil. A testcase
>> explicitly
>> for them would be good.
>>
>> Firstprivate etc: i think it'd be nice myself, but it is probably better
>> to
>> take a break from it at this point so that we can think more about that
>> and
>> not do anything rash; perhaps open up a specific thread on them and ask
>> for
>> more general input. Perhaps you want to take a break or task-switch to
>> something else (fused types?) until I can get around to review and merge
>> what you have so far? You'll know best what works for you though. If you
>> decide to implement explicit threadprivate variables because you've got
>> the
>> flow I certainly wom't object myself.
>>
> Ok, cool, I'll move on :) I already included a test with a prange and
> a numpy buffer with indexing.
Wow, you're just plowing away at this. Very cool.
+1 to disallowing nested prange, that seems to get really messy with
little benefit.
In terms of the CEP, I'm still unconvinced that firstprivate is not
safe to infer, but lets leave the initial values undefined rather than
specifying them to be NaNs (we can do that as an implementation if you
want), which will give us flexibility to change later once we've had a
chance to play around with it.
>>>
>>> Yes, they are currently undefined (and not initialized to NaN etc).
>>> The thing is that without the control flow analysis (or perhaps not
>>> until runtime) you won't know whether a variable is initialized at all
>>> before the parallel section, so making it firstprivate might actually
>>> copy an undefined value (perhaps with a trap representation!) into the
>>> thread-private copy, which might invalidate valid code. e.g. consider
>>>
>>> x_is_initialized = False
>>> if condition:
>>> x = 1
>>> x_is_initialized = True
>>>
>>> for i in prange(10, schedule='static'):
>>> if x_is_initialized:
>>> printf("%d\n", x)
>>> x = i
>>
>> Erm, that snippet I posted is invalid in any case, as x will be
>> private. So guess initializing things to NaN in such would have to
>> occur in the parallel section that should enclose the for. So e.g.
>> we'd have to do
>>
>> #pragma omp parallel private(x)
>> {
>> x = INT_MAX;
>> #pragma omp for lastprivate(i)
>> for (...)
>> ...
>> }
>>
>> Which would then mean that 'x' cannot be lastprivate anymore :). So
>> it's either "uninitialized and undefined" or "firstprivate". I
>> personally prefer the former for the implicit route.
>
> A variable can't be both first and last private? In any case, as long
> as we don't promise anything about them now, we can decide later.
It can be, but not if the binding parallel region declares it private.
So we wouldn't actually need the snippet above, we could just do
x = INT_MAX;
#pragma omp parallel for firstprivate(x) lastprivate(i, x)
for (...)
...
Yeah, that would work.
>> I do like the threadlocal=a stuff to parallel, it's basically what I
>> proposed a while back except that you don't make them strings, but
>> better because most of your variables can be inferred, so the
>> messiness is gone.
>
> Yep.
>
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 21 April 2011 11:18, Robert Bradshaw wrote:
> On Thu, Apr 21, 2011 at 1:59 AM, mark florisson
> wrote:
>> On 21 April 2011 10:37, Robert Bradshaw wrote:
>>> On Mon, Apr 18, 2011 at 7:51 AM, mark florisson
>>> wrote:
On 18 April 2011 16:41, Dag Sverre Seljebotn
wrote:
> Excellent! Sounds great! (as I won't have my laptop for some days I can't
> have a look yet but I will later)
>
> You're right about (the current) buffers and the gil. A testcase
> explicitly
> for them would be good.
>
> Firstprivate etc: i think it'd be nice myself, but it is probably better
> to
> take a break from it at this point so that we can think more about that
> and
> not do anything rash; perhaps open up a specific thread on them and ask
> for
> more general input. Perhaps you want to take a break or task-switch to
> something else (fused types?) until I can get around to review and merge
> what you have so far? You'll know best what works for you though. If you
> decide to implement explicit threadprivate variables because you've got
> the
> flow I certainly wom't object myself.
>
Ok, cool, I'll move on :) I already included a test with a prange and
a numpy buffer with indexing.
>>>
>>> Wow, you're just plowing away at this. Very cool.
>>>
>>> +1 to disallowing nested prange, that seems to get really messy with
>>> little benefit.
>>>
>>> In terms of the CEP, I'm still unconvinced that firstprivate is not
>>> safe to infer, but lets leave the initial values undefined rather than
>>> specifying them to be NaNs (we can do that as an implementation if you
>>> want), which will give us flexibility to change later once we've had a
>>> chance to play around with it.
>>
>> Yes, they are currently undefined (and not initialized to NaN etc).
>> The thing is that without the control flow analysis (or perhaps not
>> until runtime) you won't know whether a variable is initialized at all
>> before the parallel section, so making it firstprivate might actually
>> copy an undefined value (perhaps with a trap representation!) into the
>> thread-private copy, which might invalidate valid code. e.g. consider
>>
>> x_is_initialized = False
>> if condition:
>> x = 1
>> x_is_initialized = True
>>
>> for i in prange(10, schedule='static'):
>> if x_is_initialized:
>> printf("%d\n", x)
>> x = i
>
> I'm still failing to see how this is a problem (or anything new, as
> opposed to this same example with an ordinary range).
>>> The "cdef threadlocal(int) foo" declaration syntax feels odd to me...
>>> We also probably want some way of explicitly marking a variable as
>>> shared and still be able to assign to/flush/sync it. Perhaps the
>>> parallel context could be used for these declarations, i.e.
>>>
>>> with parallel(threadlocal=a, shared=(b,c)):
>>> ...
>>>
>>> which would be considered an "expert" usecase.
>>
>> Indeed, assigning to elements in an array instead doesn't seem very
>> convenient :)
>>
>>> For all the discussion of threadsavailable/threadid, the most common
>>> usecase I see is for allocating a large shared buffer and partitioning
>>> it. This seems better handled by allocating separate thread-local
>>> buffers, no? I still like the context idea, but everything in a
>>> parallel block before and after the loop(s) also seems like a natural
>>> place to put any setup/teardown code (though the context has the
>>> advantage that __exit__ is always called, even if exceptions are
>>> raised, which makes cleanup a lot easier to handle).
>>
>> Currently 'with gil' isn't merged into that branch, and if it will, it
>> will be disallowed, as I'm not yet sure how (if at all) it could be
>> handled with regard to exceptions. It seems a lot easier to disallow
>> it and have the user write a 'with gil' function, from which nothing
>> can propagate.
>
> Not being able to propagate exceptions is a pretty strong
> constraint--even if the implementation doesn't yet support it, it'd be
> nice to have an API that makes it possible as a future feature.
It would be possible, with some modifications to try/finally. I think
it'd be best to stabilize and merge with gil first.
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Build module script
>>> On 21 April 2011 08:57, Vitja Makarov wrote: >> I mean simple script that run cython compiler and then build shared >> python module. >> It's much better then manually run cython then gcc or write makefile for >> this. > >Ah, I see. Yeah that might be convenient. Although there is always pyximport. I have a script I can contribute called pycc that might be a bit more than "simple". pycc [-c] [pyrexc|cython opts] [-- C-compiler opts] [Clink opts] as in pycc -c foo.pyx to produce foo.so in the simple case, or maybe, just as an example pycc -c -- -fmudflap foo.pyx -lmudflap if you had gcc as a compiler and wanted to turn on mudflap. The script itself is a Python script which uses an exec(file(os.environ["HOME"] + "/.pyccrc").read()) to configure things in the Python language itself for what C compiler and what default compile flags, C & Python include paths, libraries, etc. It only works on Unix right now. I'm not sure how hard it would be to adapt it to work on both Unix and Windows. It actually also works for "programs" rather than "modules" if you drop the -c as in pycc -- -I/usr/include/X11 whatever.pyx -lX11 to produce a --embed generated program called "whatever". In all, it makes for a pycc that behaves a bit more like traditional compiler front ends such as 'gcc' that take source all the way to a variety of outputs usable by the host OS. It works well in traditional Makefiles or just from the command line as one-off invocations as you're doing initial writing and testing or even "just seeing" if Cython works/helps in some isolated situation with some module or script. -- There's also some complexity that dates to pre-embed days, but which I personally find useful (or Pyrex users might find useful since I believe Pyrex does not have --embed). If you do not have the "-c" and you do have a ##MAIN## comment in the source file, it will indent all code after ##MAIN## into a "Main()" function called at startup. If there's no special comment it falls back to --embed. (we could spell this some other way if anyone wants more like the compiler option comment syntax "# cython: MAIN"). The upside of this feature is that with almost no mucking about with your Python script other than adding a comment, it makes variables local to Main() which allows optimizations in the main script logic that potentially make things much faster. I see 2X speed-ups in a few of my own speed critical programs. It probably would speed up CPython interpretation, too, but not nearly as much as the Cython operation in my various experiments. The downside of the feature is that does fiddle with global/local scope or mixed statements + defines + statments + defines. Those are all "very uncommon" patterns in my usage. Also, it does have a big ##MAIN## warning marker to let a reader know something special might be going on semantics-wise. And, hey, 2X speed for a 1-line diff. :) So, you can probably see why I keep that feature around, but I could also see if people didn't want an auto-functionizing-of-Main feature. Having it around also doesn't seem harmful if the spelling of the comment marker is nigh impossible to occur by accident. If you wanted me to delete that from the script before submission, I could, but it'd require a bit work and testing. Let me know if you want it. -- An entirely alternate proposal that might be better, but would also be more involved would be to make the "cython" program grow functionality to make it more like a traditional compiler front end. As things stand, "cython" (inheriting use-concept from Pyrex) works more like "cc -E" where output needs to fed to another stage to be compiled/linked. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 04/21/2011 10:37 AM, Robert Bradshaw wrote: On Mon, Apr 18, 2011 at 7:51 AM, mark florisson wrote: On 18 April 2011 16:41, Dag Sverre Seljebotn wrote: Excellent! Sounds great! (as I won't have my laptop for some days I can't have a look yet but I will later) You're right about (the current) buffers and the gil. A testcase explicitly for them would be good. Firstprivate etc: i think it'd be nice myself, but it is probably better to take a break from it at this point so that we can think more about that and not do anything rash; perhaps open up a specific thread on them and ask for more general input. Perhaps you want to take a break or task-switch to something else (fused types?) until I can get around to review and merge what you have so far? You'll know best what works for you though. If you decide to implement explicit threadprivate variables because you've got the flow I certainly wom't object myself. Ok, cool, I'll move on :) I already included a test with a prange and a numpy buffer with indexing. Wow, you're just plowing away at this. Very cool. +1 to disallowing nested prange, that seems to get really messy with little benefit. In terms of the CEP, I'm still unconvinced that firstprivate is not safe to infer, but lets leave the initial values undefined rather than specifying them to be NaNs (we can do that as an implementation if you want), which will give us flexibility to change later once we've had a chance to play around with it. I don't see any technical issues with inferring firstprivate, the question is whether we want to. I suggest not inferring it in order to make this safer: One should be able to just try to change a loop from "range" to "prange", and either a) have things fail very hard, or b) just work correctly and be able to trust the results. Note that when I suggest using NaN, it is as initial values for EACH ITERATION, not per-thread initialization. It is not about "firstprivate" or not, but about disabling thread-private variables entirely in favor of "per-iteration" variables. I believe that by talking about "readonly" and "per-iteration" variables, rather than "thread-shared" and "thread-private" variables, this can be used much more safely and with virtually no knowledge of the details of threading. Again, what's in my mind are scientific programmers with (too) little training. In the end it's a matter of taste and what is most convenient to more users. But I believe the case of needing real thread-private variables that preserves per-thread values across iterations (and thus also can possibly benefit from firstprivate) is seldomly enough used that an explicit declaration is OK, in particular when it buys us so much in safety in the common case. To be very precise, cdef double x, z for i in prange(n): x = f(x) z = f(i) ... goes to cdef double x, z for i in prange(n): x = z = nan x = f(x) z = f(i) ... and we leave it to the C compiler to (trivially) optimize away "z = nan". And, yes, it is a stopgap solution until we've got control flow analysis so that we can outright disallow such uses of x (without threadprivate declaration, which also gives firstprivate behaviour). The "cdef threadlocal(int) foo" declaration syntax feels odd to me... We also probably want some way of explicitly marking a variable as shared and still be able to assign to/flush/sync it. Perhaps the parallel context could be used for these declarations, i.e. with parallel(threadlocal=a, shared=(b,c)): ... which would be considered an "expert" usecase. I'm not set on the syntax for threadlocal variables; although your proposal feels funny/very unpythonic, almost like a C macro. For some inspiration, here's the Python solution (with no obvious place to put the type): import threading mydata = threading.local() mydata.myvar = ... # value is threadprivate For all the discussion of threadsavailable/threadid, the most common usecase I see is for allocating a large shared buffer and partitioning it. This seems better handled by allocating separate thread-local buffers, no? I still like the context idea, but everything in a parallel block before and after the loop(s) also seems like a natural place to put any setup/teardown code (though the context has the advantage that __exit__ is always called, even if exceptions are raised, which makes cleanup a lot easier to handle). I'd *really* like to have try/finally available in cython.parallel block for this, although I realize that may have to wait for a while. A big part of our discussions at the workshop were about how to handle exceptions; I guess there'll be a "phase 2" of this where break/continue/raise is dealt with. Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On Thu, Apr 21, 2011 at 11:13 AM, Dag Sverre Seljebotn wrote: > On 04/21/2011 10:37 AM, Robert Bradshaw wrote: >> >> In terms of the CEP, I'm still unconvinced that firstprivate is not >> safe to infer, but lets leave the initial values undefined rather than >> specifying them to be NaNs (we can do that as an implementation if you >> want), which will give us flexibility to change later once we've had a >> chance to play around with it. > > I don't see any technical issues with inferring firstprivate, the question > is whether we want to. I suggest not inferring it in order to make this > safer: One should be able to just try to change a loop from "range" to > "prange", and either a) have things fail very hard, or b) just work > correctly and be able to trust the results. > > Note that when I suggest using NaN, it is as initial values for EACH > ITERATION, not per-thread initialization. It is not about "firstprivate" or > not, but about disabling thread-private variables entirely in favor of > "per-iteration" variables. > > I believe that by talking about "readonly" and "per-iteration" variables, > rather than "thread-shared" and "thread-private" variables, this can be used > much more safely and with virtually no knowledge of the details of > threading. Again, what's in my mind are scientific programmers with (too) > little training. > > In the end it's a matter of taste and what is most convenient to more users. > But I believe the case of needing real thread-private variables that > preserves per-thread values across iterations (and thus also can possibly > benefit from firstprivate) is seldomly enough used that an explicit > declaration is OK, in particular when it buys us so much in safety in the > common case. > > To be very precise, > > cdef double x, z > for i in prange(n): > x = f(x) > z = f(i) > ... > > goes to > > cdef double x, z > for i in prange(n): > x = z = nan > x = f(x) > z = f(i) > ... > > and we leave it to the C compiler to (trivially) optimize away "z = nan". > And, yes, it is a stopgap solution until we've got control flow analysis so > that we can outright disallow such uses of x (without threadprivate > declaration, which also gives firstprivate behaviour). OK, I had totally missed that these are per-iteration. In that case, it makes more sense. >> The "cdef threadlocal(int) foo" declaration syntax feels odd to me... >> We also probably want some way of explicitly marking a variable as >> shared and still be able to assign to/flush/sync it. Perhaps the >> parallel context could be used for these declarations, i.e. >> >> with parallel(threadlocal=a, shared=(b,c)): >> ... >> >> which would be considered an "expert" usecase. > > I'm not set on the syntax for threadlocal variables; although your proposal > feels funny/very unpythonic, almost like a C macro. For some inspiration, > here's the Python solution (with no obvious place to put the type): > > import threading > mydata = threading.local() > mydata.myvar = ... # value is threadprivate That's nice and Pythonic, though I'm not sure how we would handle typing and the passing of "mydata" around if we wanted to go that route. We have cython.locals, we could introduce cython.parallel.threadlocals(a=int), though this is a bit magical as well. >> For all the discussion of threadsavailable/threadid, the most common >> usecase I see is for allocating a large shared buffer and partitioning >> it. This seems better handled by allocating separate thread-local >> buffers, no? I still like the context idea, but everything in a >> parallel block before and after the loop(s) also seems like a natural >> place to put any setup/teardown code (though the context has the >> advantage that __exit__ is always called, even if exceptions are >> raised, which makes cleanup a lot easier to handle). > > I'd *really* like to have try/finally available in cython.parallel block for > this, although I realize that may have to wait for a while. A big part of > our discussions at the workshop were about how to handle exceptions; I guess > there'll be a "phase 2" of this where break/continue/raise is dealt with. Yeah, this is definitely an (important) second or third phase. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
