Re: [Cython] Control flow graph
Can I use cython-generators project on hudson for control-flow tests? So I'll move cf branch development to my master branch. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Control flow graph
Vitja Makarov, 14.04.2011 09:27: Can I use cython-generators project on hudson for control-flow tests? So I'll move cf branch development to my master branch. Sure. I renamed the tab to "cython-vitek". It's your repo, use it as you see fit. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 04/13/2011 11:13 PM, mark florisson wrote: Although there is omp_get_max_threads(): "The omp_get_max_threads routine returns an upper bound on the number of threads that could be used to form a new team if a parallel region without a num_threads clause were encountered after execution returns from this routine." So we could have threadsvailable() evaluate to that if encountered outside a parallel region. Inside, it would evaluate to omp_get_num_threads(). At worst, people would over-allocate a bit. Well, over-allocating could well mean 1 GB, which could well mean getting an unecesarry MemoryError (or, like in my case, if I'm not careful to set ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the cluster patrol process...) But even ignoring this, we also have to plan for people misusing the feature. If we put it in there, somebody somewhere *will* write code like this: nthreads = threadsavailable() with parallel: for i in prange(nthreads): for j in range(100*i, 100*(i+1)): [...] (Yes, they shouldn't. Yes, they will.) Combined with a race condition that will only very seldomly trigger, this starts to sound like a very bad idea indeed. So I agree with you that we should just leave it for now, and do single/barrier later. DS ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 14 April 2011 20:29, Dag Sverre Seljebotn wrote: > On 04/13/2011 11:13 PM, mark florisson wrote: >> >> Although there is omp_get_max_threads(): >> >> "The omp_get_max_threads routine returns an upper bound on the number >> of threads that could be used to form a new team if a parallel region >> without a num_threads clause were encountered after execution returns >> from this routine." >> >> So we could have threadsvailable() evaluate to that if encountered >> outside a parallel region. Inside, it would evaluate to >> omp_get_num_threads(). At worst, people would over-allocate a bit. > > Well, over-allocating could well mean 1 GB, which could well mean getting an > unecesarry MemoryError (or, like in my case, if I'm not careful to set > ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the > cluster patrol process...) The upper bound is not "however many threads you think you can start", but rather "how many threads are considered useful for your machine". So if you use omp_set_num_threads(), it will return the value you set there. Otherwise, if you have e.g. a quadcore, it will return 4. The spec says: "Note – The return value of the omp_get_max_threads routine can be used to dynamically allocate sufficient storage for all threads in the team formed at the subsequent active parallel region." So this sounds like a viable option. > But even ignoring this, we also have to plan for people misusing the > feature. If we put it in there, somebody somewhere *will* write code like > this: > > nthreads = threadsavailable() > with parallel: > for i in prange(nthreads): > for j in range(100*i, 100*(i+1)): [...] > > (Yes, they shouldn't. Yes, they will.) > > Combined with a race condition that will only very seldomly trigger, this > starts to sound like a very bad idea indeed. > > So I agree with you that we should just leave it for now, and do > single/barrier later. > > DS > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 14 April 2011 20:29, Dag Sverre Seljebotn wrote: > On 04/13/2011 11:13 PM, mark florisson wrote: >> >> Although there is omp_get_max_threads(): >> >> "The omp_get_max_threads routine returns an upper bound on the number >> of threads that could be used to form a new team if a parallel region >> without a num_threads clause were encountered after execution returns >> from this routine." >> >> So we could have threadsvailable() evaluate to that if encountered >> outside a parallel region. Inside, it would evaluate to >> omp_get_num_threads(). At worst, people would over-allocate a bit. > > Well, over-allocating could well mean 1 GB, which could well mean getting an > unecesarry MemoryError (or, like in my case, if I'm not careful to set > ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the > cluster patrol process...) > > But even ignoring this, we also have to plan for people misusing the > feature. If we put it in there, somebody somewhere *will* write code like > this: > > nthreads = threadsavailable() > with parallel: > for i in prange(nthreads): > for j in range(100*i, 100*(i+1)): [...] > > (Yes, they shouldn't. Yes, they will.) > > Combined with a race condition that will only very seldomly trigger, this > starts to sound like a very bad idea indeed. > > So I agree with you that we should just leave it for now, and do > single/barrier later. omp_get_max_threads() doesn't have a race, as it returns the upper bound. So e.g. if between your call and your parallel section less OpenMP threads become available, then you might get less threads, but never more. > DS > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 04/14/2011 08:39 PM, mark florisson wrote: On 14 April 2011 20:29, Dag Sverre Seljebotn wrote: On 04/13/2011 11:13 PM, mark florisson wrote: Although there is omp_get_max_threads(): "The omp_get_max_threads routine returns an upper bound on the number of threads that could be used to form a new team if a parallel region without a num_threads clause were encountered after execution returns from this routine." So we could have threadsvailable() evaluate to that if encountered outside a parallel region. Inside, it would evaluate to omp_get_num_threads(). At worst, people would over-allocate a bit. Well, over-allocating could well mean 1 GB, which could well mean getting an unecesarry MemoryError (or, like in my case, if I'm not careful to set ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the cluster patrol process...) The upper bound is not "however many threads you think you can start", but rather "how many threads are considered useful for your machine". So if you use omp_set_num_threads(), it will return the value you set there. Otherwise, if you have e.g. a quadcore, it will return 4. The spec says: "Note – The return value of the omp_get_max_threads routine can be used to dynamically allocate sufficient storage for all threads in the team formed at the subsequent active parallel region." So this sounds like a viable option. What would happen here: We have 8 cores. Some code has an OpenMP parallel section with maxthreads=2, and inside the section another function is called. That called function uses threadsavailable(), and has a parallel block that wants as many threads as it can get. I don't know the details as well as you do, but my uninformed guess is that in this case it'd be quite possible with a race where omp_get_max_threads would return 7 in each case, then the first one to the parallel would get the 7 threads. The remaining thread then has allocated storage for 7 threads but only has 1 thread running. BTW, I'm not sure what the difference is between the original idea and omp_get_max_threads -- in the absence of such races as above, my original idea with entering a parallel section (with the same scheduling parameters) just to see how many threads we got, would work as well? DS ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 04/14/2011 08:42 PM, mark florisson wrote: On 14 April 2011 20:29, Dag Sverre Seljebotn wrote: On 04/13/2011 11:13 PM, mark florisson wrote: Although there is omp_get_max_threads(): "The omp_get_max_threads routine returns an upper bound on the number of threads that could be used to form a new team if a parallel region without a num_threads clause were encountered after execution returns from this routine." So we could have threadsvailable() evaluate to that if encountered outside a parallel region. Inside, it would evaluate to omp_get_num_threads(). At worst, people would over-allocate a bit. Well, over-allocating could well mean 1 GB, which could well mean getting an unecesarry MemoryError (or, like in my case, if I'm not careful to set ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the cluster patrol process...) But even ignoring this, we also have to plan for people misusing the feature. If we put it in there, somebody somewhere *will* write code like this: nthreads = threadsavailable() with parallel: for i in prange(nthreads): for j in range(100*i, 100*(i+1)): [...] (Yes, they shouldn't. Yes, they will.) Combined with a race condition that will only very seldomly trigger, this starts to sound like a very bad idea indeed. So I agree with you that we should just leave it for now, and do single/barrier later. omp_get_max_threads() doesn't have a race, as it returns the upper bound. So e.g. if between your call and your parallel section less OpenMP threads become available, then you might get less threads, but never more. Oh, now I'm following you. Well, my argument was that I think erroring in that direction is pretty bad as well. Also, even if we're not making it available in cython.parallel, we're not stopping people from calling omp_get_max_threads directly themselves, which should be OK for the people who know enough to do this safely... Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Code examples missing in Cython User's Guide
Thanks Arthur. I actually found the code examples in a grandparent directory of the User's Guide documentation source directory. I have submitted a pull request on GitHub which corrects the User's Guide Tutorial documentation so that it now includes the code. https://github.com/cython/cython/pull/24 Could a Cython dev please review and approve this change and then build and update the docs on the Cython website for the sake of other Cython newbies? Thanks, Chris L. On Tue, Apr 12, 2011 at 1:39 PM, Arthur de Souza Ribeiro < arthurdesribe...@gmail.com> wrote: > Hey Chris, the code for primes and fib examples, are in the directory > 'Demos' of the repository... > > Best Regards. > > []s > > Arthur > > 2011/4/12 Chris Lasher > >> My apologies for cross-posting this from cython-users, but I realized I >> should have sent this bug report to cython-devel. >> >> Several code examples are missing from the User's Guide due to the source >> code files being moved or deleted. See for example the Tutorial page on the >> User's Guide http://docs.cython.org/src/userguide/tutorial.html The code >> for both fib.pyx and primes.pyx (and their setup.py files) is absent from >> the document. >> >> I looked at the ReST files and they are trying to source the files from an >> "examples" directory. Looking through the git repository, I wasn't able to >> locate this directory. Has it been moved or deleted? >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 14 April 2011 20:58, Dag Sverre Seljebotn wrote: > On 04/14/2011 08:42 PM, mark florisson wrote: >> >> On 14 April 2011 20:29, Dag Sverre Seljebotn >> wrote: >>> >>> On 04/13/2011 11:13 PM, mark florisson wrote: Although there is omp_get_max_threads(): "The omp_get_max_threads routine returns an upper bound on the number of threads that could be used to form a new team if a parallel region without a num_threads clause were encountered after execution returns from this routine." So we could have threadsvailable() evaluate to that if encountered outside a parallel region. Inside, it would evaluate to omp_get_num_threads(). At worst, people would over-allocate a bit. >>> >>> Well, over-allocating could well mean 1 GB, which could well mean getting >>> an >>> unecesarry MemoryError (or, like in my case, if I'm not careful to set >>> ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the >>> cluster patrol process...) >>> >>> But even ignoring this, we also have to plan for people misusing the >>> feature. If we put it in there, somebody somewhere *will* write code like >>> this: >>> >>> nthreads = threadsavailable() >>> with parallel: >>> for i in prange(nthreads): >>> for j in range(100*i, 100*(i+1)): [...] >>> >>> (Yes, they shouldn't. Yes, they will.) >>> >>> Combined with a race condition that will only very seldomly trigger, this >>> starts to sound like a very bad idea indeed. >>> >>> So I agree with you that we should just leave it for now, and do >>> single/barrier later. >> >> omp_get_max_threads() doesn't have a race, as it returns the upper >> bound. So e.g. if between your call and your parallel section less >> OpenMP threads become available, then you might get less threads, but >> never more. > > Oh, now I'm following you. > > Well, my argument was that I think erroring in that direction is pretty bad > as well. > > Also, even if we're not making it available in cython.parallel, we're not > stopping people from calling omp_get_max_threads directly themselves, which > should be OK for the people who know enough to do this safely... True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any event, we could just put a warning in the docs stating that using threadsavailable outside parallel sections returns an upper bound on the actual number of threads in a subsequent parallel section. > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 04/14/2011 09:08 PM, mark florisson wrote: On 14 April 2011 20:58, Dag Sverre Seljebotn wrote: On 04/14/2011 08:42 PM, mark florisson wrote: On 14 April 2011 20:29, Dag Sverre Seljebotn wrote: On 04/13/2011 11:13 PM, mark florisson wrote: Although there is omp_get_max_threads(): "The omp_get_max_threads routine returns an upper bound on the number of threads that could be used to form a new team if a parallel region without a num_threads clause were encountered after execution returns from this routine." So we could have threadsvailable() evaluate to that if encountered outside a parallel region. Inside, it would evaluate to omp_get_num_threads(). At worst, people would over-allocate a bit. Well, over-allocating could well mean 1 GB, which could well mean getting an unecesarry MemoryError (or, like in my case, if I'm not careful to set ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the cluster patrol process...) But even ignoring this, we also have to plan for people misusing the feature. If we put it in there, somebody somewhere *will* write code like this: nthreads = threadsavailable() with parallel: for i in prange(nthreads): for j in range(100*i, 100*(i+1)): [...] (Yes, they shouldn't. Yes, they will.) Combined with a race condition that will only very seldomly trigger, this starts to sound like a very bad idea indeed. So I agree with you that we should just leave it for now, and do single/barrier later. omp_get_max_threads() doesn't have a race, as it returns the upper bound. So e.g. if between your call and your parallel section less OpenMP threads become available, then you might get less threads, but never more. Oh, now I'm following you. Well, my argument was that I think erroring in that direction is pretty bad as well. Also, even if we're not making it available in cython.parallel, we're not stopping people from calling omp_get_max_threads directly themselves, which should be OK for the people who know enough to do this safely... True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any event, we could just put a warning in the docs stating that using threadsavailable outside parallel sections returns an upper bound on the actual number of threads in a subsequent parallel section. I don't think outside or within makes a difference -- what about nested parallel sections? At least my intention in the CEP was that threadsavailable was always for the next section (so often it would be 1 after entering the section). Perhaps just calling it "maxthreads" instead solves the issue. (Still, I favour just dropping threadsavailable/maxthreads for the time being. It is much simpler to add something later, when we've had some time to use it and reflect about it, than to remove something that shouldn't have been added.) Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 14 April 2011 21:37, Dag Sverre Seljebotn wrote: > On 04/14/2011 09:08 PM, mark florisson wrote: >> >> On 14 April 2011 20:58, Dag Sverre Seljebotn >> wrote: >>> >>> On 04/14/2011 08:42 PM, mark florisson wrote: On 14 April 2011 20:29, Dag Sverre Seljebotn wrote: > > On 04/13/2011 11:13 PM, mark florisson wrote: >> >> Although there is omp_get_max_threads(): >> >> "The omp_get_max_threads routine returns an upper bound on the number >> of threads that could be used to form a new team if a parallel region >> without a num_threads clause were encountered after execution returns >> from this routine." >> >> So we could have threadsvailable() evaluate to that if encountered >> outside a parallel region. Inside, it would evaluate to >> omp_get_num_threads(). At worst, people would over-allocate a bit. > > Well, over-allocating could well mean 1 GB, which could well mean > getting > an > unecesarry MemoryError (or, like in my case, if I'm not careful to set > ulimit, getting a SIGKILL sent to you 2 minutes after the fact by the > cluster patrol process...) > > But even ignoring this, we also have to plan for people misusing the > feature. If we put it in there, somebody somewhere *will* write code > like > this: > > nthreads = threadsavailable() > with parallel: > for i in prange(nthreads): > for j in range(100*i, 100*(i+1)): [...] > > (Yes, they shouldn't. Yes, they will.) > > Combined with a race condition that will only very seldomly trigger, > this > starts to sound like a very bad idea indeed. > > So I agree with you that we should just leave it for now, and do > single/barrier later. omp_get_max_threads() doesn't have a race, as it returns the upper bound. So e.g. if between your call and your parallel section less OpenMP threads become available, then you might get less threads, but never more. >>> >>> Oh, now I'm following you. >>> >>> Well, my argument was that I think erroring in that direction is pretty >>> bad >>> as well. >>> >>> Also, even if we're not making it available in cython.parallel, we're not >>> stopping people from calling omp_get_max_threads directly themselves, >>> which >>> should be OK for the people who know enough to do this safely... >> >> True, but it wouldn't be as easy to wrap in a #ifdef _OPENMP. In any >> event, we could just put a warning in the docs stating that using >> threadsavailable outside parallel sections returns an upper bound on >> the actual number of threads in a subsequent parallel section. > > I don't think outside or within makes a difference -- what about nested > parallel sections? At least my intention in the CEP was that > threadsavailable was always for the next section (so often it would be 1 > after entering the section). > > Perhaps just calling it "maxthreads" instead solves the issue. > > (Still, I favour just dropping threadsavailable/maxthreads for the time > being. It is much simpler to add something later, when we've had some time > to use it and reflect about it, than to remove something that shouldn't have > been added.) > > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > Definitely true, I'll disable it for now. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
I've created the .pyx files and it passed in all python tests. To test them, as I said, I copied the .py test files to my project directory, generated the .so files, import them instead of python modules and run. I run every test file and it passed in all of them. To run the tests, run the file 'run-tests.sh' I used just .pyx in this module, should I reimplement it using pxd with the normal .py? I'll still see the performance, and tell here... The code is updated in repository... Best Regards []s Arthur 2011/4/13 Stefan Behnel > Robert Bradshaw, 12.04.2011 22:42: > >> On Tue, Apr 12, 2011 at 11:22 AM, Stefan Behnel wrote: >> >>> Arthur de Souza Ribeiro, 12.04.2011 14:59: >>> The code is in this repository: https://github.com/arthursribeiro/JSON-module your feedback would be very important, so that I could improve my skills to get more and more able to work sooner in the project. >>> >>> I'd strongly suggest implementing this in pure Python (.py files instead >>> of >>> .pyx files), with externally provided static types for performance. A >>> single >>> code base is very advantageous for a large project like CPython, much >>> more >>> than the ultimate 5% better performance. >>> >> >> While this is advantageous for the final product, it may not be the >> easiest to get up and running with. >> > > Agreed. Arthur, it's fine if you write Cython code in a .pyx file to get > started. You can just extract the declarations later. > > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.
[please avoid top-posting] Arthur de Souza Ribeiro, 15.04.2011 04:31: I've created the .pyx files and it passed in all python tests. Fine. As far as I can see, you only added static types in some places. Did you test if they are actually required (maybe using "cython -a")? Some of them look rather counterproductive and should lead to a major slow-down. I added comments to your initial commit. Note that it's not obvious from your initial commit what you actually changed. It would have been better to import the original file first, rename it to .pyx, and then commit your changes. It appears that you accidentally added your .c and .so files to your repo. https://github.com/arthursribeiro/JSON-module To test them, as I said, I copied the .py test files to my project directory, generated the .so files, import them instead of python modules and run. I run every test file and it passed in all of them. To run the tests, run the file 'run-tests.sh' I used just .pyx in this module, should I reimplement it using pxd with the normal .py? Not at this point. I think it's more important to get some performance numbers to see how your module behaves compared to the C accelerator module (_json.c). I think the best approach to this project would actually be to start with profiling the Python implementation to see where performance problems occur (or to look through _json.c to see what the CPython developers considered performance critical), and then put the focus on trying to speed up only those parts of the Python implementation, by adding static types and potentially even rewriting them in a way that Cython can optimise them better. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
