Re: [Cython] speed.pypy.org
Vitja Makarov, 30.04.2011 08:08: Can I run this tests at home? Don't expect it to come shrink wrapped. It took me a while to get the setup running on Hudson/Jenkins, and the scripts that do it aren't completely trivial. You can take a look at the config of the job that runs them: https://sage.math.washington.edu:8091/hudson/job/cython-devel-benchmarks-py27/ I'm currently using the PyPy test suite at https://bitbucket.org/pypy/benchmarks/ instead of the future CPython one at http://hg.python.org/benchmarks/ simply because the PyPy suite has more benchmarks (that work for us ;). The major problem with the test suite is that the unladen swallow people who wrote the test runner somehow got the paranoid idea that they had to control which environment variables were set during the run, so they chose to drop the entire environment and use their own one. They clearly never tried to run distutils in a benchmark, which requires some intimate knowledge about the environment it runs in. So you basically need a wrapper around cythonrun that properly sets up the environment for you. Take a look at what I did in the build job, it generates a script that sets up the environment, and then passes that into the benchmark runner. You also need a "usercustomize.py" script that installs pyximport (but only for the Cython runs, not for plain CPython!), so that it's not only the benchmark script that gets compiled but also its dependencies. I'm currently blacklisting the stdlib modules ['threading', 'socket', 'gettext', 'locale'] in pyximport.PyxImporter.blocked_modules when I run it locally, simply because they don't work but have a substantial impact on the running code. The basic command I'm using to run the suite is: python runner.py -a ",-Xauto_cpdef=True" \ -p ./cythonrun.sh --baseline=/path/to/python \ --fast -o results.json The CPython test runner is a tad better here, because it allows you to provide at least a whitelist of environment variables that are being passed through. I'm currently trying to set up a second benchmark job to run the (smaller) CPython benchmark suite in Py3 mode. The command to run that suite is more like python perf.py -a ',-Xauto_cpdef=true' \ --inherit_env=PYTHONPATH,PYTHONHOME,PYTHON,CFLAGS,OPT,PATH \ /path/to/python ./cythonrun.sh \ --fast and it doesn't seem to have json output. (Seriously, I can't believe they even broke "PATH" in their runner script!). There is also a GSoC project that aims to fix up the benchmark suite for the different Python implementations in order to set up a "speed.python.org" site that compares them. We should stay involved in that. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On 04/30/2011 08:39 AM, Robert Bradshaw wrote: On Fri, Apr 29, 2011 at 3:53 AM, mark florisson wrote: On 29 April 2011 12:28, Pauli Virtanen wrote: No, just that real_t is specialized to float whenever struct_t is specialized to A and to double when B. Or a more realistic example, ctypedef cython.fused_type(float, double) real_t ctypedef cython.fused_type(float complex, double complex) complex_t cdef real_plus_one(complex_t a): real_t b = a.real return b + 1 which I suppose would not be a very unusual thing in numerical codes. This would also allow writing the case you had earlier as cdef cython.fused_type(string_t, int, paired=struct_t) attr_t cdef func(struct_t mystruct, int i): cdef attr_t var if typeof(mystruct) is typeof(int): var = mystruct.attrib + i ... else: var = mystruct.attrib + i ... Things would need to be done explicitly instead of implicitly, though, but it would remove the need for any special handling of the "complex" keyword. If we're going to introduce pairing, another option would be ctypedef fused_type((double complex, double), (float complex, float)) (complex_t, real_t) though I'm not sure I like that either. We're not trying to create the all-powerful templating system here, and anything that can be done with pairing can be done (though less elegantly) via branching on the types, or, as Pauli mentions, using a wider type is often (but not always) a viable option. Keeping the right balance is difficult. But, at least there's some cases of needing this in various codebases when interfacing with LAPACK. Most uses of templating with Cython code I've seen so far does a similar kind of "zip" as what you have above (as we discussed on the workshop). So at least the usage pattern you write above is very common. float32 is not about to disappear, it really is twice as fast when you're memory IO bound. Using a wider type is actually quite often not possible; any time the type is involved as the base type of an array it is not possible, and that's a pretty common case. (With LAPACK you take the address of the variable and pass it to Fortran, so using a wider type is not possible there either, although I'll agree that's a more remote case.) My proposal: Don't support either "real_t complex" or paired fused types for the time being. Then see. But my vote is for paired fused types instead of "real_t complex". Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
Robert Bradshaw, 29.04.2011 06:32: On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel wrote: mark florisson, 28.04.2011 23:29: On 28 April 2011 22:31, mark florisson wrote: On 28 April 2011 22:12, Robert Bradshaw wrote: On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote: So I fixed all that, but I'm currently wondering about the proposed cython.typeof(). I believe it currently returns a string with the type name, and not the type itself. Yes. This is just because there's not really anything better to return at this point. We should "fix" this at some point in the future. So I think it would be inconsistent to suddenly start allowing comparison with 'is' and 'isinstance' and such. I'm open to other suggestions, but would like an expression that resolves at compile time to true/false (and we need to do branch pruning on it). Note that type() is not good enough, because it has different semantics, i.e. cdef object o = [] typeof(o), type(o) so lets not touch that one. Right, so for fused types I don't mind string comparison with cython.typeof(), but retrieval of the actual type for casts and declaration remains open. I'd be fine with something like cython.gettype(). It seems that this isn't optimized yet, but it looks to me like it wouldn't be very hard to do so. At least == and != could be resolved at compile time if the other operand is a string literal. Well, the obvious place where this would happen for free would be constant folding. But that runs *way* before type analysis, so all it sees is the typeof() call, not the string. Actually, to support code like ctypedef cython.fused_type(object, double) my_fused_type cdef foo(my_fused_type x): if my_fused_type is object: print x.attr else: cdef my_fused_type *ptr =&x we need to resolve this branch and prune "dead code" before type analysis, so I'm thinking it may need to be a special phase, perhaps part of the fused-type-specialization phase. Here's another idea: cdef foo(numeric x): if numeric in floating: ... elif numeric is long: ... else: ... print numeric is double # after the specialization pass, this would be a literal True/False node. Again, this would all be resolved before type analysis. Hmm, I wonder, if we ever want to start with whole program analysis, and we find code like this: def func(arg): ... a = [1,2,3] func(a) we could infer that "arg" is actually a "fused_type(object, list)". So, I think, what we eventually want is an intermingling of type analysis and specialisation. Meaning, we'd find a new type during type analysis or inference, split off a new version of the function and then analyse it. But maybe that's just a pie in the sky for now. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Sat, Apr 30, 2011 at 12:51 AM, Dag Sverre Seljebotn wrote: > On 04/30/2011 08:39 AM, Robert Bradshaw wrote: >> >> On Fri, Apr 29, 2011 at 3:53 AM, mark florisson >> wrote: >>> >>> On 29 April 2011 12:28, Pauli Virtanen wrote: No, just that real_t is specialized to float whenever struct_t is specialized to A and to double when B. Or a more realistic example, ctypedef cython.fused_type(float, double) real_t ctypedef cython.fused_type(float complex, double complex) complex_t cdef real_plus_one(complex_t a): real_t b = a.real return b + 1 which I suppose would not be a very unusual thing in numerical codes. This would also allow writing the case you had earlier as cdef cython.fused_type(string_t, int, paired=struct_t) attr_t cdef func(struct_t mystruct, int i): cdef attr_t var if typeof(mystruct) is typeof(int): var = mystruct.attrib + i ... else: var = mystruct.attrib + i ... Things would need to be done explicitly instead of implicitly, though, but it would remove the need for any special handling of the "complex" keyword. >> >> If we're going to introduce pairing, another option would be >> >> ctypedef fused_type((double complex, double), (float complex, >> float)) (complex_t, real_t) >> >> though I'm not sure I like that either. We're not trying to create the >> all-powerful templating system here, and anything that can be done >> with pairing can be done (though less elegantly) via branching on the >> types, or, as Pauli mentions, using a wider type is often (but not >> always) a viable option. > > Keeping the right balance is difficult. But, at least there's some cases of > needing this in various codebases when interfacing with LAPACK. > > Most uses of templating with Cython code I've seen so far does a similar > kind of "zip" as what you have above (as we discussed on the workshop). So > at least the usage pattern you write above is very common. > > float32 is not about to disappear, it really is twice as fast when you're > memory IO bound. > > Using a wider type is actually quite often not possible; any time the type > is involved as the base type of an array it is not possible, and that's a > pretty common case. (With LAPACK you take the address of the variable and > pass it to Fortran, so using a wider type is not possible there either, > although I'll agree that's a more remote case.) > > My proposal: Don't support either "real_t complex" or paired fused types for > the time being. Then see. +1, then we can address real need. > But my vote is for paired fused types instead of "real_t complex". > > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
On Sat, Apr 30, 2011 at 7:24 AM, Stefan Behnel wrote: > Robert Bradshaw, 29.04.2011 06:32: >> >> On Thu, Apr 28, 2011 at 7:09 PM, Stefan Behnel >> wrote: >>> >>> mark florisson, 28.04.2011 23:29: On 28 April 2011 22:31, mark florisson wrote: > > On 28 April 2011 22:12, Robert Bradshaw wrote: >> >> On Thu, Apr 28, 2011 at 12:48 PM, mark florisson wrote: >> >>> So I fixed all that, but I'm currently wondering about the proposed >>> cython.typeof(). I believe it currently returns a string with the >>> type >>> name, and not the type itself. >> >> Yes. This is just because there's not really anything better to return >> at this point. We should "fix" this at some point in the future. >> >>> So I think it would be inconsistent to >>> suddenly start allowing comparison with 'is' and 'isinstance' and >>> such. >> >> I'm open to other suggestions, but would like an expression that >> resolves at compile time to true/false (and we need to do branch >> pruning on it). Note that type() is not good enough, because it has >> different semantics, i.e. >> >> cdef object o = [] >> typeof(o), type(o) >> >> so lets not touch that one. > > Right, so for fused types I don't mind string comparison with > cython.typeof(), but retrieval of the actual type for casts and > declaration remains open. I'd be fine with something like > cython.gettype(). It seems that this isn't optimized yet, but it looks to me like it wouldn't be very hard to do so. At least == and != could be resolved at compile time if the other operand is a string literal. >>> >>> Well, the obvious place where this would happen for free would be >>> constant >>> folding. But that runs *way* before type analysis, so all it sees is the >>> typeof() call, not the string. >> >> Actually, to support code like >> >> ctypedef cython.fused_type(object, double) my_fused_type >> >> cdef foo(my_fused_type x): >> if my_fused_type is object: >> print x.attr >> else: >> cdef my_fused_type *ptr =&x >> >> we need to resolve this branch and prune "dead code" before type >> analysis, so I'm thinking it may need to be a special phase, perhaps >> part of the fused-type-specialization phase. Here's another idea: >> >> cdef foo(numeric x): >> if numeric in floating: >> ... >> elif numeric is long: >> ... >> else: >> ... >> print numeric is double # after the specialization pass, this >> would be a literal True/False node. >> >> Again, this would all be resolved before type analysis. > > Hmm, I wonder, if we ever want to start with whole program analysis, and we > find code like this: > > def func(arg): > ... > > a = [1,2,3] > func(a) > > we could infer that "arg" is actually a "fused_type(object, list)". So, I > think, what we eventually want is an intermingling of type analysis and > specialisation. Meaning, we'd find a new type during type analysis or > inference, split off a new version of the function and then analyse it. But > maybe that's just a pie in the sky for now. This fits into the idea of creating two (or possibly more, but avoiding combinatorial explosion) branches, a fast one and a safe one, and bailing from one to the other if guards fail (all within the same function). ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Pull request emails
Finally think I figured out how to get pull request emails (thanks to Gael V). From https://github.com/organizations/cython/teams/24445: """ Owners do not receive notifications for the organization's repos by default. To receive notifications, create a team and add the owners and repos for which notifications are desired. """ I created Reviewers and added me, Stefan & Robert for now. https://github.com/organizations/cython/teams/54516 Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Pull request emails
Excellent, thanks. On Sat, Apr 30, 2011 at 2:35 PM, Dag Sverre Seljebotn wrote: > Finally think I figured out how to get pull request emails (thanks to Gael > V). From https://github.com/organizations/cython/teams/24445: > > """ > Owners do not receive notifications for the organization's repos by default. > To receive notifications, create a team and add the owners and repos for > which notifications are desired. > """ > > I created Reviewers and added me, Stefan & Robert for now. > > https://github.com/organizations/cython/teams/54516 > > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fused Types
Stefan Behnel wrote: Meaning, we'd find a new type during type analysis or inference, split off a new version of the function and then analyse it. I'm not sure that this degree of smartness would really be a good idea. I'd worry that if I made a type error somewhere, the compiler would go off and spend a few hours generating umpteen zillion versions of the code instead of stopping and telling me something was wrong. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
