Re: [Cython] Wacky idea: proper macros
Stefan Behnel wrote:
>Wes McKinney, 29.04.2012 03:14:
>> On Sat, Apr 28, 2012 at 5:25 PM, mark florisson wrote:
>>> On 28 April 2012 22:04, Nathaniel Smith wrote:
Was chatting with Wes today about the usual problem many of us have
encountered with needing to use some sort of templating system to
generate code handling multiple types, operations, etc., and a
>wacky
idea occurred to me. So I thought I'd through it out here.
What if we added a simple macro facility to Cython, that worked at
>the
AST level? (I.e. I'm talking lisp-style macros, *not* C-style
>macros.)
Basically some way to write arbitrary Python code into a .pyx file
that gets executed at compile time and can transform the AST, plus
some nice convenience APIs for simple transformations.
E.g., if we steal the illegal token sequence @@ as our marker, we
could have something like:
@@ # alone on a line, starts a block of Python code
from Cython.MacroUtil import replace_ctype
def expand_types(placeholder, typelist):
def my_decorator(function_name, ast):
functions = {}
for typename in typelist:
new_name = "%s_%s" % (function_name, typename)
functions[name] = replace_ctype(ast, placeholder, typename)
return functions
return function_decorator
@@ # this token sequence cannot occur in Python, so it's a safe
>end-marker
# Compile-time function decorator
# Results in two cdef functions named sum_double and sum_int
@@expand_types("T", ["double", "int"])
cdef T sum(np.ndarray[T] arr):
cdef T start = 0;
for i in range(arr.size):
start += arr[i]
return start
I don't know if this is a good idea, but it seems like it'd be very
easy to do on the Cython side, fairly clean, and be dramatically
>less
horrible than all the ad-hoc templating stuff people do now.
Presumably there'd be strict limits on how much backwards
compatibility we'd be willing to guarantee for code that went
>poking
around in the AST by hand, but a small handful of functions like my
notional "replace_ctype" would go a long way, and wouldn't impose
>much
of a compatibility burden.
>>>
>>> Have you looked at
>http://wiki.cython.org/enhancements/metaprogramming ?
>>>
>>> In general I would like better meta-programming support, maybe even
>>> allow defining new operators (although I'm not sure any of it is
>very
>>> pythonic), but for templates I think fused types should be used, or
>>> improved when they fall short. Maybe a plugin system could also help
>>> people.
>>
>> I referenced this problem recently in a blog post
>> (http://wesmckinney.com/blog/?p=467). My main interest these days is
>> in expressing data algorithms. I've unfortunately found myself
>working
>> around performance problems with fundamental array operations in
>NumPy
>> so a lot of the Cython work I've done has been in and around this. In
>> lieu of some kind of macro system it seems inevitable that I'm going
>> to need to create some kind of mini array language or otherwise code
>> generation framework (targeting C, Cython, or Fortran). I worry that
>> this is going to end with me creating "yet another APL [or Haskell]
>> implementation" but I really need something that runs inside
>CPython.
>
>Generally speaking, it's always better to collect and describe use
>cases
>first before adding a language feature, especially one that is as
>complex
>and far reaching as this.
>
>It might well be that fused types can (be made to) work for them, and
>it
>might be that a (non AST based) preprocessor step would work. Keeping
>metaprogramming facilities out of the compiler makes it both more
>generally
>versatile (and easier to replace or disable) and keeps both sides
>simpler.
>
>
>> And why not? Most of these algorithms could be expressed at a very
>> high level and lead to pretty clean generated C with many of the
>> special cases (contiguous memory, or low dimensions in the case of
>> n-dimensional algorithms) checked and handled in simplified loops.
>
>That sounds like what you want is a preprocessor that spells out NumPy
>array configuration options into separate code paths. However, I'm not
>sure
>a generic approach would work well (enough?) here. And it also doesn't
>sound like this needs to be done inside of the compiler. It should be
>possible to build that on top of fused types as a separate
>preprocessor.
Well, *of* *course* it is possible to do it using a preprocessor.
The question is whether it would be a nicer experience if Cython had macros,
and whether the feature would fit well with Cython. I know I would use Cython
more if it had metaprogramming, but you can't cater to everyone.
Please, everybody, look at the metaprogramming in Julia for a well done example
before we discuss this further (julialang.org)... When done at that level, you
don't need to worry about AST APIs e
Re: [Cython] Wacky idea: proper macros
On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
wrote:
> On 28 April 2012 22:04, Nathaniel Smith wrote:
>> Was chatting with Wes today about the usual problem many of us have
>> encountered with needing to use some sort of templating system to
>> generate code handling multiple types, operations, etc., and a wacky
>> idea occurred to me. So I thought I'd through it out here.
>>
>> What if we added a simple macro facility to Cython, that worked at the
>> AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
>> Basically some way to write arbitrary Python code into a .pyx file
>> that gets executed at compile time and can transform the AST, plus
>> some nice convenience APIs for simple transformations.
>>
>> E.g., if we steal the illegal token sequence @@ as our marker, we
>> could have something like:
>>
>> @@ # alone on a line, starts a block of Python code
>> from Cython.MacroUtil import replace_ctype
>> def expand_types(placeholder, typelist):
>> def my_decorator(function_name, ast):
>> functions = {}
>> for typename in typelist:
>> new_name = "%s_%s" % (function_name, typename)
>> functions[name] = replace_ctype(ast, placeholder, typename)
>> return functions
>> return function_decorator
>> @@ # this token sequence cannot occur in Python, so it's a safe end-marker
>>
>> # Compile-time function decorator
>> # Results in two cdef functions named sum_double and sum_int
>> @@expand_types("T", ["double", "int"])
>> cdef T sum(np.ndarray[T] arr):
>> cdef T start = 0;
>> for i in range(arr.size):
>> start += arr[i]
>> return start
>>
>> I don't know if this is a good idea, but it seems like it'd be very
>> easy to do on the Cython side, fairly clean, and be dramatically less
>> horrible than all the ad-hoc templating stuff people do now.
>> Presumably there'd be strict limits on how much backwards
>> compatibility we'd be willing to guarantee for code that went poking
>> around in the AST by hand, but a small handful of functions like my
>> notional "replace_ctype" would go a long way, and wouldn't impose much
>> of a compatibility burden.
>>
>> -- Nathaniel
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
>
> In general I would like better meta-programming support, maybe even
> allow defining new operators (although I'm not sure any of it is very
> pythonic), but for templates I think fused types should be used, or
> improved when they fall short. Maybe a plugin system could also help
> people.
I hadn't seen that, no -- thanks for the link.
I have to say that the examples in that link, though, give me the
impression of a cool solution looking for a problem. I've never wished
I could symbolically differentiate Python expressions at compile time,
or create a mutant Python+SQL hybrid language. Actually I guess I've
only missed define-syntax once in maybe 10 years of hacking in
Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
will peek at the caller's syntax tree to automagically label the axes
as "x" and "log(y)", and that can't be done in Python. But that's not
exactly a convincing argument for a macro system.
But generating optimized code is Cython's whole selling point, and
people really are doing klugey tricks with string-based preprocessors
just to generate multiple copies of loops in Cython and C.
Also, fused types are great, but: (1) IIUC you can't actually do
ndarray[fused_type] yet, which speaks to the feature's complexity, and
(2) to handle Wes's original example on his blog (duplicating a bunch
of code between a "sum" path and a "product" path), you'd actually
need something like "fused operators", which aren't even on the
horizon. So it seems unlikely that fused types will grow to cover all
these cases in the near future.
Of course some experimentation would be needed to find the right
syntax and convenience functions for this feature too, so maybe I'm
just being over-optimistic and it would also turn out to be very
complicated :-). But it seems like some simple AST search/replace
functions would get you a long way.
- N
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
On 29 April 2012 08:42, Nathaniel Smith wrote:
> On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
> wrote:
>> On 28 April 2012 22:04, Nathaniel Smith wrote:
>>> Was chatting with Wes today about the usual problem many of us have
>>> encountered with needing to use some sort of templating system to
>>> generate code handling multiple types, operations, etc., and a wacky
>>> idea occurred to me. So I thought I'd through it out here.
>>>
>>> What if we added a simple macro facility to Cython, that worked at the
>>> AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
>>> Basically some way to write arbitrary Python code into a .pyx file
>>> that gets executed at compile time and can transform the AST, plus
>>> some nice convenience APIs for simple transformations.
>>>
>>> E.g., if we steal the illegal token sequence @@ as our marker, we
>>> could have something like:
>>>
>>> @@ # alone on a line, starts a block of Python code
>>> from Cython.MacroUtil import replace_ctype
>>> def expand_types(placeholder, typelist):
>>> def my_decorator(function_name, ast):
>>> functions = {}
>>> for typename in typelist:
>>> new_name = "%s_%s" % (function_name, typename)
>>> functions[name] = replace_ctype(ast, placeholder, typename)
>>> return functions
>>> return function_decorator
>>> @@ # this token sequence cannot occur in Python, so it's a safe end-marker
>>>
>>> # Compile-time function decorator
>>> # Results in two cdef functions named sum_double and sum_int
>>> @@expand_types("T", ["double", "int"])
>>> cdef T sum(np.ndarray[T] arr):
>>> cdef T start = 0;
>>> for i in range(arr.size):
>>> start += arr[i]
>>> return start
>>>
>>> I don't know if this is a good idea, but it seems like it'd be very
>>> easy to do on the Cython side, fairly clean, and be dramatically less
>>> horrible than all the ad-hoc templating stuff people do now.
>>> Presumably there'd be strict limits on how much backwards
>>> compatibility we'd be willing to guarantee for code that went poking
>>> around in the AST by hand, but a small handful of functions like my
>>> notional "replace_ctype" would go a long way, and wouldn't impose much
>>> of a compatibility burden.
>>>
>>> -- Nathaniel
>>> ___
>>> cython-devel mailing list
>>> cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
>>
>> In general I would like better meta-programming support, maybe even
>> allow defining new operators (although I'm not sure any of it is very
>> pythonic), but for templates I think fused types should be used, or
>> improved when they fall short. Maybe a plugin system could also help
>> people.
>
> I hadn't seen that, no -- thanks for the link.
>
> I have to say that the examples in that link, though, give me the
> impression of a cool solution looking for a problem. I've never wished
> I could symbolically differentiate Python expressions at compile time,
> or create a mutant Python+SQL hybrid language. Actually I guess I've
> only missed define-syntax once in maybe 10 years of hacking in
> Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
> will peek at the caller's syntax tree to automagically label the axes
> as "x" and "log(y)", and that can't be done in Python. But that's not
> exactly a convincing argument for a macro system.
>
> But generating optimized code is Cython's whole selling point, and
> people really are doing klugey tricks with string-based preprocessors
> just to generate multiple copies of loops in Cython and C.
>
> Also, fused types are great, but: (1) IIUC you can't actually do
> ndarray[fused_type] yet, which speaks to the feature's complexity, and
What? Yes you can do that.
> (2) to handle Wes's original example on his blog (duplicating a bunch
> of code between a "sum" path and a "product" path), you'd actually
> need something like "fused operators", which aren't even on the
> horizon. So it seems unlikely that fused types will grow to cover all
> these cases in the near future.
Although it doesn't handle contiguity or dimensional differences,
currently the efficient fused operator is a function pointer. Wouldn't
passing in a float64_t (*reducer)(float64_t, float64_t) work in this
case (in the face of multiple types, you can have fused parameters in
the function pointer as well)?
I agree with Dag that Julia has nice metaprogramming support, maybe
functions could take arbitrary compile time expressions as extra
arguments.
> Of course some experimentation would be needed to find the right
> syntax and convenience functions for this feature too, so maybe I'm
> just being over-optimistic and it would also turn out to be very
> complicated :-). But it seems like some simple AST search/replace
> functions would get you a long way.
>
> - N
> ___
> cython-devel mailing list
> cyth
Re: [Cython] dynamically compile and import pure python modules
On 29 April 2012 05:50, Stefan Behnel wrote: > mark florisson, 28.04.2012 21:55: >> On 28 April 2012 19:39, Stefan Behnel wrote: >>> mark florisson, 28.04.2012 20:18: On 28 April 2012 18:55, Stefan Behnel wrote: > mark florisson, 27.04.2012 22:38: >> On 27 April 2012 21:16, Stefan Behnel wrote: >>> What about this deal: we remove the hard bootstrap dependency on the >>> fused >>> types code (and maybe other Cython specific features) and require its >>> compilation at install time in Py2.4 (and maybe even 2.5). That would >>> allow >>> us to use newer Python syntax (and even Cython supported syntax) there >>> (except for fused types, obviously). Failure to compile the module in >>> Python 2.4/5 at install time would then abort the installation. Bad luck >>> for the user, but easy to fix by installing a newer Python version. >>> >>> That would give us the advantage of not needing to pollute user home >>> directories with shared libraries at runtime (which I would consider a >>> very >>> annoying property). >> >> I think it's fine to require compiling in the installed case (or will >> that be a problem for some package managers?). In the non-installed >> case with python versions smaller than needed, would you prefer a >> pyximport or an error message telling you to install Cython? Because >> for development you really don't want to install every time. > > I think it's fine to require at least Python 2.6 for Cython core > development. Just the installation (basically, what we test in Jenkins > anyway) should work in Py2.4 and shouldn't require any rebuilds at > runtime. Well, sometimes stuff works in say, 2.7 but fails in 2.4. In that case you really have to test with the failing python versions, which means you'd have to reinstall every time you want to try the tests again (this is handled automatically for py3k, which runs the 2to3 tool). >>> >>> The number of times I recently ran tests in Py2.4 myself is really not >>> worth mentioning. Most of the time, when something fails there, the error I >>> get in Jenkins is so obvious that I just commit an untested fix for it. >> >> In my experience that still fails quite often, you may still forget >> some test or accidentally add some whitespace, and then you're going >> to build everything on Jenkins only to realize one hour later that >> something is still broken. Maybe it's because of the buffer >> differences and because 2.4 is the first thing that runs on Jenkins, >> but I test quite often in 2.4. > > That may be the reason. > > Still, how much overhead is it really to run "setup.py build_ext -i"? > Especially compared to getting the exact same compilation overhead with an > import time triggered rebuild? > The compilation overhead is irrelevant, the important difference is that one is *automatic* and hence preferable. Knowing myself, I will forget that module X, Y and Z need to be compiled, which means when I change the code I don't see the changes take effect, because I forgot to rebuild the modules. After a while I will start to question my sanity and insert a print statement at module level, only to release that I had to rebuild after every single change. >>> I think it's really acceptable to require a run of "setup.py build_ext -i" >>> for local developer testing in Py2.4. >>> I'm also sure many users just clone from git, add the directory to PYTHONPATH and work from there. >>> >>> I'm sure there are close to no users who try to do that with Py2.4 these >>> days. Maybe there are some who do it with Py2.5, but we are not currently >>> considering to break a plain Python run there, AFAICT. >>> >>> I think the normal way users employ Cython is after a proper installation. >>> So I guess what I'm saying is, it's fine to mandate compilation at compile time (don't allow flags to disable compilation), and (for me), pyximport is totally fine, but Cython must be workable (all functionality), without needing to install or build, in all versions. >>> >>> Workable, ok. But if fused types are only available in a compiled installed >>> version under Python 2.4, that's maybe not optimal but certainly >>> acceptable. Users of Py2.4 should be used to suffering anyway. >> >> That's a really confusing statement. If they are used to suffering, >> they why can't they bear a runtime-compiled module if they didn't >> install? :) If a user installs she will have at least one compiled >> module in the system, but if she doesn't install, she will also have >> one compiled module. >> >> I'm kind of wondering though, if this is really such a big problem, >> then it means we can also never cache any user JIT-compiled code (like >> e.g. OpenCL)? > > That's a different situation because it's the user's decision to use a > (caching) JIT compiler in the first place. We don't enforce that. > > Basically, what I'm s
Re: [Cython] [cython-users] cimport numpy fails with Python 3 semantics
mark florisson, 28.04.2012 21:57: > On 28 April 2012 19:50, Stefan Behnel wrote: >> mark florisson, 28.04.2012 20:33: >>> I think each module should have its own language level, so I think >>> that's a bug. I think the rules should be: >>> >>> - if passed as command line argument, use that for all cimported >>> modules, unless they define their only language level through the >>> directive >>> - if set as a directive, the language level will apply only to that >>> module >> >> That's how it works. We don't run the tests with language level 3 in >> Jenkins because the majority of the tests is not meant to be run with Py3 >> semantics. Maybe it's time to add a numpy_cy3 test. >> >> If there are more problems than just this (which was a bug in numpy.pxd), >> we may consider setting language level 2 explicitly in numpy.pxd. > > Ah, great. Do we have any documentation for that? We do now. ;) However, I'm not sure cimported .pxd files should always inherit the language_level setting. It's somewhat of a grey area because user provided .pxd files would benefit from it since they likely all use the same language level as the main module, whereas the Cython shipped (and otherwise globally installed) .pxd files wouldn't gain anything and could potentially break. I think we may want to keep the current behaviour and set the language level explicitly in the few shipped .pxd files that are not language level agnostic (i.e. those that actually contain code). Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
