Re: [Cython] [cython-users] Conditional import in pure Python mode
On 30 April 2012 13:14, Ian Bell wrote:
>
> On Sun, Apr 29, 2012 at 10:58 PM, mark florisson
> wrote:
>>
>> On 29 April 2012 01:33, Ian Bell wrote:
>> > Hello Cython users,
>> >
>> > I haven't the foggiest idea how easy this would be to implement, or how
>> > to
>> > do it in the first place, but my idea is to be able to have some sort of
>> > idiom like
>> >
>> > if cython.compiled:
>> > cython.import('from libc.math cimport sin')
>> > else:
>> > from math import sin
>> >
>> > that would allow for a pure Python file to still have 100% CPython code
>> > in
>> > it, but when it goes through the .py+.pxd-->.pyd compilation, use the
>> > math.h
>> > version instead. Is there any way that this could work? I can (and
>> > will)
>> > hack together a really nasty preprocessor that can edit the .py file at
>> > Cython build time, but it is a nasty hack, and very un-Pythonic. For
>> > me,
>> > the computational efficiency using the math.h trig functions justify a
>> > bit
>> > of nastiness.
>> >
>> > Regards,
>> > Ian
>>
>> I think a cython.cimport would be nice. If you want to have a crack at
>> it, you may want to look in Cython/Compiler/ParseTreeTransforms.py at
>> the TransformBuiltinMethods transform, and also at the
>> Cython/Shadow.py module.
>
>
> Mark,
>
> Any chance you can give me a wee bit more help? I can see how you need to
> add a cimport function in Shadow.py. Supposing I pass it an import string
> that it will parse for the import, whereto from there? There's quite a lot
> of code to dig through.
>
> Ian
>
Certainly, it's great to see some enthusiasm. So first, it's important
to determine how you want it. You can either simulate importing the
rightmost module from the package, or you can simulate importing
everything from the package. Personally I think if you say
cython.cimport_module("libc.stdio") you want conceptually the stdio
module to be returned. Maybe a 'cimport_star' function could cimport
everything from the scope, but I'm not sure how great an idea that is
in pure mode (likely not a good one).
So lets assume we want to use the following syntax: stdio =
cython.cimport_module("libc.stdio").
In the TransformBuiltinMethods you add another case to the
visit_SimpleCallNode, which will match for the function name
"cimport_module" (which is already known to be used as an attribute of
the Cython module). TransformBuiltinMethods is a subclass of
Visitor.EnvTransform, which keeps track of the scope (e.g. if the node
is in a function, the local function scope, etc). This class is a
subclass of CythonTransform, which has a Main.Context set up in
Pipeline.py, accessible through the 'self.context' attribute.
Main.Context has methods to find and process pxd files, i.e. through
the find_pxd_file and the process_pxd methods (or maybe the
find_module method will work here).
So you want to validate that the string that gets passed to the
'cimport_module' function call is a string literal, and that this is
in fact happening from the module scope. You can then create an Entry
with an as_module attribute and put it in the current scope. The
as_module attribute holds the scope of the cimported pxd file, which
means its a "cimported module". You can do this through the
'declare_module' method of the current scope
(self.current_env().declare_module(...)) (see Symtab.py for that
method).
I hope that helps you get set up up somewhat, don't hesitate to bug
the cython-devel mailing list with any further questions. BTW, we
probably want to formally discuss the exact syntax and semantics
before implementing it, so I think it will be a good idea to summarize
what you want and how you want it on the cython-dev mailing list. Good
luck :)
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
wrote:
> On 29 April 2012 08:42, Nathaniel Smith wrote:
>> On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
>> wrote:
>>> On 28 April 2012 22:04, Nathaniel Smith wrote:
Was chatting with Wes today about the usual problem many of us have
encountered with needing to use some sort of templating system to
generate code handling multiple types, operations, etc., and a wacky
idea occurred to me. So I thought I'd through it out here.
What if we added a simple macro facility to Cython, that worked at the
AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
Basically some way to write arbitrary Python code into a .pyx file
that gets executed at compile time and can transform the AST, plus
some nice convenience APIs for simple transformations.
E.g., if we steal the illegal token sequence @@ as our marker, we
could have something like:
@@ # alone on a line, starts a block of Python code
from Cython.MacroUtil import replace_ctype
def expand_types(placeholder, typelist):
def my_decorator(function_name, ast):
functions = {}
for typename in typelist:
new_name = "%s_%s" % (function_name, typename)
functions[name] = replace_ctype(ast, placeholder, typename)
return functions
return function_decorator
@@ # this token sequence cannot occur in Python, so it's a safe end-marker
# Compile-time function decorator
# Results in two cdef functions named sum_double and sum_int
@@expand_types("T", ["double", "int"])
cdef T sum(np.ndarray[T] arr):
cdef T start = 0;
for i in range(arr.size):
start += arr[i]
return start
I don't know if this is a good idea, but it seems like it'd be very
easy to do on the Cython side, fairly clean, and be dramatically less
horrible than all the ad-hoc templating stuff people do now.
Presumably there'd be strict limits on how much backwards
compatibility we'd be willing to guarantee for code that went poking
around in the AST by hand, but a small handful of functions like my
notional "replace_ctype" would go a long way, and wouldn't impose much
of a compatibility burden.
-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
>>>
>>> In general I would like better meta-programming support, maybe even
>>> allow defining new operators (although I'm not sure any of it is very
>>> pythonic), but for templates I think fused types should be used, or
>>> improved when they fall short. Maybe a plugin system could also help
>>> people.
>>
>> I hadn't seen that, no -- thanks for the link.
>>
>> I have to say that the examples in that link, though, give me the
>> impression of a cool solution looking for a problem. I've never wished
>> I could symbolically differentiate Python expressions at compile time,
>> or create a mutant Python+SQL hybrid language. Actually I guess I've
>> only missed define-syntax once in maybe 10 years of hacking in
>> Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
>> will peek at the caller's syntax tree to automagically label the axes
>> as "x" and "log(y)", and that can't be done in Python. But that's not
>> exactly a convincing argument for a macro system.
>>
>> But generating optimized code is Cython's whole selling point, and
>> people really are doing klugey tricks with string-based preprocessors
>> just to generate multiple copies of loops in Cython and C.
>>
>> Also, fused types are great, but: (1) IIUC you can't actually do
>> ndarray[fused_type] yet, which speaks to the feature's complexity, and
>
> What? Yes you can do that.
I haven't been able to get ndarray[fused_t] to work as we've discussed
off-list. In your own words "Unfortunately, the automatic buffer
dispatch didn't make it into 0.16, so you need to manually
specialize". I'm a bit hamstrung by other users needing to be able to
compile pandas using the latest released Cython.
>> (2) to handle Wes's original example on his blog (duplicating a bunch
>> of code between a "sum" path and a "product" path), you'd actually
>> need something like "fused operators", which aren't even on the
>> horizon. So it seems unlikely that fused types will grow to cover all
>> these cases in the near future.
>
> Although it doesn't handle contiguity or dimensional differences,
> currently the efficient fused operator is a function pointer. Wouldn't
> passing in a float64_t (*reducer)(float64_t, float64_t) work in this
> case (in the face of multiple types, you can have fused parameters in
> the function pointer as well)?
I have to think that using function pointers everywhere is going to
lose out
Re: [Cython] [cython-users] Conditional import in pure Python mode
mark florisson, 30.04.2012 15:24:
> So lets assume we want to use the following syntax: stdio =
> cython.cimport_module("libc.stdio").
>
> In the TransformBuiltinMethods you add another case to the
> visit_SimpleCallNode
That seems way too late in the pipeline to me (see Pipeline.py). I think
this is better (although maybe still not perfectly) suited for
"InterpretCompilerDirectives".
> we probably want to formally discuss the exact syntax and semantics
> before implementing it, so I think it will be a good idea to summarize
> what you want and how you want it on the cython-dev mailing list.
Absolutely.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Conditional import in pure Python mode
On 30 April 2012 14:55, Stefan Behnel wrote:
> mark florisson, 30.04.2012 15:24:
>> So lets assume we want to use the following syntax: stdio =
>> cython.cimport_module("libc.stdio").
>>
>> In the TransformBuiltinMethods you add another case to the
>> visit_SimpleCallNode
>
> That seems way too late in the pipeline to me (see Pipeline.py). I think
> this is better (although maybe still not perfectly) suited for
> "InterpretCompilerDirectives".
>
Ah good point, it seems to run pretty late. It needs to be before the
declarations are analyzed.
>> we probably want to formally discuss the exact syntax and semantics
>> before implementing it, so I think it will be a good idea to summarize
>> what you want and how you want it on the cython-dev mailing list.
>
> Absolutely.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
On 30 April 2012 14:49, Wes McKinney wrote:
> On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
> wrote:
>> On 29 April 2012 08:42, Nathaniel Smith wrote:
>>> On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
>>> wrote:
On 28 April 2012 22:04, Nathaniel Smith wrote:
> Was chatting with Wes today about the usual problem many of us have
> encountered with needing to use some sort of templating system to
> generate code handling multiple types, operations, etc., and a wacky
> idea occurred to me. So I thought I'd through it out here.
>
> What if we added a simple macro facility to Cython, that worked at the
> AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
> Basically some way to write arbitrary Python code into a .pyx file
> that gets executed at compile time and can transform the AST, plus
> some nice convenience APIs for simple transformations.
>
> E.g., if we steal the illegal token sequence @@ as our marker, we
> could have something like:
>
> @@ # alone on a line, starts a block of Python code
> from Cython.MacroUtil import replace_ctype
> def expand_types(placeholder, typelist):
> def my_decorator(function_name, ast):
> functions = {}
> for typename in typelist:
> new_name = "%s_%s" % (function_name, typename)
> functions[name] = replace_ctype(ast, placeholder, typename)
> return functions
> return function_decorator
> @@ # this token sequence cannot occur in Python, so it's a safe end-marker
>
> # Compile-time function decorator
> # Results in two cdef functions named sum_double and sum_int
> @@expand_types("T", ["double", "int"])
> cdef T sum(np.ndarray[T] arr):
> cdef T start = 0;
> for i in range(arr.size):
> start += arr[i]
> return start
>
> I don't know if this is a good idea, but it seems like it'd be very
> easy to do on the Cython side, fairly clean, and be dramatically less
> horrible than all the ad-hoc templating stuff people do now.
> Presumably there'd be strict limits on how much backwards
> compatibility we'd be willing to guarantee for code that went poking
> around in the AST by hand, but a small handful of functions like my
> notional "replace_ctype" would go a long way, and wouldn't impose much
> of a compatibility burden.
>
> -- Nathaniel
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
In general I would like better meta-programming support, maybe even
allow defining new operators (although I'm not sure any of it is very
pythonic), but for templates I think fused types should be used, or
improved when they fall short. Maybe a plugin system could also help
people.
>>>
>>> I hadn't seen that, no -- thanks for the link.
>>>
>>> I have to say that the examples in that link, though, give me the
>>> impression of a cool solution looking for a problem. I've never wished
>>> I could symbolically differentiate Python expressions at compile time,
>>> or create a mutant Python+SQL hybrid language. Actually I guess I've
>>> only missed define-syntax once in maybe 10 years of hacking in
>>> Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
>>> will peek at the caller's syntax tree to automagically label the axes
>>> as "x" and "log(y)", and that can't be done in Python. But that's not
>>> exactly a convincing argument for a macro system.
>>>
>>> But generating optimized code is Cython's whole selling point, and
>>> people really are doing klugey tricks with string-based preprocessors
>>> just to generate multiple copies of loops in Cython and C.
>>>
>>> Also, fused types are great, but: (1) IIUC you can't actually do
>>> ndarray[fused_type] yet, which speaks to the feature's complexity, and
>>
>> What? Yes you can do that.
>
> I haven't been able to get ndarray[fused_t] to work as we've discussed
> off-list. In your own words "Unfortunately, the automatic buffer
> dispatch didn't make it into 0.16, so you need to manually
> specialize". I'm a bit hamstrung by other users needing to be able to
> compile pandas using the latest released Cython.
Well, as I said, it does work, but you need to tell Cython which type
you meant. If you don't want to do that, you have to use this branch:
https://github.com/markflorisson88/cython/tree/_fused_dispatch_rebased
. This never made it in since we had no consensus on whether to allow
the compiler to bootstrap itself and because of possible immaturity of
the branch.
So what doesn't work is automatic dispatch for Python functions (def
functions and the object version of a cpdef function). They don't
automatically select the right specialization for buffer arguments.
Anyt
Re: [Cython] Wacky idea: proper macros
On 30 April 2012 14:49, Wes McKinney wrote:
> On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
> wrote:
>> On 29 April 2012 08:42, Nathaniel Smith wrote:
>>> On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
>>> wrote:
On 28 April 2012 22:04, Nathaniel Smith wrote:
> Was chatting with Wes today about the usual problem many of us have
> encountered with needing to use some sort of templating system to
> generate code handling multiple types, operations, etc., and a wacky
> idea occurred to me. So I thought I'd through it out here.
>
> What if we added a simple macro facility to Cython, that worked at the
> AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
> Basically some way to write arbitrary Python code into a .pyx file
> that gets executed at compile time and can transform the AST, plus
> some nice convenience APIs for simple transformations.
>
> E.g., if we steal the illegal token sequence @@ as our marker, we
> could have something like:
>
> @@ # alone on a line, starts a block of Python code
> from Cython.MacroUtil import replace_ctype
> def expand_types(placeholder, typelist):
> def my_decorator(function_name, ast):
> functions = {}
> for typename in typelist:
> new_name = "%s_%s" % (function_name, typename)
> functions[name] = replace_ctype(ast, placeholder, typename)
> return functions
> return function_decorator
> @@ # this token sequence cannot occur in Python, so it's a safe end-marker
>
> # Compile-time function decorator
> # Results in two cdef functions named sum_double and sum_int
> @@expand_types("T", ["double", "int"])
> cdef T sum(np.ndarray[T] arr):
> cdef T start = 0;
> for i in range(arr.size):
> start += arr[i]
> return start
>
> I don't know if this is a good idea, but it seems like it'd be very
> easy to do on the Cython side, fairly clean, and be dramatically less
> horrible than all the ad-hoc templating stuff people do now.
> Presumably there'd be strict limits on how much backwards
> compatibility we'd be willing to guarantee for code that went poking
> around in the AST by hand, but a small handful of functions like my
> notional "replace_ctype" would go a long way, and wouldn't impose much
> of a compatibility burden.
>
> -- Nathaniel
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
In general I would like better meta-programming support, maybe even
allow defining new operators (although I'm not sure any of it is very
pythonic), but for templates I think fused types should be used, or
improved when they fall short. Maybe a plugin system could also help
people.
>>>
>>> I hadn't seen that, no -- thanks for the link.
>>>
>>> I have to say that the examples in that link, though, give me the
>>> impression of a cool solution looking for a problem. I've never wished
>>> I could symbolically differentiate Python expressions at compile time,
>>> or create a mutant Python+SQL hybrid language. Actually I guess I've
>>> only missed define-syntax once in maybe 10 years of hacking in
>>> Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
>>> will peek at the caller's syntax tree to automagically label the axes
>>> as "x" and "log(y)", and that can't be done in Python. But that's not
>>> exactly a convincing argument for a macro system.
>>>
>>> But generating optimized code is Cython's whole selling point, and
>>> people really are doing klugey tricks with string-based preprocessors
>>> just to generate multiple copies of loops in Cython and C.
>>>
>>> Also, fused types are great, but: (1) IIUC you can't actually do
>>> ndarray[fused_type] yet, which speaks to the feature's complexity, and
>>
>> What? Yes you can do that.
>
> I haven't been able to get ndarray[fused_t] to work as we've discussed
> off-list. In your own words "Unfortunately, the automatic buffer
> dispatch didn't make it into 0.16, so you need to manually
> specialize". I'm a bit hamstrung by other users needing to be able to
> compile pandas using the latest released Cython.
Also, if something doesn't work in 0.16, please report it. We reverted
the numpy changes that broke pandas, so if there are other changes
that break stuff, please inform us.
>>> (2) to handle Wes's original example on his blog (duplicating a bunch
>>> of code between a "sum" path and a "product" path), you'd actually
>>> need something like "fused operators", which aren't even on the
>>> horizon. So it seems unlikely that fused types will grow to cover all
>>> these cases in the near future.
>>
>> Although it doesn't handle contiguity or dimensional differences,
>>
Re: [Cython] Wacky idea: proper macros
On Mon, Apr 30, 2012 at 11:19 AM, mark florisson
wrote:
> On 30 April 2012 14:49, Wes McKinney wrote:
>> On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
>> wrote:
>>> On 29 April 2012 08:42, Nathaniel Smith wrote:
On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
wrote:
> On 28 April 2012 22:04, Nathaniel Smith wrote:
>> Was chatting with Wes today about the usual problem many of us have
>> encountered with needing to use some sort of templating system to
>> generate code handling multiple types, operations, etc., and a wacky
>> idea occurred to me. So I thought I'd through it out here.
>>
>> What if we added a simple macro facility to Cython, that worked at the
>> AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
>> Basically some way to write arbitrary Python code into a .pyx file
>> that gets executed at compile time and can transform the AST, plus
>> some nice convenience APIs for simple transformations.
>>
>> E.g., if we steal the illegal token sequence @@ as our marker, we
>> could have something like:
>>
>> @@ # alone on a line, starts a block of Python code
>> from Cython.MacroUtil import replace_ctype
>> def expand_types(placeholder, typelist):
>> def my_decorator(function_name, ast):
>> functions = {}
>> for typename in typelist:
>> new_name = "%s_%s" % (function_name, typename)
>> functions[name] = replace_ctype(ast, placeholder, typename)
>> return functions
>> return function_decorator
>> @@ # this token sequence cannot occur in Python, so it's a safe
>> end-marker
>>
>> # Compile-time function decorator
>> # Results in two cdef functions named sum_double and sum_int
>> @@expand_types("T", ["double", "int"])
>> cdef T sum(np.ndarray[T] arr):
>> cdef T start = 0;
>> for i in range(arr.size):
>> start += arr[i]
>> return start
>>
>> I don't know if this is a good idea, but it seems like it'd be very
>> easy to do on the Cython side, fairly clean, and be dramatically less
>> horrible than all the ad-hoc templating stuff people do now.
>> Presumably there'd be strict limits on how much backwards
>> compatibility we'd be willing to guarantee for code that went poking
>> around in the AST by hand, but a small handful of functions like my
>> notional "replace_ctype" would go a long way, and wouldn't impose much
>> of a compatibility burden.
>>
>> -- Nathaniel
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
>
> In general I would like better meta-programming support, maybe even
> allow defining new operators (although I'm not sure any of it is very
> pythonic), but for templates I think fused types should be used, or
> improved when they fall short. Maybe a plugin system could also help
> people.
I hadn't seen that, no -- thanks for the link.
I have to say that the examples in that link, though, give me the
impression of a cool solution looking for a problem. I've never wished
I could symbolically differentiate Python expressions at compile time,
or create a mutant Python+SQL hybrid language. Actually I guess I've
only missed define-syntax once in maybe 10 years of hacking in
Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
will peek at the caller's syntax tree to automagically label the axes
as "x" and "log(y)", and that can't be done in Python. But that's not
exactly a convincing argument for a macro system.
But generating optimized code is Cython's whole selling point, and
people really are doing klugey tricks with string-based preprocessors
just to generate multiple copies of loops in Cython and C.
Also, fused types are great, but: (1) IIUC you can't actually do
ndarray[fused_type] yet, which speaks to the feature's complexity, and
>>>
>>> What? Yes you can do that.
>>
>> I haven't been able to get ndarray[fused_t] to work as we've discussed
>> off-list. In your own words "Unfortunately, the automatic buffer
>> dispatch didn't make it into 0.16, so you need to manually
>> specialize". I'm a bit hamstrung by other users needing to be able to
>> compile pandas using the latest released Cython.
>
> Well, as I said, it does work, but you need to tell Cython which type
> you meant. If you don't want to do that, you have to use this branch:
> https://github.com/markflorisson88/cython/tree/_fused_dispatch_rebased
> . This never made it in since we had no consensus on whether to allow
> the compiler to bootstrap itself and because of possible immaturity of
> the branch.
>
> So what doesn't work is automatic
Re: [Cython] Wacky idea: proper macros
On 04/30/2012 06:30 PM, Wes McKinney wrote:
On Mon, Apr 30, 2012 at 11:19 AM, mark florisson
wrote:
On 30 April 2012 14:49, Wes McKinney wrote:
On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
wrote:
On 29 April 2012 08:42, Nathaniel Smith wrote:
On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
wrote:
On 28 April 2012 22:04, Nathaniel Smith wrote:
Was chatting with Wes today about the usual problem many of us have
encountered with needing to use some sort of templating system to
generate code handling multiple types, operations, etc., and a wacky
idea occurred to me. So I thought I'd through it out here.
What if we added a simple macro facility to Cython, that worked at the
AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.)
Basically some way to write arbitrary Python code into a .pyx file
that gets executed at compile time and can transform the AST, plus
some nice convenience APIs for simple transformations.
E.g., if we steal the illegal token sequence @@ as our marker, we
could have something like:
@@ # alone on a line, starts a block of Python code
from Cython.MacroUtil import replace_ctype
def expand_types(placeholder, typelist):
def my_decorator(function_name, ast):
functions = {}
for typename in typelist:
new_name = "%s_%s" % (function_name, typename)
functions[name] = replace_ctype(ast, placeholder, typename)
return functions
return function_decorator
@@ # this token sequence cannot occur in Python, so it's a safe end-marker
# Compile-time function decorator
# Results in two cdef functions named sum_double and sum_int
@@expand_types("T", ["double", "int"])
cdef T sum(np.ndarray[T] arr):
cdef T start = 0;
for i in range(arr.size):
start += arr[i]
return start
I don't know if this is a good idea, but it seems like it'd be very
easy to do on the Cython side, fairly clean, and be dramatically less
horrible than all the ad-hoc templating stuff people do now.
Presumably there'd be strict limits on how much backwards
compatibility we'd be willing to guarantee for code that went poking
around in the AST by hand, but a small handful of functions like my
notional "replace_ctype" would go a long way, and wouldn't impose much
of a compatibility burden.
-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Have you looked at http://wiki.cython.org/enhancements/metaprogramming ?
In general I would like better meta-programming support, maybe even
allow defining new operators (although I'm not sure any of it is very
pythonic), but for templates I think fused types should be used, or
improved when they fall short. Maybe a plugin system could also help
people.
I hadn't seen that, no -- thanks for the link.
I have to say that the examples in that link, though, give me the
impression of a cool solution looking for a problem. I've never wished
I could symbolically differentiate Python expressions at compile time,
or create a mutant Python+SQL hybrid language. Actually I guess I've
only missed define-syntax once in maybe 10 years of hacking in
Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
will peek at the caller's syntax tree to automagically label the axes
as "x" and "log(y)", and that can't be done in Python. But that's not
exactly a convincing argument for a macro system.
But generating optimized code is Cython's whole selling point, and
people really are doing klugey tricks with string-based preprocessors
just to generate multiple copies of loops in Cython and C.
Also, fused types are great, but: (1) IIUC you can't actually do
ndarray[fused_type] yet, which speaks to the feature's complexity, and
What? Yes you can do that.
I haven't been able to get ndarray[fused_t] to work as we've discussed
off-list. In your own words "Unfortunately, the automatic buffer
dispatch didn't make it into 0.16, so you need to manually
specialize". I'm a bit hamstrung by other users needing to be able to
compile pandas using the latest released Cython.
Well, as I said, it does work, but you need to tell Cython which type
you meant. If you don't want to do that, you have to use this branch:
https://github.com/markflorisson88/cython/tree/_fused_dispatch_rebased
. This never made it in since we had no consensus on whether to allow
the compiler to bootstrap itself and because of possible immaturity of
the branch.
So what doesn't work is automatic dispatch for Python functions (def
functions and the object version of a cpdef function). They don't
automatically select the right specialization for buffer arguments.
Anything else should work, otherwise it's a bug.
Note also that figuring out which specialization to call dynamically
(i.e. not from Cython space at compile time, but from Python space at
runtime) has non-trivial overhead on top of just argument unpacking.
But you can't say "doesn't work" without giving a co
Re: [Cython] Wacky idea: proper macros
On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn wrote: > JIT is really the way to go. It is one thing that a JIT could optimize the > case where you pass a callback to a function and inline it run-time. But > even if it doesn't get that fancy, it'd be great to just be able to write > something like "cython.eval(s)" and have that be compiled (I guess you could > do that now, but the sheer overhead of the C compiler and all the .so files > involved means nobody would sanely use that as the main way of stringing > together something like pandas). The overhead of running a fully optimizing compiler over pandas on every import is pretty high, though. You can come up with various caching mechanisms, but they all mean introducing some kind of compile time/run time distinction. So I'm skeptical we'll just be able to get rid of that concept, even in a brave new LLVM/PyPy/Julia world. -- Nathaniel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
On Mon, Apr 30, 2012 at 4:49 PM, Dag Sverre Seljebotn
wrote:
> On 04/30/2012 06:30 PM, Wes McKinney wrote:
>>
>> On Mon, Apr 30, 2012 at 11:19 AM, mark florisson
>> wrote:
>>>
>>> On 30 April 2012 14:49, Wes McKinney wrote:
On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
wrote:
>
> On 29 April 2012 08:42, Nathaniel Smith wrote:
>>
>> On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
>> wrote:
>>>
>>> On 28 April 2012 22:04, Nathaniel Smith wrote:
Was chatting with Wes today about the usual problem many of us have
encountered with needing to use some sort of templating system to
generate code handling multiple types, operations, etc., and a wacky
idea occurred to me. So I thought I'd through it out here.
What if we added a simple macro facility to Cython, that worked at
the
AST level? (I.e. I'm talking lisp-style macros, *not* C-style
macros.)
Basically some way to write arbitrary Python code into a .pyx file
that gets executed at compile time and can transform the AST, plus
some nice convenience APIs for simple transformations.
E.g., if we steal the illegal token sequence @@ as our marker, we
could have something like:
@@ # alone on a line, starts a block of Python code
from Cython.MacroUtil import replace_ctype
def expand_types(placeholder, typelist):
def my_decorator(function_name, ast):
functions = {}
for typename in typelist:
new_name = "%s_%s" % (function_name, typename)
functions[name] = replace_ctype(ast, placeholder, typename)
return functions
return function_decorator
@@ # this token sequence cannot occur in Python, so it's a safe
end-marker
# Compile-time function decorator
# Results in two cdef functions named sum_double and sum_int
@@expand_types("T", ["double", "int"])
cdef T sum(np.ndarray[T] arr):
cdef T start = 0;
for i in range(arr.size):
start += arr[i]
return start
I don't know if this is a good idea, but it seems like it'd be very
easy to do on the Cython side, fairly clean, and be dramatically
less
horrible than all the ad-hoc templating stuff people do now.
Presumably there'd be strict limits on how much backwards
compatibility we'd be willing to guarantee for code that went poking
around in the AST by hand, but a small handful of functions like my
notional "replace_ctype" would go a long way, and wouldn't impose
much
of a compatibility burden.
-- Nathaniel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>>
>>> Have you looked at
>>> http://wiki.cython.org/enhancements/metaprogramming ?
>>>
>>> In general I would like better meta-programming support, maybe even
>>> allow defining new operators (although I'm not sure any of it is very
>>> pythonic), but for templates I think fused types should be used, or
>>> improved when they fall short. Maybe a plugin system could also help
>>> people.
>>
>>
>> I hadn't seen that, no -- thanks for the link.
>>
>> I have to say that the examples in that link, though, give me the
>> impression of a cool solution looking for a problem. I've never wished
>> I could symbolically differentiate Python expressions at compile time,
>> or create a mutant Python+SQL hybrid language. Actually I guess I've
>> only missed define-syntax once in maybe 10 years of hacking in
>> Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
>> will peek at the caller's syntax tree to automagically label the axes
>> as "x" and "log(y)", and that can't be done in Python. But that's not
>> exactly a convincing argument for a macro system.
>>
>> But generating optimized code is Cython's whole selling point, and
>> people really are doing klugey tricks with string-based preprocessors
>> just to generate multiple copies of loops in Cython and C.
>>
>> Also, fused types are great, but: (1) IIUC you can't actually do
>> ndarray[fused_type] yet, which speaks to the feature's complexity, and
>
>
> What? Yes you can do that.
I haven't been able to get ndarray[fused_t] to work as we've discussed
off-list. In your own words "Unfortunately, the automatic buffer
dispatch didn't make it into 0.16, so you need to manually
specialize". I'm a bit hamstrung by other users needing to be able to
compile pandas using the latest released Cython.
>>>
Re: [Cython] Wacky idea: proper macros
On Mon, Apr 30, 2012 at 4:55 PM, Nathaniel Smith wrote: > On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn > wrote: >> JIT is really the way to go. It is one thing that a JIT could optimize the >> case where you pass a callback to a function and inline it run-time. But >> even if it doesn't get that fancy, it'd be great to just be able to write >> something like "cython.eval(s)" and have that be compiled (I guess you could >> do that now, but the sheer overhead of the C compiler and all the .so files >> involved means nobody would sanely use that as the main way of stringing >> together something like pandas). > > The overhead of running a fully optimizing compiler over pandas on > every import is pretty high, though. You can come up with various > caching mechanisms, but they all mean introducing some kind of compile > time/run time distinction. So I'm skeptical we'll just be able to get > rid of that concept, even in a brave new LLVM/PyPy/Julia world. > > -- Nathaniel > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel I'd be perfectly OK with just having to compile pandas's "data engine" and generate loads of C/C++ code. JIT-compiling little array expressions would be cool too. I've got enough of an itch that I might have to start scratching pretty soon. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
Wes McKinney wrote: >On Mon, Apr 30, 2012 at 4:55 PM, Nathaniel Smith wrote: >> On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn >> wrote: >>> JIT is really the way to go. It is one thing that a JIT could >optimize the >>> case where you pass a callback to a function and inline it run-time. >But >>> even if it doesn't get that fancy, it'd be great to just be able to >write >>> something like "cython.eval(s)" and have that be compiled (I guess >you could >>> do that now, but the sheer overhead of the C compiler and all the >.so files >>> involved means nobody would sanely use that as the main way of >stringing >>> together something like pandas). >> >> The overhead of running a fully optimizing compiler over pandas on >> every import is pretty high, though. You can come up with various >> caching mechanisms, but they all mean introducing some kind of >compile >> time/run time distinction. So I'm skeptical we'll just be able to get >> rid of that concept, even in a brave new LLVM/PyPy/Julia world. >> >> -- Nathaniel >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > >I'd be perfectly OK with just having to compile pandas's "data engine" >and generate loads of C/C++ code. JIT-compiling little array >expressions would be cool too. I've got enough of an itch that I might >have to start scratching pretty soon. I think a good start is: Myself I'd look into just using Jinja2 to generate all the Cython code, rather than those horrible Python interpolated strings...that should give you something that's at least rather pleasant for you to work with once you are used to it (even if it is a bit horrible to newcomers to the code base). You can even check in the generated sources. And we've discussed letting cython be smart with templating languages and error report on a line in the original template, such features will certainly accepted once somebody codes it up. (I can give you me breakdown of how I eliminate other templating languages than Jinja2 for this purpose tomorrow if you are interested). Dag >___ >cython-devel mailing list >cython-devel@python.org >http://mail.python.org/mailman/listinfo/cython-devel -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
On Mon, Apr 30, 2012 at 2:32 PM, Dag Sverre Seljebotn wrote: > > > Wes McKinney wrote: > >>On Mon, Apr 30, 2012 at 4:55 PM, Nathaniel Smith wrote: >>> On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn >>> wrote: JIT is really the way to go. It is one thing that a JIT could >>optimize the case where you pass a callback to a function and inline it run-time. >>But even if it doesn't get that fancy, it'd be great to just be able to >>write something like "cython.eval(s)" and have that be compiled (I guess >>you could do that now, but the sheer overhead of the C compiler and all the >>.so files involved means nobody would sanely use that as the main way of >>stringing together something like pandas). >>> >>> The overhead of running a fully optimizing compiler over pandas on >>> every import is pretty high, though. You can come up with various >>> caching mechanisms, but they all mean introducing some kind of >>compile >>> time/run time distinction. So I'm skeptical we'll just be able to get >>> rid of that concept, even in a brave new LLVM/PyPy/Julia world. >>> >>> -- Nathaniel >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> >>I'd be perfectly OK with just having to compile pandas's "data engine" >>and generate loads of C/C++ code. JIT-compiling little array >>expressions would be cool too. I've got enough of an itch that I might >>have to start scratching pretty soon. > > I think a good start is: > > Myself I'd look into just using Jinja2 to generate all the Cython code, > rather than those horrible Python interpolated strings...that should give you > something that's at least rather pleasant for you to work with once you are > used to it (even if it is a bit horrible to newcomers to the code base). > > You can even check in the generated sources. > > And we've discussed letting cython be smart with templating languages and > error report on a line in the original template, such features will certainly > accepted once somebody codes it up. > > (I can give you me breakdown of how I eliminate other templating languages > than Jinja2 for this purpose tomorrow if you are interested). Can you point us to a good example of you using jinja2 for this purpose? I'm a big fan of Jinja2 in general (e.g., for HTML)... > > Dag > >>___ >>cython-devel mailing list >>cython-devel@python.org >>http://mail.python.org/mailman/listinfo/cython-devel > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel -- William Stein Professor of Mathematics University of Washington http://wstein.org ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Wacky idea: proper macros
On Mon, Apr 30, 2012 at 5:36 PM, William Stein wrote: > On Mon, Apr 30, 2012 at 2:32 PM, Dag Sverre Seljebotn > wrote: >> >> >> Wes McKinney wrote: >> >>>On Mon, Apr 30, 2012 at 4:55 PM, Nathaniel Smith wrote: On Mon, Apr 30, 2012 at 9:49 PM, Dag Sverre Seljebotn wrote: > JIT is really the way to go. It is one thing that a JIT could >>>optimize the > case where you pass a callback to a function and inline it run-time. >>>But > even if it doesn't get that fancy, it'd be great to just be able to >>>write > something like "cython.eval(s)" and have that be compiled (I guess >>>you could > do that now, but the sheer overhead of the C compiler and all the >>>.so files > involved means nobody would sanely use that as the main way of >>>stringing > together something like pandas). The overhead of running a fully optimizing compiler over pandas on every import is pretty high, though. You can come up with various caching mechanisms, but they all mean introducing some kind of >>>compile time/run time distinction. So I'm skeptical we'll just be able to get rid of that concept, even in a brave new LLVM/PyPy/Julia world. -- Nathaniel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel >>> >>>I'd be perfectly OK with just having to compile pandas's "data engine" >>>and generate loads of C/C++ code. JIT-compiling little array >>>expressions would be cool too. I've got enough of an itch that I might >>>have to start scratching pretty soon. >> >> I think a good start is: >> >> Myself I'd look into just using Jinja2 to generate all the Cython code, >> rather than those horrible Python interpolated strings...that should give >> you something that's at least rather pleasant for you to work with once you >> are used to it (even if it is a bit horrible to newcomers to the code base). >> >> You can even check in the generated sources. >> >> And we've discussed letting cython be smart with templating languages and >> error report on a line in the original template, such features will >> certainly accepted once somebody codes it up. >> >> (I can give you me breakdown of how I eliminate other templating languages >> than Jinja2 for this purpose tomorrow if you are interested). > > Can you point us to a good example of you using jinja2 for this purpose? > > I'm a big fan of Jinja2 in general (e.g., for HTML)... > >> >> Dag >> >>>___ >>>cython-devel mailing list >>>cython-devel@python.org >>>http://mail.python.org/mailman/listinfo/cython-devel >> >> -- >> Sent from my Android phone with K-9 Mail. Please excuse my brevity. >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > > -- > William Stein > Professor of Mathematics > University of Washington > http://wstein.org > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel I agree, it'd be cool to see an example or two. I have some ideas for a mini DSL / code-generation framework that might suit my needs; jinja2 might be then the right tool for doing the templating / codegen. If I could cut the amount of Cython code I have in half (and make it easier to write simple functions, which are currently more than 50% boilerplate) that would be a big win for me. - Wes ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
