Re: [Cython] [ENH] Compile-time code execution (macros / templates)

2021-02-24 Thread Celelibi
Le Tue, Feb 23, 2021 at 11:24:42AM -0500, Prakhar Goel a écrit :
>I had similar ideas in the past (assuming I understand your proposal). I'd
>love it if a decorator could link to some python function. This python
>function would get something that represents the cdef function (with the
>ast). It would then return a transformed ast that could be spliced into
>the original program.

As I said, I'm not sure allowing AST manipulation would be that useful.
I mean, it's rarely ever needed in python. Why would it be needed with
cython?
However, it would surely make the code much less readable, since the
code that's executed is not the code that's written.

The use-cases I have in mind are more like these. Which are common
patterns in python.

Compile-time implementation choice:

if os == "Windows":
cdef thread_something():
# Handle Windows-specific behavior
else:
cdef thread_something():
# Assume POSIX behavior


Decorators on cdef functions:
def twice(f):
cdef fun(x):
return f(2*x)
return fun

@twice
cdef foo(x):
# Do something math-y



Creating cdef classes on demand:

def make_ptr_class(T):
cdef class C:
cdef T *ptr
return C

IntPtr = make_ptr_class(int)
FloatPtr = make_ptr_class(float)

My current use-case is actually exactly this one. Creating wrappers
classes for pointer types. Which I currently cannot make both generic
and type safe.


The python code run at compile-time would basically manipulate python
objects representing cdef functions, classes and C types.

Internally, the way it could be implemented is that cython could
generate a python program that would produce the AST. But it doesn't
mean the AST itself has to be visible.

Best regards,
Celelibi
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] [ENH] Compile-time code execution (macros / templates)

2021-02-24 Thread da-woods

A few things to point out:
1) Because Cython is mainly a is code translation tool it doesn't 
currently have an overview of how things would eventually be evaluated - 
it leaves a lot for the C compiler to sort out. Your suggestion would 
presumably add a dependency on a C compiler at the .pyx -> .c stage 
(currently the only dependencies are Python+Cython itself).
2) Presumably `twice` and `make_ptr_class` would also have to work at 
run-time? They are regular `def` functions after all? That seems like a 
big change to now have to dynamically create `cdef` functions and 
classes at both Cythonization-time and runtime. And also a 
Python-runtime representation of every C type (so that you can call 
`make_ptr_class` at runtime)


I totally get the appeal of being able to wrap template-types quickly 
and easily. However, to me this idea actually seems a lot harder than 
something which creates new syntax, because you don't distinguish these 
special functions and so they have to work universally as Python 
functions too.


In summary: if it worked it'd be very nice, but I personally don't have 
any idea how you'd implement it within how Cython current works.


David



On 24/02/2021 14:29, Celelibi wrote:

Le Tue, Feb 23, 2021 at 11:24:42AM -0500, Prakhar Goel a écrit :

I had similar ideas in the past (assuming I understand your proposal). I'd
love it if a decorator could link to some python function. This python
function would get something that represents the cdef function (with the
ast). It would then return a transformed ast that could be spliced into
the original program.

As I said, I'm not sure allowing AST manipulation would be that useful.
I mean, it's rarely ever needed in python. Why would it be needed with
cython?
However, it would surely make the code much less readable, since the
code that's executed is not the code that's written.

The use-cases I have in mind are more like these. Which are common
patterns in python.

Compile-time implementation choice:

if os == "Windows":
cdef thread_something():
# Handle Windows-specific behavior
else:
cdef thread_something():
# Assume POSIX behavior


Decorators on cdef functions:
def twice(f):
cdef fun(x):
return f(2*x)
return fun

@twice
cdef foo(x):
# Do something math-y



Creating cdef classes on demand:

def make_ptr_class(T):
cdef class C:
cdef T *ptr
return C

IntPtr = make_ptr_class(int)
FloatPtr = make_ptr_class(float)

My current use-case is actually exactly this one. Creating wrappers
classes for pointer types. Which I currently cannot make both generic
and type safe.


The python code run at compile-time would basically manipulate python
objects representing cdef functions, classes and C types.

Internally, the way it could be implemented is that cython could
generate a python program that would produce the AST. But it doesn't
mean the AST itself has to be visible.

Best regards,
Celelibi
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel



___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] [ENH] Compile-time code execution (macros / templates)

2021-02-24 Thread Prakhar Goel
Celelibi,

AST modification is a rare use-case but when you need it, you _really_
need it (speaking from actual experience here)! Think something like
ISPC. Cython already deals with the function AST. Your proposals will
implicitly require rewriting the AST. Why not support it explicitly?
It feels like a small change for a big win (admittedly in only a
limited number of cases). As for debugging, that's up to the person
who wrote the AST rewriting magic.

David,

1. I'm not sure why there would be any dependency on a C compiler in
either mine or Celelibi's proposal. Could you please elaborate? At
least in my proposal, we only ever go from Cython AST to Cython AST
and use Python to do it. None of these things require a C compiler.

2. Again I can't speak for Celelibi but my proposal would be
compile-time only. I'm happy to add restrictions (decorators on cdef
functions only or maybe even special syntax) to ensure that this is
the case.

--

Warm Regards
Prakhar Goel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel