Re: [Cython] [ENH] Compile-time code execution (macros / templates)
1. I'm not sure why there would be any dependency on a C compiler in either mine or Celelibi's proposal. Could you please elaborate? At least in my proposal, we only ever go from Cython AST to Cython AST and use Python to do it. None of these things require a C compiler. To me it looked like Celelibi's proposal was basically asking for "some subset" of regular Cython code to be evaluated at Cythonization time. Since evaluation of Cython code requires compiling the C code first, it seems to me it'd need a C compiler. It depends on the subset of course. I agree that your suggestion of modifying the AST can/should be done in Python. 2. Again I can't speak for Celelibi but my proposal would be compile-time only. I'm happy to add restrictions (decorators on cdef functions only or maybe even special syntax) to ensure that this is the case. The D-language which Celelibi linked as an example claims "All functions that execute in CTFE must also be executable at run time. The compile time evaluation of a function does the equivalent of running the function at run time" and Celelibi didn't seem to be proposing any special syntax. So on that basis I don't see where any compile-time only restrictions can come from. It's quite possible that I've missed the point of the proposal in both cases though. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
1) My suggestion would only run some standard python code during the compilation. Running the cdef functions at compile-time doesn't have to be supported. At least not for a first version. Even if it gets supported, calling external functions (written in C) doesn't have to be supported either. If only running the cdef functions is supported, then it means cython has the full code and would "only" have to implement the C semantics in python. And actually, not the whole C semantics has to be supported. The language D has quite some restrictions on the code that can run at compile-time. Especially when it comes to pointers. But first, running only python code seems quite doable. 2) That's something I didn't elaborate on in my initial message to keep it short (kinda). But indeed, some compile-time functions just wouldn't make sens at runtime. Especially, `twice` and `make_ptr_class` wouldn't as far as I can tell, because returning a cdef function or class at run time doesn't really make sens right now. (Maybe in the future if cython support compiling closures to C?) I guess some kind annotations would be necessary to mark a function (or maybe even a class?) as "compile-time only". Of course, other functions and classes might make sens to be used at compile-time and at run-time. The only thing I'm not sure about, is how to determine what should get evaluated at compile-time. A few places in the code should probably get automatically arked for CTFE, like decorators on cdef functions. IIRC, D uses "enum" to declare a compile-time constant. But this only solves half the issue. What about top-level code evaluation, like the `if os =="windows"` example? Maybe this could be wrapped in a function evaluated at compile-time. And thuse only function evaluation would be requred. Something like this: @cython.ctfe_only def choice_thread_something(os): if os == "windows": cdef thread_something(): # Windows behavior else: cdef thread_something(): # POSIX behavior return thread_something ctfe thread_something = choice_thread_something(os) Where the "ctfe" keyword would play a similar role to "enum" in D and force the evaluation of the right hand side during the compilation. As a technical detail, the "cdef" lines that define the thread_something functions could be remplaced during the compilation by some python code like: thread_something = CdefFunction(...) And the calling line `ctfe thread_something = ...` would just insert the returned object into the AST. All the remaining (non-CTFE) code would be converted to code that build the AST. I hope that makes it more clear. Celelibi Le Wed, Feb 24, 2021 at 10:20:39PM +, da-woods a écrit : > A few things to point out: > 1) Because Cython is mainly a is code translation tool it doesn't currently > have an overview of how things would eventually be evaluated - it leaves a > lot for the C compiler to sort out. Your suggestion would presumably add a > dependency on a C compiler at the .pyx -> .c stage (currently the only > dependencies are Python+Cython itself). > 2) Presumably `twice` and `make_ptr_class` would also have to work at > run-time? They are regular `def` functions after all? That seems like a big > change to now have to dynamically create `cdef` functions and classes at > both Cythonization-time and runtime. And also a Python-runtime > representation of every C type (so that you can call `make_ptr_class` at > runtime) > > I totally get the appeal of being able to wrap template-types quickly and > easily. However, to me this idea actually seems a lot harder than something > which creates new syntax, because you don't distinguish these special > functions and so they have to work universally as Python functions too. > > In summary: if it worked it'd be very nice, but I personally don't have any > idea how you'd implement it within how Cython current works. > > David > > > > On 24/02/2021 14:29, Celelibi wrote: > > Le Tue, Feb 23, 2021 at 11:24:42AM -0500, Prakhar Goel a écrit : > > > I had similar ideas in the past (assuming I understand your > > > proposal). I'd > > > love it if a decorator could link to some python function. This python > > > function would get something that represents the cdef function (with > > > the > > > ast). It would then return a transformed ast that could be spliced > > > into > > > the original program. > > As I said, I'm not sure allowing AST manipulation would be that useful. > > I mean, it's rarely ever needed in python. Why would it be needed with > > cython? > > However, it would surely make the code much less readable, since the > > code that's executed is not the code that's written. > > > > The use-cases I have in mind are more like these. Which are common > > patterns in python. > > > > Compile-time implementation choice: > > > > if os == "Windows": > > cdef thread_somethin
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
On 26/02/21 9:29 am, Celelibi wrote: Maybe in the future if cython support compiling closures to C? Your "twice" example already needs some closure functionality. It relies on being able to manufacture a cdef function inside a Python function, with the expectation that it will have access to arguments of the Python function. With all or some of that happening at compile time rather than run time. I'm having trouble imagining how it would be implemented. -- Greg ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
Le Fri, Feb 26, 2021 at 01:32:34PM +1300, Greg Ewing a écrit :
> On 26/02/21 9:29 am, Celelibi wrote:
> > Maybe in the future if cython
> > support compiling closures to C?
>
> Your "twice" example already needs some closure functionality.
Yes, at the python level only. That's precisely the point.
> It relies on being able to manufacture a cdef function inside
> a Python function, with the expectation that it will have
> access to arguments of the Python function. With all or some
> of that happening at compile time rather than run time. I'm
> having trouble imagining how it would be implemented.
It relies on creating a python object representing the inner cdef
function during the compile-time execution. Which is, like, the trivial
part.
Just picture the `twice` example being converted like this for the
compile-time execution.
def twice(f):
fun = CdefFunction(... use f ...)
return fun
foo = CdefFunction(...)
ast.append(AstCdefFunction("foo", twice(foo)))
Where CdefFunction is not necessarily callable. It's just a class
representing a cdef function. And AstCdefFunction is an AST node
representing a cdef function definition.
Does that seem far fetched to you?
I think this might even be doable without having to even detect the
closure in cython. We'd just have to let python perform the name lookup.
Celelibi
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
Celelibi wrote:
> def twice(f):
> fun = CdefFunction(... use f ...)
> return fun
>
> foo = CdefFunction(...)
> ast.append(AstCdefFunction("foo", twice(foo)))
Doesn't this just punt on how CdefFunction works? Feels like we're
back to AST re-writing.
--
Warm Regards
Prakhar Goel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
On 26/02/21 3:21 pm, Celelibi wrote:
def twice(f):
fun = CdefFunction(... use f ...)
return fun
foo = CdefFunction(...)
ast.append(AstCdefFunction("foo", twice(foo)))
I think this might even be doable without having to even detect the
closure in cython. We'd just have to let python perform the name lookup.
The cdef function created inside twice() is going to have
to be some kind of closure, because it needs access at run
time to the particular f that was passed to twice() at
compile time.
--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
