Re: [Cython] Utility Codes and templates
Robert Bradshaw, 22.07.2011 23:44:
On Fri, Jul 22, 2011 at 1:39 PM, mark florisson
wrote:
On 22 July 2011 22:05, Robert Bradshaw wrote:
On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
wrote:
For my work on the _memview branch (and also on fused types) I noticed
that UtilityCodes started weighing heavily on me in their current
form, so I wrote a little loader in the _memview branch:
https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
The idea is simple: you put your utility codes in Cython/Utility in
.pyx, .c, .h files etc, and then load them. It works for both
prototypes and implementations, for UtilityCode and CythonUtilityCode:
This sounds like it could be a nice way to organize our UtilityCode
snippets. So far we haven't really needed any more templating than
simple substitution, but for what you're doing I can see this being
quite handy. This may also provide a more flexible way forward for
supporting multiple backends.
It's unlikely to be useful for the ctypes backend, but C/C++ will benefit
from it, and so may others.
myutility.c
// UtilityProto: MyUtility
header code here
// UtilityCode: MyUtility
implementation code here
You can add as many other utilities as you like to the same file. You
can then load it using
UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
I agree with you that having multiple related, named snippets in same
file is worthwhile. What about
// MyUtility.proto ///
and
MyCyUtility ##
Hmm, right. Cython utility code doesn't actually need any separate parts. I
hadn't realised that.
+1, the above is well visible and doesn't hurt the file syntax.
Sounds like a good idea. How would we handle the requires list in that
case though? Or should we leave that to the Python code that loads it
for now?
For now, lets leave them with the Python code.
-1, I see no reason to do it the wrong way now only to clean it up later.
Why not just do it right? It's not like that's so much more work. I can
help out.
Eventually we could do
something similar to how we do Cython pragmas, i.e. parse the comments
that occur at the top of the file (chunk) as having special meaning.
Requiring a short declaration at the top of the file makes it easier to
find out what's in that file from a quick look at the top, rather than
having to skip (or grep) through it. Yes, I think it makes sense to put
this into some kind of file header.
Coming back to my ini file proposal, what about a file syntax like this:
/*
[MyUtil1]
requires = file1:abc
[MyUtil2]
; nothing to declare ... (I think it's ; for comments - don't care)
*/
/ MyUtil1.proto /
...
and the same for Cython code:
"""
[MyUtil1]
requires = file1:abc
[MyUtil2]
requires = MyUtil1
"""
### MyUtil1 #
...
The initial section is trivial to extract and to drop into RawConfigParser,
and we are completely free to extend the metadata section later on, without
having to touch the code as well.
In terms of delimiters, I prefer some form of strings over comments
(often a value is expected, and Python's comments always extend to the
line break)
Yes, it's clearly a problem in Cython code. It would work very nicely for C
code, though. I think we should use it there right from the start.
but sticking with the {{ }} syntax appeals to me even
more. It's pretty standard, and decent syntax hilighters shouldn't
choke on it (well, the surrounding code) too bad.
Ok, that's fine with me. It would be easily changeable if we revert
later anyways.
Yep, which makes it a much easier decision than trying to figure out a
templating solution for the user-visible front-end.
I'm fine with {{ }} for Cython code. It's just as good as anything else.
I'll add tempita to Cython.
Sounds good. (If anyone else has objections, bring them up now...)
Tempita is fine with me.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On 23 July 2011 09:37, Stefan Behnel wrote:
> Robert Bradshaw, 22.07.2011 23:44:
>>
>> On Fri, Jul 22, 2011 at 1:39 PM, mark florisson
>> wrote:
>>>
>>> On 22 July 2011 22:05, Robert Bradshaw
>>> wrote:
On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
wrote:
>
> For my work on the _memview branch (and also on fused types) I noticed
> that UtilityCodes started weighing heavily on me in their current
> form, so I wrote a little loader in the _memview branch:
>
>
> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>
> The idea is simple: you put your utility codes in Cython/Utility in
> .pyx, .c, .h files etc, and then load them. It works for both
> prototypes and implementations, for UtilityCode and CythonUtilityCode:
This sounds like it could be a nice way to organize our UtilityCode
snippets. So far we haven't really needed any more templating than
simple substitution, but for what you're doing I can see this being
quite handy. This may also provide a more flexible way forward for
supporting multiple backends.
>
> It's unlikely to be useful for the ctypes backend, but C/C++ will benefit
> from it, and so may others.
>
>
> myutility.c
>
> // UtilityProto: MyUtility
> header code here
>
> // UtilityCode: MyUtility
> implementation code here
>
> You can add as many other utilities as you like to the same file. You
> can then load it using
>
> UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
I agree with you that having multiple related, named snippets in same
file is worthwhile. What about
// MyUtility.proto ///
and
MyCyUtility ##
>
> Hmm, right. Cython utility code doesn't actually need any separate parts. I
> hadn't realised that.
>
> +1, the above is well visible and doesn't hurt the file syntax.
>
>
>>> Sounds like a good idea. How would we handle the requires list in that
>>> case though? Or should we leave that to the Python code that loads it
>>> for now?
>>
>> For now, lets leave them with the Python code.
>
> -1, I see no reason to do it the wrong way now only to clean it up later.
> Why not just do it right? It's not like that's so much more work. I can help
> out.
>
>
>> Eventually we could do
>> something similar to how we do Cython pragmas, i.e. parse the comments
>> that occur at the top of the file (chunk) as having special meaning.
>
> Requiring a short declaration at the top of the file makes it easier to find
> out what's in that file from a quick look at the top, rather than having to
> skip (or grep) through it. Yes, I think it makes sense to put this into some
> kind of file header.
>
> Coming back to my ini file proposal, what about a file syntax like this:
>
>
> /*
> [MyUtil1]
> requires = file1:abc
>
> [MyUtil2]
> ; nothing to declare ... (I think it's ; for comments - don't care)
> */
>
> / MyUtil1.proto /
> ...
>
>
> and the same for Cython code:
>
> """
> [MyUtil1]
> requires = file1:abc
>
> [MyUtil2]
> requires = MyUtil1
> """
>
> ### MyUtil1 #
> ...
>
> The initial section is trivial to extract and to drop into RawConfigParser,
> and we are completely free to extend the metadata section later on, without
> having to touch the code as well.
>
Looks good to me. Up to now you could also use comments at the top but
they were just dropped.
In terms of delimiters, I prefer some form of strings over comments
(often a value is expected, and Python's comments always extend to the
line break)
>
> Yes, it's clearly a problem in Cython code. It would work very nicely for C
> code, though. I think we should use it there right from the start.
>
I concur that the {{ }} can show up like an ugly red in some
situations. I do think comments are nicer. However for Cython
templates you'd have to use triple-quoted strings in order to get a
benefit (I assume you occasionally want something that stretches
multiple lines).
but sticking with the {{ }} syntax appeals to me even
more. It's pretty standard, and decent syntax hilighters shouldn't
choke on it (well, the surrounding code) too bad.
>>>
>>> Ok, that's fine with me. It would be easily changeable if we revert
>>> later anyways.
>>
>> Yep, which makes it a much easier decision than trying to figure out a
>> templating solution for the user-visible front-end.
>
> I'm fine with {{ }} for Cython code. It's just as good as anything else.
>
>
>>> I'll add tempita to Cython.
>>
>> Sounds good. (If anyone else has objections, bring them up now...)
>
> Tempita is fine with me.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-d
Re: [Cython] Transformation of pxds
Hi Romain, CC-ing cython-devel here, as this is not an entirely trivial question to answer. Romain Guillebert, 23.07.2011 06:52: During this week, I've been trying to find a way to implement cimport, I think that the cleanest way of implementing this is to compile pxds just like pyxs are compiled. You are aware that there are two types of .pxd files? One next to a .pyx or .py file with the same name, which provides additional declarations for that source file and can also serve as a cimportable .pxd for other modules, thus providing a C-level API between modules, and a second type that only contains external declarations (well, and potentially inline functions etc.), usually for external C libraries or header files, but that does not correspond to a .pyx/.py source file. The idea of materialising the external .pxd files does sound appealing. I imagine that this means that the resulting .py file would basically just set up ctypes wrappers for external declarations and store them under the corresponding name, right? Then a cimport from an external .pxd file would turn into a regular import. Plus, you'd end up with a single place where the external library (if any, or potentially many libraries) is loaded and managed. Cool. Even the first kind of .pxd file would fit into this quite nicely. Here, the corresponding .pyx/.py file would be enough to represent the module, and the exported declarations in the .pxd file would just be ignored. Consequently, the .pxd file would only be used to provide declarations for the module itself and serve no other purpose. Thus, in both cases, there would simply be an importable .py file for each cimported .pxd file, be it user provided or generated. I'm having trouble knowing where to implement that though. I think that I will implement that in the method process_pxd of Context in Cython/Compiler/Main.py (and it will probably call compile_single) but I don't know if it's a good design. What do you think ? I think it could be done on the fly, when cimporting the .pxd file. Or maybe it's better to just queue up the cimported files that need to get compiled, and then compile them after compiling the main source file. For the client side of the cimport mechanism, parsing the .pxd file is enough to figure out what it will eventually export, so there is no actual need to generate a .py file for it at the same time. I think the compilation will be a separate operation anyway, because the .pxd file is currently parsed into the context of the .pyx file being compiled. Hacking an additional compilation onto that step may not be trivial. I don't mind either way, i.e. if it's done while cimporting the .pxd or afterwards. In any case, this means that compiling a .pyx file can potentially generate multiple .py files, which means that there should be a way to reuse already generated files from other compiler runs (i.e. if you compile multiple .pyx files which cimport the same .pxd files, then each .pxd file only needs to get compiled once). Basically, this means that the compiler would check if there is already a .py file for a .pxd, and not generate it in that case. However, this is problematic, because we do not necessarily know if the .py file was user provided or generated... This makes me think that the ctypes backend may generally want to generate its output files into a separate target *directory*, rather than just providing a target file name. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
