Re: [Cython] Utility Codes and templates
2011/7/25 Stefan Behnel : > Vitja Makarov, 25.07.2011 08:41: >> >> 2011/7/23 Robert Bradshaw: >>> >>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>> wrote: For my work on the _memview branch (and also on fused types) I noticed that UtilityCodes started weighing heavily on me in their current form, so I wrote a little loader in the _memview branch: https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 The idea is simple: you put your utility codes in Cython/Utility in .pyx, .c, .h files etc, and then load them. It works for both prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> This sounds like it could be a nice way to organize our UtilityCode >>> snippets. So far we haven't really needed any more templating than >>> simple substitution, but for what you're doing I can see this being >>> quite handy. This may also provide a more flexible way forward for >>> supporting multiple backends. >>> myutility.c // UtilityProto: MyUtility header code here // UtilityCode: MyUtility implementation code here You can add as many other utilities as you like to the same file. You can then load it using UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>> >>> I agree with you that having multiple related, named snippets in same >>> file is worthwhile. What about >>> >>> // MyUtility.proto /// >>> >>> and >>> >>> MyCyUtility ## >>> >>> so the chunks are easy to see. >>> >> >> C++ comments looks ugly. May be it's better to have something like this: >> >> /* UtilityCode: MyUtility.proto */ >> and >> >> # UtilityCode: MyCyUtility >> >> That's also pretty easy to parse > > For the parser, it makes no difference. For a human, it does. A big fat > marker like "/" is hard to miss, whereas a tiny one like > "/* ... */" is easily overlooked within a longer piece of code. > Ok, but how many slashes should be there? I'd prefer something like: ## UtilityCode: MyUtility For both C and Cython -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
Vitja Makarov, 25.07.2011 10:25: 2011/7/25 Stefan Behnel: Vitja Makarov, 25.07.2011 08:41: 2011/7/23 Robert Bradshaw: On Fri, Jul 22, 2011 at 3:12 AM, mark florisson wrote: For my work on the _memview branch (and also on fused types) I noticed that UtilityCodes started weighing heavily on me in their current form, so I wrote a little loader in the _memview branch: https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 The idea is simple: you put your utility codes in Cython/Utility in .pyx, .c, .h files etc, and then load them. It works for both prototypes and implementations, for UtilityCode and CythonUtilityCode: This sounds like it could be a nice way to organize our UtilityCode snippets. So far we haven't really needed any more templating than simple substitution, but for what you're doing I can see this being quite handy. This may also provide a more flexible way forward for supporting multiple backends. myutility.c // UtilityProto: MyUtility header code here // UtilityCode: MyUtility implementation code here You can add as many other utilities as you like to the same file. You can then load it using UtilityCode.load_utility_from_file("myutility.c", "MyUtility") I agree with you that having multiple related, named snippets in same file is worthwhile. What about // MyUtility.proto /// and MyCyUtility ## so the chunks are easy to see. C++ comments looks ugly. May be it's better to have something like this: /* UtilityCode: MyUtility.proto */ and # UtilityCode: MyCyUtility That's also pretty easy to parse For the parser, it makes no difference. For a human, it does. A big fat marker like "/" is hard to miss, whereas a tiny one like "/* ... */" is easily overlooked within a longer piece of code. Ok, but how many slashes should be there? I'd prefer something like: ## UtilityCode: MyUtility For both C and Cython I don't think the parser should care, and I don't think we should put much thought into this kind of detail, either. I'd suggest making the separator lines at most 78 characters long, and putting as many slashes or hashes in there as are needed to fill up the lines after writing the text content. What's important to the parser is that there's more than one hash/slash at start and end, and something parsable in between. The rest is up to those who write the file. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On 25 July 2011 12:00, Stefan Behnel wrote: > Vitja Makarov, 25.07.2011 10:25: >> >> 2011/7/25 Stefan Behnel: >>> >>> Vitja Makarov, 25.07.2011 08:41: 2011/7/23 Robert Bradshaw: > > On Fri, Jul 22, 2011 at 3:12 AM, mark florisson > wrote: >> >> For my work on the _memview branch (and also on fused types) I noticed >> that UtilityCodes started weighing heavily on me in their current >> form, so I wrote a little loader in the _memview branch: >> >> >> >> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >> >> The idea is simple: you put your utility codes in Cython/Utility in >> .pyx, .c, .h files etc, and then load them. It works for both >> prototypes and implementations, for UtilityCode and CythonUtilityCode: > > This sounds like it could be a nice way to organize our UtilityCode > snippets. So far we haven't really needed any more templating than > simple substitution, but for what you're doing I can see this being > quite handy. This may also provide a more flexible way forward for > supporting multiple backends. > >> myutility.c >> >> // UtilityProto: MyUtility >> header code here >> >> // UtilityCode: MyUtility >> implementation code here >> >> You can add as many other utilities as you like to the same file. You >> can then load it using >> >> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > I agree with you that having multiple related, named snippets in same > file is worthwhile. What about > > // MyUtility.proto /// > > and > > MyCyUtility ## > > so the chunks are easy to see. > C++ comments looks ugly. May be it's better to have something like this: /* UtilityCode: MyUtility.proto */ and # UtilityCode: MyCyUtility That's also pretty easy to parse >>> >>> For the parser, it makes no difference. For a human, it does. A big fat >>> marker like "/" is hard to miss, whereas a tiny one >>> like >>> "/* ... */" is easily overlooked within a longer piece of code. >>> >> >> Ok, but how many slashes should be there? >> >> I'd prefer something like: >> >> ## UtilityCode: MyUtility >> >> For both C and Cython > > I don't think the parser should care, and I don't think we should put much > thought into this kind of detail, either. I'd suggest making the separator > lines at most 78 characters long, and putting as many slashes or hashes in > there as are needed to fill up the lines after writing the text content. > What's important to the parser is that there's more than one hash/slash at > start and end, and something parsable in between. The rest is up to those > who write the file. Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either side (symmetry is not enforced). It's now 'MyUtility' and 'MyUtility.proto'. If there's no objection to the ini-style header (with requirements and other metadata possibly), then I'll implement that one of these days. > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Transformation of pxds
Hi I can now compile pxd files, but I have encountered something rather strange : AnalyseExpressionsTransform turns : cimport foo def test(): foo.printf() into : cimport foo def test(): printf() It is due to the method analyse_as_cimported_attribute of AttributeNode in the file Cython/Compiler/ExprNodes.py Is this useful (since I don't think it has anything to do with expression analysis) and do you have an idea on how I can solve this (my first idea is : extend the class to disable the effect of this method but it's more a hack than anything else) ? Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On Mon, Jul 25, 2011 at 3:07 AM, mark florisson wrote: > On 25 July 2011 12:00, Stefan Behnel wrote: >> Vitja Makarov, 25.07.2011 10:25: >>> >>> 2011/7/25 Stefan Behnel: Vitja Makarov, 25.07.2011 08:41: > > 2011/7/23 Robert Bradshaw: >> >> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >> wrote: >>> >>> For my work on the _memview branch (and also on fused types) I noticed >>> that UtilityCodes started weighing heavily on me in their current >>> form, so I wrote a little loader in the _memview branch: >>> >>> >>> >>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>> >>> The idea is simple: you put your utility codes in Cython/Utility in >>> .pyx, .c, .h files etc, and then load them. It works for both >>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >> >> This sounds like it could be a nice way to organize our UtilityCode >> snippets. So far we haven't really needed any more templating than >> simple substitution, but for what you're doing I can see this being >> quite handy. This may also provide a more flexible way forward for >> supporting multiple backends. >> >>> myutility.c >>> >>> // UtilityProto: MyUtility >>> header code here >>> >>> // UtilityCode: MyUtility >>> implementation code here >>> >>> You can add as many other utilities as you like to the same file. You >>> can then load it using >>> >>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >> >> I agree with you that having multiple related, named snippets in same >> file is worthwhile. What about >> >> // MyUtility.proto /// >> >> and >> >> MyCyUtility ## >> >> so the chunks are easy to see. >> > > C++ comments looks ugly. May be it's better to have something like this: > > /* UtilityCode: MyUtility.proto */ > and > > # UtilityCode: MyCyUtility > > That's also pretty easy to parse For the parser, it makes no difference. For a human, it does. A big fat marker like "/" is hard to miss, whereas a tiny one like "/* ... */" is easily overlooked within a longer piece of code. >>> >>> Ok, but how many slashes should be there? >>> >>> I'd prefer something like: >>> >>> ## UtilityCode: MyUtility >>> >>> For both C and Cython >> >> I don't think the parser should care, and I don't think we should put much >> thought into this kind of detail, either. I'd suggest making the separator >> lines at most 78 characters long, and putting as many slashes or hashes in >> there as are needed to fill up the lines after writing the text content. >> What's important to the parser is that there's more than one hash/slash at >> start and end, and something parsable in between. The rest is up to those >> who write the file. > > Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either > side (symmetry is not enforced). I don't even think we need a max. > It's now 'MyUtility' and > 'MyUtility.proto'. If there's no objection to the ini-style header > (with requirements and other metadata possibly), then I'll implement > that one of these days. One drawback with the ini style is that it detaches the metadata from the code block itself (and require duplicating their names). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
2011/7/26 Robert Bradshaw : > On Mon, Jul 25, 2011 at 3:07 AM, mark florisson > wrote: >> On 25 July 2011 12:00, Stefan Behnel wrote: >>> Vitja Makarov, 25.07.2011 10:25: 2011/7/25 Stefan Behnel: > > Vitja Makarov, 25.07.2011 08:41: >> >> 2011/7/23 Robert Bradshaw: >>> >>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>> wrote: For my work on the _memview branch (and also on fused types) I noticed that UtilityCodes started weighing heavily on me in their current form, so I wrote a little loader in the _memview branch: https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 The idea is simple: you put your utility codes in Cython/Utility in .pyx, .c, .h files etc, and then load them. It works for both prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> This sounds like it could be a nice way to organize our UtilityCode >>> snippets. So far we haven't really needed any more templating than >>> simple substitution, but for what you're doing I can see this being >>> quite handy. This may also provide a more flexible way forward for >>> supporting multiple backends. >>> myutility.c // UtilityProto: MyUtility header code here // UtilityCode: MyUtility implementation code here You can add as many other utilities as you like to the same file. You can then load it using UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>> >>> I agree with you that having multiple related, named snippets in same >>> file is worthwhile. What about >>> >>> // MyUtility.proto /// >>> >>> and >>> >>> MyCyUtility ## >>> >>> so the chunks are easy to see. >>> >> >> C++ comments looks ugly. May be it's better to have something like this: >> >> /* UtilityCode: MyUtility.proto */ >> and >> >> # UtilityCode: MyCyUtility >> >> That's also pretty easy to parse > > For the parser, it makes no difference. For a human, it does. A big fat > marker like "/" is hard to miss, whereas a tiny one > like > "/* ... */" is easily overlooked within a longer piece of code. > Ok, but how many slashes should be there? I'd prefer something like: ## UtilityCode: MyUtility For both C and Cython >>> >>> I don't think the parser should care, and I don't think we should put much >>> thought into this kind of detail, either. I'd suggest making the separator >>> lines at most 78 characters long, and putting as many slashes or hashes in >>> there as are needed to fill up the lines after writing the text content. >>> What's important to the parser is that there's more than one hash/slash at >>> start and end, and something parsable in between. The rest is up to those >>> who write the file. >> >> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either >> side (symmetry is not enforced). > > I don't even think we need a max. > >> It's now 'MyUtility' and >> 'MyUtility.proto'. If there's no objection to the ini-style header >> (with requirements and other metadata possibly), then I'll implement >> that one of these days. > > One drawback with the ini style is that it detaches the metadata from > the code block itself (and require duplicating their names). > Are we going to move all the existing utility code away from python code? I think that would seriously help with moving shared C-code into cython library (.h and .so). Some things like generator class implementation and cyfunction could be move into .so file. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: > 2011/7/26 Robert Bradshaw : >> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >> wrote: >>> On 25 July 2011 12:00, Stefan Behnel wrote: Vitja Makarov, 25.07.2011 10:25: > > 2011/7/25 Stefan Behnel: >> >> Vitja Makarov, 25.07.2011 08:41: >>> >>> 2011/7/23 Robert Bradshaw: On Fri, Jul 22, 2011 at 3:12 AM, mark florisson wrote: > > For my work on the _memview branch (and also on fused types) I noticed > that UtilityCodes started weighing heavily on me in their current > form, so I wrote a little loader in the _memview branch: > > > > https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 > > The idea is simple: you put your utility codes in Cython/Utility in > .pyx, .c, .h files etc, and then load them. It works for both > prototypes and implementations, for UtilityCode and CythonUtilityCode: This sounds like it could be a nice way to organize our UtilityCode snippets. So far we haven't really needed any more templating than simple substitution, but for what you're doing I can see this being quite handy. This may also provide a more flexible way forward for supporting multiple backends. > myutility.c > > // UtilityProto: MyUtility > header code here > > // UtilityCode: MyUtility > implementation code here > > You can add as many other utilities as you like to the same file. You > can then load it using > > UtilityCode.load_utility_from_file("myutility.c", "MyUtility") I agree with you that having multiple related, named snippets in same file is worthwhile. What about // MyUtility.proto /// and MyCyUtility ## so the chunks are easy to see. >>> >>> C++ comments looks ugly. May be it's better to have something like this: >>> >>> /* UtilityCode: MyUtility.proto */ >>> and >>> >>> # UtilityCode: MyCyUtility >>> >>> That's also pretty easy to parse >> >> For the parser, it makes no difference. For a human, it does. A big fat >> marker like "/" is hard to miss, whereas a tiny one >> like >> "/* ... */" is easily overlooked within a longer piece of code. >> > > Ok, but how many slashes should be there? > > I'd prefer something like: > > ## UtilityCode: MyUtility > > For both C and Cython I don't think the parser should care, and I don't think we should put much thought into this kind of detail, either. I'd suggest making the separator lines at most 78 characters long, and putting as many slashes or hashes in there as are needed to fill up the lines after writing the text content. What's important to the parser is that there's more than one hash/slash at start and end, and something parsable in between. The rest is up to those who write the file. >>> >>> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either >>> side (symmetry is not enforced). >> >> I don't even think we need a max. >> >>> It's now 'MyUtility' and >>> 'MyUtility.proto'. If there's no objection to the ini-style header >>> (with requirements and other metadata possibly), then I'll implement >>> that one of these days. >> >> One drawback with the ini style is that it detaches the metadata from >> the code block itself (and require duplicating their names). >> > > Are we going to move all the existing utility code away from python code? Yes, though I don't see a huge hurry to get rid of the old style. > I think that would seriously help with moving shared C-code into > cython library (.h and .so). > > Some things like generator class implementation and cyfunction could > be move into .so file. Yes, that would be good. Of course we don't want a single .so file for every tiny chunk of utility code, but one needs to be careful lumping them together as well (e.g. things may compile differently depending on the set of headers included in the .pyx file). To do this right I we need to compile things at a higher level than module-by-module with Cython.Build.cythonize or similar. Certainly CyFunction and the generator class are easy and high-benefit targets (compared to, e.g. tuple indexing utility code). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
Robert Bradshaw, 26.07.2011 06:29: On Mon, Jul 25, 2011 at 3:07 AM, mark florisson It's now 'MyUtility' and 'MyUtility.proto'. If there's no objection to the ini-style header (with requirements and other metadata possibly), then I'll implement that one of these days. One drawback with the ini style is that it detaches the metadata from the code block itself (and require duplicating their names). The names are duplicated in the file already, since impl and proto are separated and can be anywhere in the file (e.g., you could put all protos at the top and all implementation parts further down). The advantages of putting in a header is that a) it's immediately visible from the top of the file what it contains b) the dependencies on other files are explicit and clearly visible, again, from the top of the file c) the metadata is easy and quick to parse, as the header is trivially separated from the rest d) only the header needs to be parsed in order to know what can be found in it or what other files are required to compile it e) we don't need to write our own parser, and the overall file format is trivial to parse Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel wrote: > Robert Bradshaw, 26.07.2011 06:29: >> >> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >>> >>> It's now 'MyUtility' and >>> 'MyUtility.proto'. If there's no objection to the ini-style header >>> (with requirements and other metadata possibly), then I'll implement >>> that one of these days. >> >> One drawback with the ini style is that it detaches the metadata from >> the code block itself (and require duplicating their names). > > The names are duplicated in the file already, since impl and proto are > separated and can be anywhere in the file (e.g., you could put all protos at > the top and all implementation parts further down). The advantages of > putting in a header is that > > a) it's immediately visible from the top of the file what it contains > > b) the dependencies on other files are explicit and clearly visible, again, > from the top of the file > > c) the metadata is easy and quick to parse, as the header is trivially > separated from the rest > > d) only the header needs to be parsed in order to know what can be found in > it or what other files are required to compile it > > e) we don't need to write our own parser, and the overall file format is > trivial to parse I prefer keeping the metadata close, but perhaps that's just a matter of opinion. Slurping in the entire file and parsing the block headers is pretty trivial as well, and with pre-fetching (on so many levels) I don't think there's a technical advantage for files so small. For those (likely common) cases where no metadata is required, I'm -1 on having to manually keep a "table of contents." - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
Robert Bradshaw, 26.07.2011 07:00: On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: I think that would seriously help with moving shared C-code into cython library (.h and .so). Some things like generator class implementation and cyfunction could be move into .so file. Yes, that would be good. Well, *if* we find a way to make sure the additional modules get properly distributed. It's not obvious to me how to do that, certainly not in a cross-package way, and not even in a per-package way, as that would still require some kind of interaction with distutils and maybe even changes to the users' setup.py scripts (hopefully in a way that doesn't require them to know what exact additional modules will be generated). Of course we don't want a single .so file for every tiny chunk of utility code, but one needs to be careful lumping them together as well (e.g. things may compile differently depending on the set of headers included in the .pyx file). To do this right I we need to compile things at a higher level than module-by-module with Cython.Build.cythonize or similar. Certainly CyFunction and the generator class are easy and high-benefit targets (compared to, e.g. tuple indexing utility code). Now that we have a way to group utility code in a single file, including automatically resolved dependencies to other files, it should be trivial to take such a file and to compile it into a separate module. Cython modules would then trivially export their public names anyway, and utility code in C files could have an additional Cython wrapper file that would simply depend on the C file and that properly exports the C functions through a Python API and inter-module C-API. We could even write empty utility files that only contain dependencies, thus grouping together multiple utility files to one larger module. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On Mon, Jul 25, 2011 at 11:50 PM, Stefan Behnel wrote: > Robert Bradshaw, 26.07.2011 07:00: >> >> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: >>> >>> I think that would seriously help with moving shared C-code into >>> cython library (.h and .so). >>> >>> Some things like generator class implementation and cyfunction could >>> be move into .so file. >> >> Yes, that would be good. > > Well, *if* we find a way to make sure the additional modules get properly > distributed. It's not obvious to me how to do that, certainly not in a > cross-package way, and not even in a per-package way, as that would still > require some kind of interaction with distutils and maybe even changes to > the users' setup.py scripts (hopefully in a way that doesn't require them to > know what exact additional modules will be generated). This is where the ext_modules = Cython.Build.cythonize(...) pattern could come in handy. We'd just add an extra module to the list. (The generated .c file could also be shipped, but might have to be listed explicitly.) >> Of course we don't want a single .so file for >> every tiny chunk of utility code, but one needs to be careful lumping >> them together as well (e.g. things may compile differently depending >> on the set of headers included in the .pyx file). To do this right I >> we need to compile things at a higher level than module-by-module with >> Cython.Build.cythonize or similar. >> >> Certainly CyFunction and the generator class are easy and high-benefit >> targets (compared to, e.g. tuple indexing utility code). > > Now that we have a way to group utility code in a single file, including > automatically resolved dependencies to other files, it should be trivial to > take such a file and to compile it into a separate module. Cython modules > would then trivially export their public names anyway, and utility code in C > files could have an additional Cython wrapper file that would simply depend > on the C file and that properly exports the C functions through a Python API > and inter-module C-API. > > We could even write empty utility files that only contain dependencies, thus > grouping together multiple utility files to one larger module. I was thinking about stuff like how we handle complex numbers where the pre-processor has a large role to play. Low-level stuff like this probably belongs in a .h file rather than a .so file though. I'm also wondering if there are any efficiency gains to be had from actually linking the resulting .so file at compile time rather than using the runtime inter-module C-API stuff. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel