Re: [Cython] Transformation of pxds
Romain Guillebert, 26.07.2011 04:10: I can now compile pxd files, but I have encountered something rather strange : AnalyseExpressionsTransform turns : cimport foo def test(): foo.printf() into : cimport foo def test(): printf() This makes sense from the POV of C, which has a flat namespace. It is true that this doesn't make much sense for the Python backend. Specifically, we do not want to flatten the .pxd namespace here, because the attribute name may already exist in the module namespace. Only the qualified name gives us proper and safe Python code. It is due to the method analyse_as_cimported_attribute of AttributeNode in the file Cython/Compiler/ExprNodes.py Is this useful (since I don't think it has anything to do with expression analysis) and do you have an idea on how I can solve this (my first idea is : extend the class to disable the effect of this method but it's more a hack than anything else) ? Well, the current implementation is already a huge hack, see mutate_into_name_node(). The proper fix we envisioned for cases like this is to globally refactor the analyse_types() methods into methods that return self, or any other node that they deem more appropriate. The caller would then put the returned value in the place of the original node. This approach would allow analyse_as_cimported_attribute() to simply instantiate and return a new NameNode, instead of having to changing the type of itself locally. However, so far, no-one has done anything to get this refactoring done, likely because it's a lot of rather boring work. Regardless of this refactoring, I think it might be best to explicitly implement this method in a backend specific way. When targeting C, it should flatten the attribute access. When targeting Python, it should still determine the entry (and thus the type of the attribute) but should not build a NameNode. I assume the problem here is that the method does not actually know which backend it is targeting, right? The "env" parameter (i.e. the scope) won't know that, and I'm not sure it should know that. My suggestion is to let the module scope of the .pxd know that it's being read into (or as) a flat or prefixed namespace, and to propagate the information from there into the Entry objects it describes. Then the Entry of the attribute would know if it needs a namespace or not, and the analysis method could get its information directly from there. Does that make sense? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On 26 July 2011 08:46, Robert Bradshaw wrote: > On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel wrote: >> Robert Bradshaw, 26.07.2011 06:29: >>> >>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson It's now 'MyUtility' and 'MyUtility.proto'. If there's no objection to the ini-style header (with requirements and other metadata possibly), then I'll implement that one of these days. >>> >>> One drawback with the ini style is that it detaches the metadata from >>> the code block itself (and require duplicating their names). >> >> The names are duplicated in the file already, since impl and proto are >> separated and can be anywhere in the file (e.g., you could put all protos at >> the top and all implementation parts further down). The advantages of >> putting in a header is that >> >> a) it's immediately visible from the top of the file what it contains >> >> b) the dependencies on other files are explicit and clearly visible, again, >> from the top of the file >> >> c) the metadata is easy and quick to parse, as the header is trivially >> separated from the rest >> >> d) only the header needs to be parsed in order to know what can be found in >> it or what other files are required to compile it >> >> e) we don't need to write our own parser, and the overall file format is >> trivial to parse > > I prefer keeping the metadata close, but perhaps that's just a matter > of opinion. Slurping in the entire file and parsing the block headers > is pretty trivial as well, and with pre-fetching (on so many levels) I > don't think there's a technical advantage for files so small. For > those (likely common) cases where no metadata is required, I'm -1 on > having to manually keep a "table of contents." Right, you woud only list metadata if there is something worthwhile to list. Otherwise you just omit it. (Perhaps that was not what Stefan had in mind though, I'm not sure). > - Robert > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On 26 July 2011 08:50, Stefan Behnel wrote: > Robert Bradshaw, 26.07.2011 07:00: >> >> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: >>> >>> I think that would seriously help with moving shared C-code into >>> cython library (.h and .so). >>> >>> Some things like generator class implementation and cyfunction could >>> be move into .so file. >> >> Yes, that would be good. > > Well, *if* we find a way to make sure the additional modules get properly > distributed. It's not obvious to me how to do that, certainly not in a > cross-package way, and not even in a per-package way, as that would still > require some kind of interaction with distutils and maybe even changes to > the users' setup.py scripts (hopefully in a way that doesn't require them to > know what exact additional modules will be generated). > > >> Of course we don't want a single .so file for >> every tiny chunk of utility code, but one needs to be careful lumping >> them together as well (e.g. things may compile differently depending >> on the set of headers included in the .pyx file). To do this right I >> we need to compile things at a higher level than module-by-module with >> Cython.Build.cythonize or similar. >> >> Certainly CyFunction and the generator class are easy and high-benefit >> targets (compared to, e.g. tuple indexing utility code). > > Now that we have a way to group utility code in a single file, including > automatically resolved dependencies to other files, it should be trivial to > take such a file and to compile it into a separate module. Cython modules > would then trivially export their public names anyway, and utility code in C > files could have an additional Cython wrapper file that would simply depend > on the C file and that properly exports the C functions through a Python API > and inter-module C-API. > > We could even write empty utility files that only contain dependencies, thus > grouping together multiple utility files to one larger module. As you said earlier, you'd lose the inline functionality though. But I suppose we could put inline utilities in a header and non-inline utilities in .so files. I'm then not sure how to prevent compiler warnings for unused static inline functions though... > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
mark florisson, 26.07.2011 10:57: On 26 July 2011 08:46, Robert Bradshaw wrote: On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel wrote: Robert Bradshaw, 26.07.2011 06:29: On Mon, Jul 25, 2011 at 3:07 AM, mark florisson It's now 'MyUtility' and 'MyUtility.proto'. If there's no objection to the ini-style header (with requirements and other metadata possibly), then I'll implement that one of these days. One drawback with the ini style is that it detaches the metadata from the code block itself (and require duplicating their names). The names are duplicated in the file already, since impl and proto are separated and can be anywhere in the file (e.g., you could put all protos at the top and all implementation parts further down). The advantages of putting in a header is that a) it's immediately visible from the top of the file what it contains b) the dependencies on other files are explicit and clearly visible, again, from the top of the file c) the metadata is easy and quick to parse, as the header is trivially separated from the rest d) only the header needs to be parsed in order to know what can be found in it or what other files are required to compile it e) we don't need to write our own parser, and the overall file format is trivial to parse I prefer keeping the metadata close, but perhaps that's just a matter of opinion. Slurping in the entire file and parsing the block headers is pretty trivial as well, and with pre-fetching (on so many levels) I don't think there's a technical advantage for files so small. For those (likely common) cases where no metadata is required, I'm -1 on having to manually keep a "table of contents." Right, you woud only list metadata if there is something worthwhile to list. Otherwise you just omit it. (Perhaps that was not what Stefan had in mind though, I'm not sure). I don't care so much here, but I consider the file header an interface description. I.e. you could also include utility code in a file that is only available as dependencies to other utility code, simply by not listing it in the header. Allowing to skip the header makes the semantics a bit more ugly here. But as I said, I wouldn't mind if the header was optional, as long as that's easy enough to handle. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Utility Codes and templates
On 26 July 2011 11:26, Stefan Behnel wrote: > mark florisson, 26.07.2011 10:57: >> >> On 26 July 2011 08:46, Robert Bradshaw >> wrote: >>> >>> On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel >>> wrote: Robert Bradshaw, 26.07.2011 06:29: > > On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >> >> It's now 'MyUtility' and >> 'MyUtility.proto'. If there's no objection to the ini-style header >> (with requirements and other metadata possibly), then I'll implement >> that one of these days. > > One drawback with the ini style is that it detaches the metadata from > the code block itself (and require duplicating their names). The names are duplicated in the file already, since impl and proto are separated and can be anywhere in the file (e.g., you could put all protos at the top and all implementation parts further down). The advantages of putting in a header is that a) it's immediately visible from the top of the file what it contains b) the dependencies on other files are explicit and clearly visible, again, from the top of the file c) the metadata is easy and quick to parse, as the header is trivially separated from the rest d) only the header needs to be parsed in order to know what can be found in it or what other files are required to compile it e) we don't need to write our own parser, and the overall file format is trivial to parse >>> >>> I prefer keeping the metadata close, but perhaps that's just a matter >>> of opinion. Slurping in the entire file and parsing the block headers >>> is pretty trivial as well, and with pre-fetching (on so many levels) I >>> don't think there's a technical advantage for files so small. For >>> those (likely common) cases where no metadata is required, I'm -1 on >>> having to manually keep a "table of contents." >> >> Right, you woud only list metadata if there is something worthwhile to >> list. Otherwise you just omit it. (Perhaps that was not what Stefan >> had in mind though, I'm not sure). > > I don't care so much here, but I consider the file header an interface > description. I.e. you could also include utility code in a file that is only > available as dependencies to other utility code, simply by not listing it in > the header. Allowing to skip the header makes the semantics a bit more ugly > here. > > But as I said, I wouldn't mind if the header was optional, as long as that's > easy enough to handle. Right, in that case I think the solution is quite appealing. > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython 0.15 release candidate
On 25 July 2011 08:08, Arfrever Frehtes Taifersar Arahesis wrote: > There are 4 NumPy-related test errors with Python 3.1 and 3.2. > Output with Python 3.2: > > == > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_bufacc_T155 > -- > Traceback (most recent call last): > File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > extra_postargs) > File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > spawn(cmd, dry_run=self.dry_run) > File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > _spawn_posix(cmd, search_path, dry_run=dry_run) > File "/usr/lib64/python3.2/distutils/spawn.py", line 165, in _spawn_posix > % (cmd[0], exit_status)) > distutils.errors.DistutilsExecError: command 'x86_64-pc-linux-gnu-g++' failed > with exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "runtests.py", line 678, in run > self.runCompileTest() > File "runtests.py", line 490, in runCompileTest > self.test_directory, self.expect_errors, self.annotate) > File "runtests.py", line 665, in compile > self.run_distutils(test_directory, module, workdir, incdir) > File "runtests.py", line 625, in run_distutils > build_extension.run() > File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 347, in run > self.build_extensions() > File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 456, in > build_extensions > self.build_extension(ext) > File "runtests.py", line 257, in build_extension > _build_ext.build_extension(self, ext) > File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 511, in > build_extension > depends=ext.depends) > File "/usr/lib64/python3.2/distutils/ccompiler.py", line 576, in compile > self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 188, in _compile > raise CompileError(msg) > distutils.errors.CompileError: command 'x86_64-pc-linux-gnu-g++' failed with > exit status 1 > > == > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_cimport > -- > Traceback (most recent call last): > File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > extra_postargs) > File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > spawn(cmd, dry_run=self.dry_run) > File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > _spawn_posix(cmd, search_path, dry_run=dry_run) > File "/usr/lib64/python3.2/distutils/spawn.py", line 165, in _spawn_posix > % (cmd[0], exit_status)) > distutils.errors.DistutilsExecError: command 'x86_64-pc-linux-gnu-g++' failed > with exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "runtests.py", line 678, in run > self.runCompileTest() > File "runtests.py", line 490, in runCompileTest > self.test_directory, self.expect_errors, self.annotate) > File "runtests.py", line 665, in compile > self.run_distutils(test_directory, module, workdir, incdir) > File "runtests.py", line 625, in run_distutils > build_extension.run() > File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 347, in run > self.build_extensions() > File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 456, in > build_extensions > self.build_extension(ext) > File "runtests.py", line 257, in build_extension > _build_ext.build_extension(self, ext) > File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 511, in > build_extension > depends=ext.depends) > File "/usr/lib64/python3.2/distutils/ccompiler.py", line 576, in compile > self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 188, in _compile > raise CompileError(msg) > distutils.errors.CompileError: command 'x86_64-pc-linux-gnu-g++' failed with > exit status 1 > > == > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_parallel > -- > Traceback (most recent call last): > File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > extra_postargs) > File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > spawn(cmd, dry_run=self.dry_run) > File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > _spawn_posix(cmd, search_path, dry_run=dry_run) > File "/usr/lib64/python3.2
Re: [Cython] Cython 0.15 release candidate
2011-07-25 08:17:24 Stefan Behnel napisał(a): > Arfrever Frehtes Taifersar Arahesis, 25.07.2011 08:08: > > There are 4 NumPy-related test errors with Python 3.1 and 3.2. > > I assume that's NumPy 1.5? Could you provide the C compiler output from the > test logs? > > This is on 64bit Linux, right? What gcc version? NumPy 1.6.1, 64bit Linux, GCC 4.5.2. -- Arfrever Frehtes Taifersar Arahesis signature.asc Description: This is a digitally signed message part. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
