Re: [Cython] dynamically compile and import pure python modules
On 22 March 2012 20:07, Stefan Behnel wrote: > mark florisson, 22.03.2012 19:50: >> For the fused type runtime dispatch I found it very convenient to use >> the with statement, but that is not supported in Python 2.4. However, >> the compiler could dynamically compile compiler code with the compiler >> itself and import it (pyximport), if it is not needed to compile >> Cython itself. I gave it a try and it seems to work like a charm (but >> probably needs more testing :): >> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2 > > The advantages are limited, so I'm leaning towards seeing the drawbacks, of > which there are at least some. For one, *running* Cython (as opposed to > installing it) becomes more complex and involves a (much) higher first time > overhead. We'd also start putting shared libraries into user directories > without asking them first. Might be a problem on shared installations with > many users. The overhead would only be for certain python versions that try to use certain functionality, in this case, python2.4 and fused types. To be honest, the overhead isn't very large. As for compiling shared libraries, I don't think people will complain about shared libraries, that's the only way in which Python and Cython can be used. > Note also that Cython no longer compiles itself when installing in PyPy (at > all), but that would be easy to special case here (and PyPy obviously has > features like the "with" statement). > > Next, I think it would tempt us to split source files into separate modules > just because that way we can use a specific feature in one of them because > it'll get compiled (and the other half is needed for bootstrapping). That > would be bad design. Possibly. In the case of fused types, the code of the fused node is nearly 800 lines, which is probably good to separate from the other, typically smaller nodes, especially considering it's kind of a specific feature. In my case, and I wouldn't mind limiting the functionality until further discussion to that case only, using the with statement really helps keeping track of blocks, and the resulting code is much more readable than it would otherwise be. > OTOH, it might be worth taking a general look at the > code base to see what's really required for bootstrapping, or maybe for > compiling pure Python code in general. Factoring that out, and thus > separating the Python compiler from the Cython specific language features > might (might!) be an actual improvement. (Then again, there's .pxd > overriding and the "cython" module, which add Cython features back in, and > those two make Cython much more attactive...) > > I also started to dislike pyximport because it's way outdated, fragile, > complicated and its features are overly hacked together (and I'm not > entirely innocent in that regard). I would love to see a rewrite that also > supports compiling packages properly. Not a GSoC by itself, but it's > certainly a worthy project. What about a project that aims for separating > out a Python compiler and rewriting pyximport as a jitty frontend for it? > Maybe not the greatest use case ever, but a fun project, I'd say. Yeah, the code is pretty terrible, but it seems to work. > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] dynamically compile and import pure python modules
mark florisson, 27.04.2012 21:16: > On 22 March 2012 20:07, Stefan Behnel wrote: >> mark florisson, 22.03.2012 19:50: >>> For the fused type runtime dispatch I found it very convenient to use >>> the with statement, but that is not supported in Python 2.4. However, >>> the compiler could dynamically compile compiler code with the compiler >>> itself and import it (pyximport), if it is not needed to compile >>> Cython itself. I gave it a try and it seems to work like a charm (but >>> probably needs more testing :): >>> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2 >> >> The advantages are limited, so I'm leaning towards seeing the drawbacks, of >> which there are at least some. For one, *running* Cython (as opposed to >> installing it) becomes more complex and involves a (much) higher first time >> overhead. We'd also start putting shared libraries into user directories >> without asking them first. Might be a problem on shared installations with >> many users. > > The overhead would only be for certain python versions that try to use > certain functionality, in this case, python2.4 and fused types. To be > honest, the overhead isn't very large. As for compiling shared > libraries, I don't think people will complain about shared libraries, > that's the only way in which Python and Cython can be used. > >> Note also that Cython no longer compiles itself when installing in PyPy (at >> all), but that would be easy to special case here (and PyPy obviously has >> features like the "with" statement). >> >> Next, I think it would tempt us to split source files into separate modules >> just because that way we can use a specific feature in one of them because >> it'll get compiled (and the other half is needed for bootstrapping). That >> would be bad design. > > Possibly. In the case of fused types, the code of the fused node is > nearly 800 lines, which is probably good to separate from the other, > typically smaller nodes, especially considering it's kind of a > specific feature. In my case, and I wouldn't mind limiting the > functionality until further discussion to that case only, using the > with statement really helps keeping track of blocks, and the resulting > code is much more readable than it would otherwise be. What about this deal: we remove the hard bootstrap dependency on the fused types code (and maybe other Cython specific features) and require its compilation at install time in Py2.4 (and maybe even 2.5). That would allow us to use newer Python syntax (and even Cython supported syntax) there (except for fused types, obviously). Failure to compile the module in Python 2.4/5 at install time would then abort the installation. Bad luck for the user, but easy to fix by installing a newer Python version. That would give us the advantage of not needing to pollute user home directories with shared libraries at runtime (which I would consider a very annoying property). Making the dependency optional can be as simple as using a try-except conditional import at the module level and setting the module name to None in the failure case. If we want to prevent weird error messages, we can just ignore the import failure during Cython's own installation (by setting a flag somewhere) and during a normal run, we use a guard that checks that the import worked (i.e. the module is non-None) before starting the compilation and raises an error otherwise. We should then clearly document in the module comment that this source file can use newer syntax while all other files cannot. Oh, and we should still take care *not* to split our source base by the "can be compiled or not" predicate. What do you think? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] dynamically compile and import pure python modules
On 27 April 2012 21:16, Stefan Behnel wrote: > mark florisson, 27.04.2012 21:16: >> On 22 March 2012 20:07, Stefan Behnel wrote: >>> mark florisson, 22.03.2012 19:50: For the fused type runtime dispatch I found it very convenient to use the with statement, but that is not supported in Python 2.4. However, the compiler could dynamically compile compiler code with the compiler itself and import it (pyximport), if it is not needed to compile Cython itself. I gave it a try and it seems to work like a charm (but probably needs more testing :): https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2 >>> >>> The advantages are limited, so I'm leaning towards seeing the drawbacks, of >>> which there are at least some. For one, *running* Cython (as opposed to >>> installing it) becomes more complex and involves a (much) higher first time >>> overhead. We'd also start putting shared libraries into user directories >>> without asking them first. Might be a problem on shared installations with >>> many users. >> >> The overhead would only be for certain python versions that try to use >> certain functionality, in this case, python2.4 and fused types. To be >> honest, the overhead isn't very large. As for compiling shared >> libraries, I don't think people will complain about shared libraries, >> that's the only way in which Python and Cython can be used. >> >>> Note also that Cython no longer compiles itself when installing in PyPy (at >>> all), but that would be easy to special case here (and PyPy obviously has >>> features like the "with" statement). >>> >>> Next, I think it would tempt us to split source files into separate modules >>> just because that way we can use a specific feature in one of them because >>> it'll get compiled (and the other half is needed for bootstrapping). That >>> would be bad design. >> >> Possibly. In the case of fused types, the code of the fused node is >> nearly 800 lines, which is probably good to separate from the other, >> typically smaller nodes, especially considering it's kind of a >> specific feature. In my case, and I wouldn't mind limiting the >> functionality until further discussion to that case only, using the >> with statement really helps keeping track of blocks, and the resulting >> code is much more readable than it would otherwise be. > > What about this deal: we remove the hard bootstrap dependency on the fused > types code (and maybe other Cython specific features) and require its > compilation at install time in Py2.4 (and maybe even 2.5). That would allow > us to use newer Python syntax (and even Cython supported syntax) there > (except for fused types, obviously). Failure to compile the module in > Python 2.4/5 at install time would then abort the installation. Bad luck > for the user, but easy to fix by installing a newer Python version. > > That would give us the advantage of not needing to pollute user home > directories with shared libraries at runtime (which I would consider a very > annoying property). I think it's fine to require compiling in the installed case (or will that be a problem for some package managers?). In the non-installed case with python versions smaller than needed, would you prefer a pyximport or an error message telling you to install Cython? Because for development you really don't want to install every time. > Making the dependency optional can be as simple as using a try-except > conditional import at the module level and setting the module name to None > in the failure case. If we want to prevent weird error messages, we can > just ignore the import failure during Cython's own installation (by setting > a flag somewhere) and during a normal run, we use a guard that checks that > the import worked (i.e. the module is non-None) before starting the > compilation and raises an error otherwise. > > We should then clearly document in the module comment that this source file > can use newer syntax while all other files cannot. Oh, and we should still > take care *not* to split our source base by the "can be compiled or not" > predicate. Right. > What do you think? > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
