[Cython] Multiple modules in one compilation unit
This has been raised earlier, but I don't think there was such a
demonstrative use-case as what I have now.
Fwrap is suppose to be able to wrap Fortran "modules", which is
essentially a namespace mechanism. It makes sense to convert the
namespace to Python by creating one Cython pyx file per Fortran module.
However, if you are wrapping a Fortran library this way, you suddenly
have lots of opportunity to mess up the build:
- If you build the Fortran code as a static library (rather
common...), then each pyx file will have their own copy. This will link
successfully but likely have a rather poor effect.
- If you link each Cython file with only the corresponding Fortran
file, things won't work (likely to get missing symbols from cross-module
calls Fortran-side).
Yes, linking each Cython file to the same shared Fortran library should
work. Still, this seems dangerous.
Options:
a) Simply make sure to link with shared versions of Fortran libraries
("this is a documentation problem")
b) Use some other namespace mechanism in the same pyx (classes with
static methods...)
c) Somehow provide more than one module in the same compilation unit.
Again, this requires the build to work correctly, but seems less
dangerous, and also has the advantage of *allowing* static linking of
the Fortran library, if one wants to.
But is something like this possible? Could one implement
cython -o assembly.c file1.pyx file2.pyx file3.pyx
...where assembly.so would contain three Python modules? (initfile1,
initfile2, and so on...)
Dag Sverre
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
Dag Sverre Seljebotn, 02.03.2011 11:20: c) Somehow provide more than one module in the same compilation unit. Again, this requires the build to work correctly, but seems less dangerous, and also has the advantage of *allowing* static linking of the Fortran library, if one wants to. But is something like this possible? Could one implement cython -o assembly.c file1.pyx file2.pyx file3.pyx ...where assembly.so would contain three Python modules? (initfile1, initfile2, and so on...) Can't currently work because the three modules would define the same static C names. This could be fixed as part of the PEP 3121 implementation: http://trac.cython.org/cython_trac/ticket/173 Or it could be worked around by overriding the prefixes in Naming.py (which sounds ugly). Generally speaking, I'm -1 on this idea. I don't see a real use case, and you're saying yourself that this isn't required to make your Fortran use case work either. - If you build the Fortran code as a static library (rather common...), then each pyx file will have their own copy. This will link successfully but likely have a rather poor effect. So? lxml has two main modules, and if you build it statically against libxml2/libxslt, you end up with two static versions of that library in the two modules. Takes up some additional space, but otherwise works beautifully. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On 03/02/2011 11:48 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 11:20: c) Somehow provide more than one module in the same compilation unit. Again, this requires the build to work correctly, but seems less dangerous, and also has the advantage of *allowing* static linking of the Fortran library, if one wants to. But is something like this possible? Could one implement cython -o assembly.c file1.pyx file2.pyx file3.pyx ...where assembly.so would contain three Python modules? (initfile1, initfile2, and so on...) Can't currently work because the three modules would define the same static C names. This could be fixed as part of the PEP 3121 implementation: http://trac.cython.org/cython_trac/ticket/173 Or it could be worked around by overriding the prefixes in Naming.py (which sounds ugly). Generally speaking, I'm -1 on this idea. I don't see a real use case, and you're saying yourself that this isn't required to make your Fortran use case work either. But assuming work is spent on Cython, it *could* work? I.e. there's not a problem with import mechanisms; Python assumes one module per .so or similar? Or would one have to write a custom importer? - If you build the Fortran code as a static library (rather common...), then each pyx file will have their own copy. This will link successfully but likely have a rather poor effect. So? lxml has two main modules, and if you build it statically against libxml2/libxslt, you end up with two static versions of that library in the two modules. Takes up some additional space, but otherwise works beautifully. Problem is that Fortran code often has...interesting...programming practices. Global variables abound, and are often initialised between modules. Imagine: settings_mod.set_alpha(0.34) print compute_mod.get_alpha_squared() This behaves quite differently with two static versions rather than one... Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
Dag Sverre Seljebotn, 02.03.2011 11:54: On 03/02/2011 11:48 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 11:20: c) Somehow provide more than one module in the same compilation unit. Again, this requires the build to work correctly, but seems less dangerous, and also has the advantage of *allowing* static linking of the Fortran library, if one wants to. But is something like this possible? Could one implement cython -o assembly.c file1.pyx file2.pyx file3.pyx ...where assembly.so would contain three Python modules? (initfile1, initfile2, and so on...) Can't currently work because the three modules would define the same static C names. This could be fixed as part of the PEP 3121 implementation: http://trac.cython.org/cython_trac/ticket/173 Or it could be worked around by overriding the prefixes in Naming.py (which sounds ugly). Generally speaking, I'm -1 on this idea. I don't see a real use case, and you're saying yourself that this isn't required to make your Fortran use case work either. But assuming work is spent on Cython, it *could* work? I.e. there's not a problem with import mechanisms You can have as many modules in a .so as you like, see the embedding support, for example. You just have to find a way to import them explicitly, which usually requires some call into the .so (as CPython does when calling the initmodule() function) to let it register the modules. Potentially, you could have one module in there that has the name of the .so file, and let that register the other modules automatically in its initmodule() function. Similar to a package's __init__.py that can import other modules from the package when it gets imported itself. You could think of your feature as a "package in a module". Actually, if Cython had direct support for this, it could simply join the static definitions in the separate modules into one, including merged string constants etc. - If you build the Fortran code as a static library (rather common...), then each pyx file will have their own copy. This will link successfully but likely have a rather poor effect. So? lxml has two main modules, and if you build it statically against libxml2/libxslt, you end up with two static versions of that library in the two modules. Takes up some additional space, but otherwise works beautifully. Problem is that Fortran code often has...interesting...programming practices. Global variables abound, and are often initialised between modules. Imagine: settings_mod.set_alpha(0.34) print compute_mod.get_alpha_squared() This behaves quite differently with two static versions rather than one... Then I'd suggest always linking dynamically. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On 2 March 2011 08:35, Stefan Behnel wrote: > Dag Sverre Seljebotn, 02.03.2011 11:54: >> >> Problem is that Fortran code often has...interesting...programming >> practices. Global variables abound, and are often initialised between >> modules. Imagine: >> >> settings_mod.set_alpha(0.34) >> print compute_mod.get_alpha_squared() >> >> This behaves quite differently with two static versions rather than one... > > Then I'd suggest always linking dynamically. > And where are you going to put your fortran shared libraries? Dynamic linking details varies wildly across platforms... I'm very much understand Dag's use case and concerns, and I do think that some research in all this is worth it. -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: On 2 March 2011 08:35, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 11:54: Problem is that Fortran code often has...interesting...programming practices. Global variables abound, and are often initialised between modules. Imagine: settings_mod.set_alpha(0.34) print compute_mod.get_alpha_squared() This behaves quite differently with two static versions rather than one... Then I'd suggest always linking dynamically. And where are you going to put your fortran shared libraries? Dynamic linking details varies wildly across platforms... I'm very much understand Dag's use case and concerns, and I do think that some research in all this is worth it. I'm not sure if there's much more to research at the moment -- Stefan says it is possible, and that's what I wanted to know at this stage. If I want it, I obviously need to implement it myself. (And if such a patch implements PEP 3121 and there's a demonstrated need for it with some users, I really can't see it getting rejected just out of it being in "poor taste"). I.e., I'm going to make Fwrap spit out multiple pyx files and worry about this later. If multiple .pyx in one .so was fundamentally impossible, I might have gone another route with Fwrap. That was all. Thanks Stefan!, Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
Dag Sverre Seljebotn, 02.03.2011 16:37: On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: On 2 March 2011 08:35, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 11:54: Problem is that Fortran code often has...interesting...programming practices. Global variables abound, and are often initialised between modules. Imagine: settings_mod.set_alpha(0.34) print compute_mod.get_alpha_squared() This behaves quite differently with two static versions rather than one... Then I'd suggest always linking dynamically. And where are you going to put your fortran shared libraries? Dynamic linking details varies wildly across platforms... I'm very much understand Dag's use case and concerns, and I do think that some research in all this is worth it. I'm not sure if there's much more to research at the moment -- Stefan says it is possible, and that's what I wanted to know at this stage. If I want it, I obviously need to implement it myself. (And if such a patch implements PEP 3121 and there's a demonstrated need for it with some users, I really can't see it getting rejected just out of it being in "poor taste"). I.e., I'm going to make Fwrap spit out multiple pyx files and worry about this later. If multiple .pyx in one .so was fundamentally impossible, I might have gone another route with Fwrap. That was all. The feature I could imagine becoming part of Cython is "compiling packages". I.e. you'd call "cython" on a package and it would output a directory with a single __init__.so that contains the modules compiled from all .pyx/.py files in that package. Importing the package would then trigger an import of that __init__.so, which in turn will execute code in its init__init__() function to register the other modules. Would that work for you? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On 03/02/2011 05:01 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 16:37: On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: On 2 March 2011 08:35, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 11:54: Problem is that Fortran code often has...interesting...programming practices. Global variables abound, and are often initialised between modules. Imagine: settings_mod.set_alpha(0.34) print compute_mod.get_alpha_squared() This behaves quite differently with two static versions rather than one... Then I'd suggest always linking dynamically. And where are you going to put your fortran shared libraries? Dynamic linking details varies wildly across platforms... I'm very much understand Dag's use case and concerns, and I do think that some research in all this is worth it. I'm not sure if there's much more to research at the moment -- Stefan says it is possible, and that's what I wanted to know at this stage. If I want it, I obviously need to implement it myself. (And if such a patch implements PEP 3121 and there's a demonstrated need for it with some users, I really can't see it getting rejected just out of it being in "poor taste"). I.e., I'm going to make Fwrap spit out multiple pyx files and worry about this later. If multiple .pyx in one .so was fundamentally impossible, I might have gone another route with Fwrap. That was all. The feature I could imagine becoming part of Cython is "compiling packages". I.e. you'd call "cython" on a package and it would output a directory with a single __init__.so that contains the modules compiled from all .pyx/.py files in that package. Importing the package would then trigger an import of that __init__.so, which in turn will execute code in its init__init__() function to register the other modules. Would that work for you? Yes, that sounds like exactly what I'm after. Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On 2 March 2011 13:01, Stefan Behnel wrote: > Dag Sverre Seljebotn, 02.03.2011 16:37: >> >> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: >>> >>> On 2 March 2011 08:35, Stefan Behnel wrote: Dag Sverre Seljebotn, 02.03.2011 11:54: > > Problem is that Fortran code often has...interesting...programming > practices. Global variables abound, and are often initialised between > modules. Imagine: > > settings_mod.set_alpha(0.34) > print compute_mod.get_alpha_squared() > > This behaves quite differently with two static versions rather than > one... Then I'd suggest always linking dynamically. >>> And where are you going to put your fortran shared libraries? Dynamic >>> linking details varies wildly across platforms... I'm very much >>> understand Dag's use case and concerns, and I do think that some >>> research in all this is worth it. >> >> I'm not sure if there's much more to research at the moment -- Stefan says >> it is possible, and that's what I wanted to know at this stage. If I want >> it, I obviously need to implement it myself. (And if such a patch >> implements PEP 3121 and there's a demonstrated need for it with some >> users, >> I really can't see it getting rejected just out of it being in "poor >> taste"). >> >> I.e., I'm going to make Fwrap spit out multiple pyx files and worry about >> this later. If multiple .pyx in one .so was fundamentally impossible, I >> might have gone another route with Fwrap. That was all. > > The feature I could imagine becoming part of Cython is "compiling packages". > I.e. you'd call "cython" on a package and it would output a directory with a > single __init__.so that contains the modules compiled from all .pyx/.py > files in that package. Importing the package would then trigger an import of > that __init__.so, which in turn will execute code in its init__init__() > function to register the other modules. > $ cd /tmp/ $ mkdir pkg $ touch pkg/__init__.so $ python -c 'import pkg' Traceback (most recent call last): File "", line 1, in ImportError: No module named pkg So that's not going to work... The import machinery ignores __init__.so files. Instead, you need a "package.so" file, the initpackage() init function should in turn setup the package structure and initialize submodules. I've sent privately to Dag an example of how this should be done. -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] "__ipow__ has wrong number of arguments" is back here
Hi! I noticed that this error came back again. https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "__ipow__ has wrong number of arguments" is back here
On 2 March 2011 16:18, Vitja Makarov wrote: > Hi! > > I noticed that this error came back again. > > https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ > I've pushed a fix for __ipow__ to take just two arguments, like regular Python classes. https://github.com/cython/cython/commit/490fc80e6b494f8ed65f78fc2032d856ec75b8ef But I forgot to update test for ticket #561, now this should fixed: https://github.com/cython/cython/commit/829d6485b77b5688671959f840712c01e6886556 -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] strange bug in starred target
2011/3/3 Vitja Makarov : > This doesn't work: > def assign(): > a, *b = 1,2,3,4,5 > return a, b > import x x.assign() > Traceback (most recent call last): > File "", line 1, in > File "x.pyx", line 6, in x.assign (x.c:445) > a, *b = 1,2,3,4,5 > TypeError: Expected list, got int > > And this is ok: > > def assign(): > a, *b = 1,2,3,4,5 > return a, b > *a, b = 1,2,3,4,5 > return a, b > import x x.assign() > (1, [2, 3, 4, 5]) > > I hope problem is somewhere in map_starred_assignment() -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
Stefan Behnel wrote:
you'd call "cython" on a package and it would output a
directory with a single __init__.so that contains the modules compiled
from all .pyx/.py files in that package. Importing the package would
then trigger an import of that __init__.so, which in turn will execute
code in its init__init__() function to register the other modules.
I don't think it even has to be a directory with an __init__,
it could just be an ordinary .so file with the name of the
package.
I just tried an experiment in Python:
# onefilepackage.py
import new, sys
blarg = new.module("blarg")
blarg.thing = "This is the thing"
sys.modules["onefilepackage.blarg"] = blarg
and two different ways of importing it:
>>> from onefilepackage import blarg
>>> blarg
>>> blarg.thing
'This is the thing'
>>> import onefilepackage.blarg
>>> onefilepackage.blarg.thing
'This is the thing'
So assuming the same thing works with a .so instead of a .py,
all you need to do is emit a .so whose init function stuffs
appropriate entries into sys.modules to make it look like
a package.
--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 26, 2011 at 07:11:48PM -0800, Robert Bradshaw wrote: > On Sat, Feb 26, 2011 at 6:14 PM, W. Trevor King wrote: > > On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote: > >> On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: > >> > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: > >> >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King > >> >> wrote: > >> >> > An easy, if uglier, workaround would be to prepend attributes with the > >> >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. > >> >> > Then the Ctx class could subclass the current CtxAttribute classes > >> >> > instead of binding instances of each of them. That way Ctx would keep > >> >> > its traditional flat attribute namespace and easy deepcopy, but > >> >> > eveyone in Nodes, etc. that will use the attributes would become > >> >> > class-name dependent. > >> >> > >> >> I'd be up for flattening this. In particular, changing every > >> >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn > >> >> and extra verbiage for not much benefit. The only overlap I see is > >> >> name and visibility, and keeping name/cname and adding cvisibility > >> >> would be preferable to me. > >> > > >> > That works for me, but I see possible ambiguity in cname. From the > >> > "external C code" docs: > >> > > >> >The other way is to use a C name specification to give different > >> >Cython and C names to the C function. Suppose, for example, that > >> >you want to wrap an external function called eject_tomato(). If > >> >you declare it as: > >> > > >> >cdef extern void c_eject_tomato "eject_tomato" (float speed) > >> > > >> >then its name inside the Cython module will be c_eject_tomato, > >> >whereas its name in C will be eject_tomato. > >> > > >> > In this case I was eventually going to use > >> > > >> >c_source.name = eject_tomato > >> >c_source.namespace = ... > >> >c_binding.name = c_eject_tomato > >> >c_binding.namespace = ... > >> >python_binding.name = eject_tomato > >> > > >> > I'm not sure how Cython handles name/cname with this case at the > >> > moment, but it seems like there are two cnames to keep track of. > >> > >> In this case, cname is "eject_tomato" (which actually gets emitted in > >> the C source file to use it) and name (in both Python and Cython > >> namespace) is "c_eject_tomato." > > > > What is the "Cython namespace"? Is that what you get when you cimport > > something (vs. the Python namespace being what you get when you import > > something). > > Yes, that would be a good way to understand it. Okay, I'm back to a flat attribute namespace in my branch, and I'm moving through Nodes removing the places where I currently collapse back to an overloaded visibility. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt pgpsvqE3jNR30.pgp Description: PGP signature ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "__ipow__ has wrong number of arguments" is back here
On 2 March 2011 17:14, Lisandro Dalcin wrote: > On 2 March 2011 16:18, Vitja Makarov wrote: >> Hi! >> >> I noticed that this error came back again. >> >> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ >> > > I've pushed a fix for __ipow__ to take just two arguments, like > regular Python classes. > https://github.com/cython/cython/commit/490fc80e6b494f8ed65f78fc2032d856ec75b8ef > > But I forgot to update test for ticket #561, now this should fixed: > https://github.com/cython/cython/commit/829d6485b77b5688671959f840712c01e6886556 > BTW, do you all agree with this change? I expect this change to break some code, particularly in Sage... -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
Previous discussions in this thread [1,2] have discussed the issues associated with overloading the 'public' keyword. For an example of the difficulties this causes, see the ugly workaround [3] in my recent commit [4]. Definately worth fixing this syntax. How do syntax changes with deprecations work? The Parsing module doesn't seem to be setup to handle things like that. [1] http://mail.python.org/pipermail/cython-devel/2011-February/81.html [2] http://mail.python.org/pipermail/cython-devel/2011-February/86.html [3] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f#L1R2552 [4] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt pgp1dbiqfsl8X.pgp Description: PGP signature ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King wrote: > Previous discussions in this thread [1,2] have discussed the issues > associated with overloading the 'public' keyword. For an example of > the difficulties this causes, see the ugly workaround [3] in my recent > commit [4]. Definately worth fixing this syntax. > > How do syntax changes with deprecations work? The Parsing module > doesn't seem to be setup to handle things like that. It's not. We rarely deprecate syntax, and we should at least give people a chance to move away from the old syntax first to the new first, so we'll have to support both for a bit at least. When we do deprecate the old way of doing things, I think we'll just use Errors.warning(...). In the long run it'd be nice to have a formal grammar for Cython and use a generated parser rather than maintaining this all by hand, but Parsing.py does more than just parsing at the moment. - Robert > [1] http://mail.python.org/pipermail/cython-devel/2011-February/81.html > [2] http://mail.python.org/pipermail/cython-devel/2011-February/86.html > [3] > https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f#L1R2552 > [4] > https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f > > -- > This email may be signed or encrypted with GPG (http://www.gnupg.org). > The GPG signature (if present) will be attached as 'signature.asc'. > For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy > > My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
On 2 March 2011 21:01, Greg Ewing wrote:
> Stefan Behnel wrote:
>>
>> you'd call "cython" on a package and it would output a directory with a
>> single __init__.so that contains the modules compiled from all .pyx/.py
>> files in that package. Importing the package would then trigger an import of
>> that __init__.so, which in turn will execute code in its init__init__()
>> function to register the other modules.
>
> I don't think it even has to be a directory with an __init__,
> it could just be an ordinary .so file with the name of the
> package.
>
> I just tried an experiment in Python:
>
> # onefilepackage.py
> import new, sys
> blarg = new.module("blarg")
> blarg.thing = "This is the thing"
> sys.modules["onefilepackage.blarg"] = blarg
>
> and two different ways of importing it:
>
from onefilepackage import blarg
blarg
>
blarg.thing
> 'This is the thing'
>
import onefilepackage.blarg
onefilepackage.blarg.thing
> 'This is the thing'
>
I'm hacking around these lines. However, I'm working to maintain
different modules in different C compilation units, in order to
workaround the obvious issue with duplicated global C symbols.
> So assuming the same thing works with a .so instead of a .py,
> all you need to do is emit a .so whose init function stuffs
> appropriate entries into sys.modules to make it look like
> a package.
>
However, the import machinery does not treat .so the same as *.pyx.
For example, I have a problem with Python 3. For .py modules, before
the module code starts to execute, the matching entries in sys.modules
is already there. The same happens in Python 2 for .so modules, as
Py_InitModule() add the entry in sys.modules early. However, in Python
3 that is not te case (and only for the .so, for .py is the same as in
Py2), the import machinery adds the entry later, after the
finalization of the module init function. I'm tempted to workaround
this by setting the entry in sys.modules right after the call to
PyModule_Create() ... What do you think about this? Any potential
issue?
--
Lisandro Dalcin
---
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Multiple modules in one compilation unit
Lisandro Dalcin, 03.03.2011 05:38:
On 2 March 2011 21:01, Greg Ewing wrote:
Stefan Behnel wrote:
you'd call "cython" on a package and it would output a directory with a
single __init__.so that contains the modules compiled from all .pyx/.py
files in that package. Importing the package would then trigger an import of
that __init__.so, which in turn will execute code in its init__init__()
function to register the other modules.
I don't think it even has to be a directory with an __init__,
it could just be an ordinary .so file with the name of the
package.
I just tried an experiment in Python:
# onefilepackage.py
import new, sys
blarg = new.module("blarg")
blarg.thing = "This is the thing"
sys.modules["onefilepackage.blarg"] = blarg
and two different ways of importing it:
>>> from onefilepackage import blarg
>>> blarg
>>> blarg.thing
'This is the thing'
>>> import onefilepackage.blarg
>>> onefilepackage.blarg.thing
'This is the thing'
I'm hacking around these lines. However, I'm working to maintain
different modules in different C compilation units, in order to
workaround the obvious issue with duplicated global C symbols.
That should be ok as a first attempt to get it working quickly. I'd still
like to see the modules merged in the long term in order to increase the
benefit of the more compact format. They'd all share the same code
generator and Cython's C internals, C helper functions, constants,
builtins, etc., but each of them would use a separate (name mangling) scope
to keep the visible C names separate.
So assuming the same thing works with a .so instead of a .py,
all you need to do is emit a .so whose init function stuffs
appropriate entries into sys.modules to make it look like
a package.
However, the import machinery does not treat .so the same as *.pyx.
For example, I have a problem with Python 3. For .py modules, before
the module code starts to execute, the matching entries in sys.modules
is already there.
And it has to be, in order to prevent accidental reimports.
The same happens in Python 2 for .so modules, as
Py_InitModule() add the entry in sys.modules early. However, in Python
3 that is not te case (and only for the .so, for .py is the same as in
Py2), the import machinery adds the entry later, after the
finalization of the module init function. I'm tempted to workaround
this by setting the entry in sys.modules right after the call to
PyModule_Create() ... What do you think about this? Any potential
issue?
No idea. I'd say, read the source and give it a try.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Control flow graph
2011/2/28 Vitja Makarov :
> 2011/2/23 Vitja Makarov :
>> 2011/2/23 Vitja Makarov :
>>> 2011/2/22 Stefan Behnel :
Vitja Makarov, 20.02.2011 18:23:
>
> 2011/2/16 Vitja Makarov:
>>
>> Hmm... both python and codespeaks in the thread
Yes, we should keep it to cython-devel only. Sorry for mixing it up.
>> Here is my commit it's mostly broken now but anyway
>>
>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
Flow control support is large enough to merit its own module. Not sure how
'smart' git is here, but you can always keep the history by explicitly
copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated
sections from both files.
>>>
>>> Ok.
>>>
>>
>> Oops there is no copy command in git.
>>
>>
You are duplicating some code from the type inferencer. We might want to
clean that up at some point. However, given that flow control analysis will
allow us to improve the type inferencer, I think it's best to keep this
code
in the FCA part.
>>>
>>> Yes, I think it could replace MarkAssignments transform later.
>>> Unreachable code could be delete there too.
>>>
> I've update stuff:
> - algo for finding definitions
> - warnings for uninitialized and may be uninitialised use
> - few test cases
That looks very nice so far. Any idea how well it scales?
>>>
>>> "Usually iterative algorithm takes no more then 5 iterations"
>>>
>>> For ExprNodes.py max number is 15 while avg is about 3
>>>
>>> About execution time:
>>>
>>> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325,
>>> ~10% slow down.
>>> -O flag could be introduced but I don't think that's a good idea.
>>>
>>> Should later try to execute cython compiled code.
>>>
> Trying to compile ParseTreeTransforms.py I've found this for example:
>
> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
> 'template' may be used uninitialized
>
> def create_Property(self, entry):
> if entry.visibility == 'public':
> if entry.type.is_pyobject:
> template = self.basic_pyobject_property
> else:
> template = self.basic_property
> elif entry.visibility == 'readonly':
> template = self.basic_property_ro
> property = template.substitute({
> u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,
>
> obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
> attribute=entry.name),
> }, pos=entry.pos).stats[0]
Ok, I guess that code generally works, but it's better to get rid of the
code smell.
>>>
>>> Might be used warning should be disabled by default, because algorithm
>>> isn't smart enough:
>>>
>>> a = 1
>>> if (a): b = 1
>>> if (a): print b
>>>
>>> See also:
>>> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325
>>>
>
>
> I've updated things:
>
> - Now it passes all the tests (some tests are run with
> remove_unreachable=False)
> - Control flow graph support all the control statements (hope so)
> - Analyse PyClassDefNode (now only inner, need some more work)
> - Now we have the following optional warnings:
> 1. uninitialized variable use [default on]
> 2. possibly uninitialized use [-Wextra]
> 3. unreachable code [default on] (this needs rework, unreachable
> code detection could be better handled inside CreateControlFlowGraph)
> 4. unused variable [-Wextra]
> 5. unused argument [-X warn.unused_arg=True]
> 6. unused result [-X warn_unused_result=True]
>
> - Optional dot graph output:
>
> $ cython foo.pyx -X control_flow.dot_output=foo.dot
> $ dot -Tpng -o foo.png foo.dot
>
I've pushed unused varaiable/result removal.
Unused argument removal require some more work, especially generic
args and kwargs.
Now I'm going to handle local variable deletion and then remove
Py_None initialization.
--
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
