On 4 August 2013 23:40, Eli Bendersky <eli...@gmail.com> wrote:
> On Sat, Aug 3, 2013 at 7:08 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>> Sure, it's just unusual to have a case where "importing is blocked by adding
>> None to sys.modules" differs from "not actually available", so I'd like to
>> understand the situation better.
>
> I must admit I'm confused by the behavior of import_fresh_module too.
>
> Snippet #1 raises the expected ImportError:
>
> sys.modules['pyexpat'] = None
> import _elementtree
>
> However, snippet #2 succeeds importing:
>
> ET = import_fresh_module('_elementtree', blocked=['pyexpat'])
> print(ET)/me goes and looks That function was much simpler when it was first created :P Still, I'm fairly confident the complexity of that dance isn't relevant to the problem you're seeing. > I believe this happens because import_fresh_module does an import of > the 'name' it's given before even looking at the blocked list. Then, > it assigns None to sys.modules for the blocked names and re-imports > the module. So in essence, this is somewhat equivalent to snippet #3: > > modname = '_elementtree' > __import__(modname) > del sys.modules[modname] > for m in sys.modules: > if modname == m or m.startswith(modname + '.'): > del sys.modules[m] > sys.modules['pyexpat'] = None > ET = importlib.import_module(modname) > print(ET) > > Which also succeeds. > > I fear I'm not familiar enough with the logic of importing to > understand what's going on, but it has been my impression that this > problem is occasionally encountered with import_fresh_module and C > code that imports stuff (the import of pyexpat is done by C code in > this case). I had missed it was a C module doing the import. Looking into the _elementtree.c source, the problem in this case is the fact that a shared library that doesn't use PEP 3121 style per-module state is only loaded and initialised once, so reimporting it gets the same module back (from the extension loading cache), even if the Python level reference has been removed from sys.modules. Non PEP 3121 C extension modules thus don't work properly with test.support.import_fresh_module (as there's an extra level of caching involved that *can't* be cleared from Python, because it would break things). To fix this, _elementree would need to move the pyexpat C API pointer to per-module state, rather than using a static variable (see http://docs.python.org/3/c-api/module.html#initializing-c-modules). With per-module state defined, the import machine should rerun the init function when the fresh import happens, thus creating a new copy of the module. However, this isn't an entirely trivial change for _elementree, since: 1. Getting from the XMLParser instance back to the module where it was defined in order to retrieve the capsule pointer via PyModule_GetState() isn't entirely trivial in C. You'd likely do it once in the init method, store the result in an XMLParser attribute, and then tweak the EXPAT() using functions to include an appropriate local variable definition at the start of the method implementation. 2. expat_set_error would need to be updated to accept the pyexpat capsule pointer as a function parameter Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
