[Python-Dev] Dealing with import lock deadlock in Import Hooks
[I initially posted this email to python-list but didn't get any reply, probably because this is too related to python core, so I'm posting it again here, hope that's ok...] Hello, I'm currently working on implementing Import Hooks (PEP302) with Python 2.7 to be able to import modules whose code is in ZODB. However, I have stumbled upon a widely known issue about import deadlock[0][1] (note that this issue is not directly related to ZODB, but a more general question about dealing with import lock deadlock for Import Hooks), basically: Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a package containing dynamic modules) handled by Import Hooks I implemented, so import lock is acquired before even running the hooks (Python/import.c:PyImport_ImportModuleLevel()). Then, these import hooks try to load objects from ZODB and a request is sent and handled by another thread (Thread 2) which itself tries to import another module. Of course, this causes a deadlock because the first thread still holds import lock. I have thought about the following solutions: * Backport the patch applied in python 3.3 from issue 9260[0]. This would be the best option because it would mean that even when trying to import any module from package 'foo', other modules and packages can be imported, which would solve my issue. However, I'm not sure it could be released into python 2.7? * Within the hooks, protect the Import Hooks with a separate lock for the loader method. This would prevent any other thread to import any modules from 'foo' package but still allows to call the finder method (ignoring module fullname not starting with 'foo') along with other finder methods, so that other ZODB modules can be imported. Then, in the loader method, until the module is actually inserted into sys.modules and then other load_module() PEP302 responsabilities being taken care of (such as exec the code), release the import lock so that Thread 2 can process requests and send objects back to Thread 1. About the finder method, I think that the separate lock is enough and releasing the import lock until the end of the method should be enough. However, even after trying to understand import.c, I'm not sure this is enough and that releasing import lock would not have nasty side-effects, any thoughts about that? * Fix the ZODB code to not avoid import but to me this seems like a dirty hack because it could happen again and I would prefer to fix this issue once and for all. Any thoughts or suggestion welcome, thanks! Regards, Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Victor Stinner writes: >>I'm currently working on implementing Import Hooks (PEP302) with Python >> 2.7 to be able to import modules whose code is in ZODB. However, I have >> stumbled upon a widely known issue about import deadlock[0][1] (...) > > In Python 3.3, the import machinery has been rewritten (importlib is used > by default) and the import lock is now per module, no more global. Yes, I saw the bug report and its patch implementing the import lock per module (mentioned in my initial email) and watched the presentation by Brett Cannon (BTW, I could not find the diagram explained during the presentation, anyone knows if it's available somewhere?). > Backporting such huge change is difficult and risky. > > Upgrading to Python 3.3 is more future proof and don't require to hack > Python 2.7. I wish I could use Python 3.3 but unfortunately, Zope 2 does not support it. What about the other solution I suggested though? Regards, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Hi, Armin Rigo writes: > On Mon, Aug 12, 2013 at 9:39 AM, Arnaud Fontaine > wrote: >> Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a >> package containing dynamic modules) handled by Import Hooks I >> implemented, so import lock is acquired before even running the hooks >> (Python/import.c:PyImport_ImportModuleLevel()). Then, these import >> hooks try to load objects from ZODB and a request is sent and handled >> by another thread (Thread 2) which itself tries to import another >> module. > > A quick hack might be to call imp.release_lock() and > imp.acquire_lock() explicitly, from your import hook code, around > calls to ZODB. I suggested the same in my initial email, but I was wondering if there could be any issue by releasing the lock in find_module()/load_module() until the module is actually added to sys.modules. Cheers, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Brett Cannon writes: > On Mon, Aug 12, 2013 at 5:12 AM, Arnaud Fontaine wrote: >> Yes, I saw the bug report and its patch implementing the import lock per >> module (mentioned in my initial email) and watched the presentation by >> Brett Cannon (BTW, I could not find the diagram explained during the >> presentation, anyone knows if it's available somewhere?). > > http://prezi.com/mqptpza9xbic/?utm_campaign=share&utm_medium=copy Thanks. Is the full diagram only available somewhere? (I mean as an image or PDF file, not within the presentation document itself) Cheers, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Antoine Pitrou writes: > On Tue, 13 Aug 2013 11:06:51 +0900 Arnaud Fontaine > wrote: >> I suggested the same in my initial email, but I was wondering if there >> could be any issue by releasing the lock in find_module()/load_module() >> until the module is actually added to sys.modules. > > Well, you are obviously on your own with such hacks. There is a reason > the lock exists. Yes. Actually, I was thinking about implementing something similar to what has been done in Python 3.3 but for Python 2.7 with a corser-grain lock. From my understanding of import.c, it should work but I was hoping that someone with more experience in import code would confirm: Currently, I have a package, foo, registered into sys.meta_path which loads modules through its find_module() and load_module() methods (PEP302). Access to load_module() of this package is protected by a RLock I defined, so that modules within foo cannot be imported in parallel. Until the module is added to sys.modules and then the code loaded, release the import lock. For find_module(), only filter the full module name and if this is a module from foo package, then acquired the same RLock defined for load_module() to access variables shared with find_module(). Also, if a patch backporting the features from Python 3.3 to import.c for Python 2.7 would be written, is there any chance it could be accepted? Regards, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Antoine Pitrou writes: > Le Tue, 13 Aug 2013 17:28:42 +0900, Arnaud Fontaine > a écrit : >> Yes. Actually, I was thinking about implementing something similar to >> what has been done in Python 3.3 but for Python 2.7 with a >> corser-grain lock. From my understanding of import.c, it should work >> but I was hoping that someone with more experience in import code >> would confirm: > > It's probably possible, but it will be non-trivial and delicate to get > right. From my understanding of import.c source code, until something is added to sys.modules or the code loaded, there should be no side-effect to releasing the lock, right? (eg there is no global variables/data being shared for importing modules, meaning that releasing the lock should be safe as long as the modules loaded through import hooks are protected by a lock) >> Also, if a patch backporting the features from Python 3.3 to import.c >> for Python 2.7 would be written, is there any chance it could be >> accepted? > > Definitely not. We generally don't backport any features, especially > when the risk is high due to a feature's implementation complexity. I see, thanks for your answer. Regards, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Antoine Pitrou writes: > Le Wed, 14 Aug 2013 14:17:59 +0900, Arnaud Fontaine > a écrit : >> From my understanding of import.c source code, until something is >> added to sys.modules or the code loaded, there should be no >> side-effect to releasing the lock, right? (eg there is no global >> variables/data being shared for importing modules, meaning that >> releasing the lock should be safe as long as the modules loaded >> through import hooks are protected by a lock) > > Er, probably, but import.c is a nasty pile of code. > It's true the import lock is there mainly to: > - avoid incomplete modules from being seen by other threads > - avoid a module from being executed twice Yes. Hopefully, the implementation in Python 3.3 should be much better! ;-) > But that doesn't mean it can't have had any other - unintended - > benefits ;-) Indeed, that's why I checked the source code, but I will check again anyway to make sure. > (also, some import hooks might not be thread-safe, something which they > haven't had to bother about until now) Good point, I didn't think about that. Thanks! Regards, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
