[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Sagiv Malihi]

New submission from Sagiv Malihi :

When trying to cPickle.loads() from several threads at once, there is a race 
condition when threads try to import modules.

An example will explain it best:
suppose I have module foo.py which takes some time to load:

import time
class A(object):
def __setstate__(self, state):
self.x = x
time.sleep(1)
x = 5

and a pickled version of an A() object stored in 'A.pkl'.
the following code, when run for the first time, will raise a NameError about 
'x':

>>> p = open('A.pkl','rb').read()
>>> [thread.start_new(cPickle.loads, (p,)) for x in xrange(2)]

Unhandled exception in thread started by 
Traceback (most recent call last):
  File "foo.py", line 7, in __setstate__
self.x = x
NameError: global name 'x' is not defined


since the module is now loaded, subsequent calls to cPickle.loads will work as 
expected.

This was tested on 2.5.2, 2.7.1, and 3.2 on Ubuntu and on Windows 7.

please note that this bug was discovered when unpickling the standard 
'decimal.Decimal' class (decimal.py is quite long and takes some time to 
import), and this is not some corner case.

--
components: Extension Modules
messages: 141548
nosy: Sagiv.Malihi
priority: normal
severity: normal
status: open
title: cPickle.loads is not thread safe due to non-thread-safe imports
type: crash
versions: Python 2.7

___
Python tracker 
<http://bugs.python.org/issue12680>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Sagiv Malihi]

Sagiv Malihi  added the comment:

OK, digging deeper reveals that there are actually two bugs here, one is 
conceptual in the python importing mechanism, and the other is technical in 
cPickle.

The first bug: 
PyImport_ExecCodeModuleEx adds the module to sys.modules *before* actually 
executing the code. This is a design flaw (can it really be changed? )
Demonstrating this bug is easy using the foo.py module from the previous 
comment:

def f():
if 'bla' in sys.modules:
bla = sys.modules['bla']
else:
import bla
return bla.A()
running two instances of f in two threads results in the same error.

The second bug: in cPickle.c: func_class() 
cPickle 'manually' checks if a module is in sys.modules instead of letting the 
import mechanism do it for him (hence breaking the import lock's defense here).

--

___
Python tracker 
<http://bugs.python.org/issue12680>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12680] cPickle.loads is not thread safe due to non-thread-safe imports

[Sagiv Malihi]

Sagiv Malihi  added the comment:

As I said - it certainly happenes on 3.2 (tested).

@pitrou - what you suggested will not work since the actual import will block 
on the import lock.
The optimization there is not needed, it's already implemented in __import__.

--

___
Python tracker 
<http://bugs.python.org/issue12680>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8733] list type and UserList do not call super in __init__ and therefore, they cannot be parents in a multiple inheritence scheme

[Sagiv Malihi]

Changes by Sagiv Malihi :


--
nosy: +Sagiv.Malihi

___
Python tracker 
<http://bugs.python.org/issue8733>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8733] list type and UserList do not call super in __init__ and therefore, they cannot be parents in a multiple inheritence scheme

[Sagiv Malihi]

Sagiv Malihi  added the comment:

Example:

class A(object):
 def __init__(self):
  print "initializing something very important"
  # and then pay it forward...
  super(A, self).__init__()

class B(list, A): pass

b = B()
# A's init is never called...

--

___
Python tracker 
<http://bugs.python.org/issue8733>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com