[issue3139] bytearrays are not thread safe

Martin v. Löwis added the comment: A problem can only occur if you preserve a pointer to the buffer, and the object gets a chance to change its buffer underneath. This can happen when there are user-defined callback, and when other threads can get control. In the cases being fixed, other threads

[issue3139] bytearrays are not thread safe

Boya Sun added the comment: I am still a little bit confused. Can you explain a little more in detail? What is the difference between the suspicious code and the ones that are fixed? -- ___ Python tracker

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc added the comment: In theory, yes. But it happens that the GIL is not released before the buffer is used. -- ___ Python tracker ___

[issue3139] bytearrays are not thread safe

Boya Sun added the comment: Although the bug is fixed, the following three code segments seems suspicious in _codecsmodule.c in the latest revision 74624, and they are similar to the bug described here: (1) escape_decode(PyObject *self, PyObject *args) { ... const char *da

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Hi Travis, Glad you're back! > I'm not convinced that Py_buffer should have grown a link to an object. > I think this is a shortcut solution due to misuse of the protocol that > may have unfortunate consequences. What consequences are you

[issue3139] bytearrays are not thread safe

Travis Oliphant <[EMAIL PROTECTED]> added the comment: I'm sorry that I was unavailable for comment during July and August as it looks like a lot of decisions were made that have changed the semantics a bit. I'm still trying to figure out why the decisions were made that were made. I get the

[issue3139] bytearrays are not thread safe

Ismail Donmez <[EMAIL PROTECTED]> added the comment: > Can you run Lib/test/test_unicode.py directly to know which test > precisely crashes? The latest py3k branch is fine now and the test passes. ___ Python tracker <[EMAIL PROTECTED]>

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le samedi 16 août 2008 à 05:50 +, Ismail Donmez a écrit : > This seems to break test_unicode on MacOSX 10.5.4, > > test_unicode > python(80320,0xa0659fa0) malloc: *** mmap(size=2147483648) failed (error > code=12) > *** error: can't allo

[issue3139] bytearrays are not thread safe

Ismail Donmez <[EMAIL PROTECTED]> added the comment: This seems to break test_unicode on MacOSX 10.5.4, test_unicode python(80320,0xa0659fa0) malloc: *** mmap(size=2147483648) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug python(8032

[issue3139] bytearrays are not thread safe

Changes by Martin v. Löwis <[EMAIL PROTECTED]>: -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> ___ _

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > It doesn't happen with bytearrays, so I suspect it's that you only > DECREF when releasebuffer method exists: Thanks, that was indeed the problem; this is now fixed in r65683 and r65685. My initial observation that test_unicode leaks a lo

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Sorry, the roundup e-mail interface ate some lines of code: >>> b = b'' >>> sys.getrefcount(b) 30 >>> m = memoryview(b) >>> sys.getrefcount(b) 32 >>> del m >>> sys.getrefcount(b) 31 It doesn't happen with bytearrays, so I suspect it's that y

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le jeudi 14 août 2008 à 19:06 +, Martin v. Löwis a écrit : > Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > > > By the way, even without that patch, there is a memory leak: > > With the patch, this memory leak goes away. But n

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > By the way, even without that patch, there is a memory leak: With the patch, this memory leak goes away. Regards, Martin ___ Python tracker <[EMAIL PROTECTED]> __

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le jeudi 14 août 2008 à 18:52 +, Martin v. Löwis a écrit : > The patch is really trivial, and attached. > > Added file: http://bugs.python.org/file4/refcount.diff By the way, even without that patch, there is a memory leak: Python 3

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: The patch is really trivial, and attached. Added file: http://bugs.python.org/file4/refcount.diff ___ Python tracker <[EMAIL PROTECTED]>

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le jeudi 14 août 2008 à 16:13 +, Martin v. Löwis a écrit : > Now, when I try to merge that into the 3k branch, test_unicode terribly > leaks memory :-( It's really frustrating. If you don't have the time to work on it anymore, you can pos

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > I'm a bit confused. In the PyBuffer_Release implementation you > committed, there is no DECREF at all. Oops, I meant to make the reference own by Py_buffer, but actually forgot to implement that. Fixed in r65677, r65678. Now, when I try t

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: > In any case, the corresponding DECREF *does* exist: in memory_dealloc, > PyBuffer_Release is invoked, which releases the reference. I'm a bit confused. In the PyBuffer_Release implementation you committed, there is no DECREF at all. > > By

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > I've seen your recent merge and I don't know if you have finished with > it. Indeed, I'm done, r65654. Unless there are actual bugs in these patches (in the sense that they don't fix the reported problem, or introduce new bugs), I'd like t

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le mardi 12 août 2008 à 16:15 +, Martin v. Löwis a écrit : > I also started working on porting it to 3.0, but couldn't complete that > port yet - the memoryview object doesn't play nicely. I've seen your recent merge and I don't know if y

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: I also started working on porting it to 3.0, but couldn't complete that port yet - the memoryview object doesn't play nicely. ___ Python tracker <[EMAIL PROTECTED]> _

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: I have now committed the patch to 2.6 as r65654, adding changes for the bz2module. I also decided to make the Py_buffer structure own its reference, as I was running out of arguments why not to. In the process, I removed PyObject_ReleaseBuff

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > The last beta is arriving soon. We need to go ahead on this, or retarget > it for 2.7/3.1. I'll look into it tomorrow. Regards, Martin ___ Python tracker <[EMAIL PROTECTED]>

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: The last beta is arriving soon. We need to go ahead on this, or retarget it for 2.7/3.1. ___ Python tracker <[EMAIL PROTECTED]> ___ ___

[issue3139] bytearrays are not thread safe

Changes by Antoine Pitrou <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11033/getargs.patch ___ Python tracker <[EMAIL PROTECTED]> ___ _

[issue3139] bytearrays are not thread safe

Changes by Antoine Pitrou <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11032/codecs.patch ___ Python tracker <[EMAIL PROTECTED]> ___ __

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Here is a new patch wrapping up Martin's patch with the following additions: - getargs.c fixes - codecs module fixes - multiprocessing module fix - fileobject readinto fix - in bytearray deallocator, print out a SystemError if there are existi

[issue3139] bytearrays are not thread safe

Marc-Andre Lemburg <[EMAIL PROTECTED]> added the comment: On 2008-08-02 16:49, Antoine Pitrou wrote: > Antoine Pitrou <[EMAIL PROTECTED]> added the comment: > > You've missed the preceding line that says: > > +consumed = pbuf.len; > > If final is NULL, consumed isn't updated by the call to

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: You've missed the preceding line that says: +consumed = pbuf.len; If final is NULL, consumed isn't updated by the call to PyUnicode_DecodeMBCSStateful and keeps its original value of pbuf.len. (in all honesty, I didn't test under Window

[issue3139] bytearrays are not thread safe

Marc-Andre Lemburg <[EMAIL PROTECTED]> added the comment: Two comments: * I like the new *-getarg parameters, but it would be better to test for '#' first since this is still by far the most used getarg parameter. * Antoine, I think your codecs.c patch has a glitch: +decoded = PyUn

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > I don't know, is there supposed to be a semantic difference between > PyObject_ReleaseBuffer and PyBuffer_Release? If not, I'd say drop it. There are existing callers of it which would need to be changed, perhaps outside the core also; plu

[issue3139] bytearrays are not thread safe

Changes by Giampaolo Rodola' <[EMAIL PROTECTED]>: -- nosy: -giampaolo.rodola ___ Python tracker <[EMAIL PROTECTED]> ___ ___ Python-bugs

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: I'm attaching a patch for getargs.c and another patch for the codecs module. Added file: http://bugs.python.org/file11032/codecs.patch Added file: http://bugs.python.org/file11033/getargs.patch ___ Python

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le vendredi 01 août 2008 à 17:53 +, Amaury Forgeot d'Arc a écrit : > There is a small issue with the patch: in the "w#" format handler, > bf_getwritebuffer(arg, 0, res) is wrong. The third argument should be > &res (there is a compilation

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: Martin, There is a small issue with the patch: in the "w#" format handler, bf_getwritebuffer(arg, 0, res) is wrong. The third argument should be &res (there is a compilation warning on windows), And a few lines after, in the "if (*for

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Selon "Martin v. Löwis" <[EMAIL PROTECTED]>: > > As for making Py_buffer own a reference to the object: what should be > the semantics for PyObject_ReleaseBuffer? I see the following options: > - Drop PyObject_ReleaseBuffer > - make it DECREF

[issue3139] bytearrays are not thread safe

Changes by Martin v. Löwis <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file10969/s_star.diff ___ Python tracker <[EMAIL PROTECTED]> ___ __

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: I have now updated the patch to fix the socket bug, and the rejects bytearray for t#. As for making Py_buffer own a reference to the object: what should be the semantics for PyObject_ReleaseBuffer? I see the following options: - Drop PyObjec

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Selon "Martin v. Löwis" <[EMAIL PROTECTED]>: > > Amaury, I think the issue of thread-safety of the io library should be > decoupled from this issue. I don't think it is release-critical that the > io library is thread-safe (and even if it is

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: Amaury, I think the issue of thread-safety of the io library should be decoupled from this issue. I don't think it is release-critical that the io library is thread-safe (and even if it is release-critical, the problem should be tracked in a

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > But t# uses bf_getcharbuffer(), which does not seem to lock anything. Indeed. I think we can safely drop support for passing buffer objects into t# which have getbuffer/releasebuffer, so the check for releasebuffer being NULL should be add

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: Ok for s#. But t# uses bf_getcharbuffer(), which does not seem to lock anything. I wonder if this can lead to the same kind of problems, for example when calling file_write with a binary file. ___ Py

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > I don't see any changes to s# and t# in your patch. Do you plan to do it > later or do you think this part should be dropped? It's implemented. When bf_releasebuffer is not NULL, convertbuffer will complain "string or read-only buffer". _

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Martin, sorry for noticing this now but on python-dev you had proposed the following: « I propose that new codes s*, t*, w* are added, and that s#,t#,w# refuses objects which implement a releasebuffer procedure (alternatively, s# etc might be

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le jeudi 31 juillet 2008 à 00:00 +, Amaury Forgeot d'Arc a écrit : > Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: > > > If it was rewritten to use a fixed-size bytearray > does such an object exist today? Manually, yes :)

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: > If it was rewritten to use a fixed-size bytearray does such an object exist today? ___ Python tracker <[EMAIL PROTECTED]> ___ _

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le mercredi 30 juillet 2008 à 23:03 +, Amaury Forgeot d'Arc a écrit : > Again, I think this is unfortunate for a simple script that prints from > several threads. Yes, but it's an issue with the BufferedWriter implementation, since it cha

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: It's indeed better. Now with when running my previous script I can see the exception ;-) Exception in thread Thread-2: Traceback (most recent call last): File "C:\dev\python\trunk1\lib\threading.py", line 523, in __bootstrap_inner

[issue3139] bytearrays are not thread safe

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: Sorry, that was my oversight! I've backported the fix. -- nosy: +benjamin.peterson ___ Python tracker <[EMAIL PROTECTED]> _

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: The problem is that the fix for #3295 was committed in the py3k branch (in r64751) rather thank on the trunk! Once PyExc_BufferError is defined properly the crash disappears and exceptions are printed instead.

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: With the patch the following script crashes the 2.6 interpreter on Windows: import sys, io, threading stdout2 = io.open(sys.stdout.fileno(), mode="w") def f(i): for i in range(10): stdout2.write(unicode((x, i)) + '\n') for x

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > (1) are you sure it is safe not to INCREF the obj pointer in the > Py_buffer? Yes, that's the semantics of the current buffer interface, and I cannot see a flaw in it. When you call getbuffer, you certainly hold a reference, and it is your

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Travis, it would be really nice to have your input on this. -- nosy: +teoliphant ___ Python tracker <[EMAIL PROTECTED]> ___ __

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: I haven't yet studied the patch in detail but I have a few questions: (1) are you sure it is safe not to INCREF the obj pointer in the Py_buffer? in the cases handled by your patch we still hold a reference to the original args tuple and dict

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: Here is a patch adding the s* format, and changing files, sockets, and fileio to use it. For bz2, the immediate effect is that you get a type error (as an object providing bf_releasebuffer cannot be converted through s#/w# anymore); it would

[issue3139] bytearrays are not thread safe

Changes by Barry A. Warsaw <[EMAIL PROTECTED]>: -- priority: deferred blocker -> release blocker ___ Python tracker <[EMAIL PROTECTED]> ___

[issue3139] bytearrays are not thread safe

Barry A. Warsaw <[EMAIL PROTECTED]> added the comment: Not blocking beta 2 because there's not enough time for the fix, but this will definitely block beta 3. -- nosy: +barry priority: release blocker -> deferred blocker ___ Python tracker <[EMAIL PRO

[issue3139] bytearrays are not thread safe

Changes by Martin v. Löwis <[EMAIL PROTECTED]>: -- priority: -> release blocker ___ Python tracker <[EMAIL PROTECTED]> ___ ___ Python-b

[issue3139] bytearrays are not thread safe

Changes by Ismail Donmez <[EMAIL PROTECTED]>: -- nosy: +cartman ___ Python tracker <[EMAIL PROTECTED]> ___ ___ Python-bugs-list mailing

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: Fixing this in the methods themselves has the disadvantage that the error reporting is not that good anymore. ___ Python tracker <[EMAIL PROTECTED]> _

[issue3139] bytearrays are not thread safe

Changes by Antoine Pitrou <[EMAIL PROTECTED]>: Added file: http://bugs.python.org/file10823/bzcrash.patch ___ Python tracker <[EMAIL PROTECTED]> ___ ___

[issue3139] bytearrays are not thread safe

Changes by Antoine Pitrou <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file10822/bzcrash.py ___ Python tracker <[EMAIL PROTECTED]> ___

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: For reference, here is a proof-of-concept patch which shows how to fix the bytearray crasher above (it raises a BufferError instead). Added file: http://bugs.python.org/file10822/bzcrash.py ___ Python trac

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Le samedi 05 juillet 2008 à 22:20 +, Martin v. Löwis a écrit : > Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > > I believe the 2.6 s# processing works correctly; the error is in the > bytearray object. This either shouldn't sup

[issue3139] bytearrays are not thread safe

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: I believe the 2.6 s# processing works correctly; the error is in the bytearray object. This either shouldn't support the buffer interface, or it shouldn't reallocate the buffer after having returned a pointer to it. For 3k, convertbuffer sho

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Now I've just discovered there is the same problem with the array.array() type (see following code). import bz2, threading, array bz2c = bz2.BZ2Compressor() # Span at least a whole arena (256KB long) junk_len = 512 * 1024 junk = array.array

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: By the way, here's a more reliable way to make it crash (on my Linux machine): import bz2, threading bz2c = bz2.BZ2Compressor() # Span at least a whole arena (256KB long) junk_len = 512 * 1024 junk = b"a" * junk_len buf = bytearray() for x

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: If I try to follow the chain the consequences: - all PyArg_ParseTuple("s#") calls that release the GIL afterwards should be re-written to use another API (which one I don't know exactly, but hopefully the appropriate functions are already pro

[issue3139] bytearrays are not thread safe

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: Probably, but this affects all PyArg_ParseTuple("s#") calls that release the GIL afterwards. How many of them are there? ___ Python tracker <[EMAIL PROTECTED]> _

[issue3139] bytearrays are not thread safe

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Wow... Isn't this kind of code supposed to ask for a buffer on the bytearray object, together with an optional lock on it (or something like that)? -- nosy: +pitrou ___ Python tracker <[EMAIL PROTE

[issue3139] bytearrays are not thread safe

Changes by Amaury Forgeot d'Arc <[EMAIL PROTECTED]>: -- title: print is not thread safe -> bytearrays are not thread safe ___ Python tracker <[EMAIL PROTECTED]> ___ ___