[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Change by Matthias Bussonnier : -- pull_requests: +6645 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Raymond Hettinger <[EMAIL PROTECTED]> added the comment: Amended this fix to avoid quadratic behavior altogether. See issue 3116. -- nosy: +rhettinger ___ Python tracker <[EMAIL PROTECTED]>

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

A.M. Kuchling <[EMAIL PROTECTED]> added the comment: I've applied Facundo's version of the patch in r63059. -- nosy: +akuchling resolution: -> accepted status: open -> closed __ Tracker <[EMAIL PROTECTED]> _

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Dustin J. Mitchell <[EMAIL PROTECTED]> added the comment: This test passes -- is this ready to commit? I see a reduction from 1.9s to 1.5s for the test script in msg59715. -- nosy: +djmitche __ Tracker <[EMAIL PROTECTED]>

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Changes by Georg Brandl <[EMAIL PROTECTED]>: -- type: resource usage -> performance __ Tracker <[EMAIL PROTECTED]> __ ___ Python-bugs-list maili

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Facundo Batista added the comment: Christian: you can not do that, as you *must* increase the size. Anyway, I think that is a good idea to not let it duplicate after some point. Which point? It's arbitrary. You suggested 512KB, but I think that is too low. *My* arbitrary point is, when the new

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Christian Heimes added the comment: Yet another easy C task for the bug day. -- keywords: +easy __ Tracker <[EMAIL PROTECTED]> __ ___ Python-bug

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Christian Heimes added the comment: It looks like a reasonable and good patch. Aaron's tests clearly show the speed up. However the maximum resize should be limited to a value around 256 to 1024k: size = min(2*size + 1024, 512*1024); -- keywords: +patch nosy: +tiran priority: -> hig

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Aaron Watters added the comment: also: I just modified the code to do iterations using increasingly large data sizes and I see the kind of very unpleasant behaviour for the old implementation (max time varies wildly from min time) that I saw in my more complex program. The new implementation do

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Aaron Watters added the comment: Facundo 1) the +1024 was an accelerator to jump up to over 1k at the first resize. I think it's a good idea or at least doesn't hurt. 2) Here is an example program: def test(): from marshal import dumps from time import time testString = "abc"*1

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

Facundo Batista added the comment: Why not just double the size? The "doubling + 1024" address some specific issue? If so, it should be commented. Also, do you have an example of a marshal.dumps() that suffers from this issue? Thank you! -- nosy: +facundobatista __

[issue1792] o(n*n) marshal.dumps performance for largish objects with patch

New submission from Aaron Watters: Much to my surprise I found that one of my applications seemed to be running slow as a result of marshal.dumps. I think the culprit is the w_more(...) function, which grows the marshal buffer in 1k units. This means that a marshal of size 100k will have 100 rea