[issue5683] Speed up cPickle's pickling generally

Alexandre Vassalotti added the comment: Too late to make this change in 2.x. And the patch in issue 9410 includes the optimization for 3.x. -- resolution: -> duplicate stage: patch review -> committed/rejected status: open -> closed superseder: -> Add Unladen Swallow's optimizations

[issue5683] Speed up cPickle's pickling generally

Changes by Skip Montanaro : -- nosy: -skip.montanaro ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: Oh, BTW, the proposed fix is in Rietveld: http://codereview.appspot.com/189051 -- ___ Python tracker ___ ___

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: Perhaps. Let's take it one step at a time though. If I change your large pickle example to use dumps() instead of dump() in an unsullied Python2.5 I get a MemoryError. In general, I think you have to be careful using dumps(). Any attempt to solve that proble

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: Do we need an intermediate buffer at all when called from dumps()? How about allocating the buffer as a PyStringObject, so that it can be used directly for the result in that case? (IIRC there's a handy _PyString_Resize function) -- _

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: You can fix it if you are dumping to a file, however if you are calling dumps() you are kind of screwed if dumping large objects. There's no place to flush the buffer. I have a fix to Unladen Swallow's cPickle module. I'm run it by them before submitting i

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: Antoine> With the patch, the same command quickly swaps hopelessly and Antoine> after 5 minutes of elapsed time I finally manage to kill the Antoine> process. Verified with an Unladen Swallow test case. I'll see if I can fix it. S -- ___

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: Quick test on a 3GB machine: Without patch ("top" shows the process reaches 1.2GB RAM max): $ time ./python -c "import cPickle;l=['a'*1024 for i in xrange(100)];cPickle.dump(l, open('/dev/null', 'wb'))" 10.67user 1.47system 0:12.92elapsed 93%CPU (0avgtext

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: > Got a test case in mind? If so, I'll code it up and compare 2.6 and Unladen > Swallow as well as offer it up for inclusion in the U-S test suite. Trying to pickle a structure that's larger than half the RAM should do the trick. Something like a list of large

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: Antoine> The main thing I'm worried about is the potentially unbounded Antoine> buffering, since it could reduce performance (or even thrash Antoine> the machine) instead of improving it. Got a test case in mind? If so, I'll code it up and compare 2.6

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: > Indeed there are. Given that the Unladen Swallow folks were focusing on the > 2.6 branch and their goal was to improve performance I don't see any reason > to not accept what they've done, then tweak it for 2.7/3.1 assuming the > changes you and Alexandre sug

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: Antoine> There were a couple of comments on the Rietveld code review Antoine> above. Indeed there are. Given that the Unladen Swallow folks were focusing on the 2.6 branch and their goal was to improve performance I don't see any reason to not accept what

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: There were a couple of comments on the Rietveld code review above. -- ___ Python tracker ___ ___ Pyt

[issue5683] Speed up cPickle's pickling generally

Skip Montanaro added the comment: Updated the patch against the latest version of cPickle.c (r77393). All tests pass on my Mac. -- nosy: +skip.montanaro Added file: http://bugs.python.org/file15808/cPickle.-r77393.patch ___ Python tracker

[issue5683] Speed up cPickle's pickling generally

Alexandre Vassalotti added the comment: Last august, I worked on integrating Collin's optimization work into py3k in a local Mercurial branch. So, I can champion these changes into py3k, if Collin is unavailable. And if Collin allows me, I would like to merge the other pickle optimizations curr

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: Are you still willing to work on this? -- versions: +Python 3.2 -Python 3.1 ___ Python tracker ___ _

[issue5683] Speed up cPickle's pickling generally

Jerry Chen added the comment: Applied collinwinter's patch and svn up'd to r73872; replaced additional instances of self->write_func with _Pickler_Write. Passed test_xpickle.py as referenced by issue 5665. -- nosy: +jcsalterego Added file: http://bugs.python.org/file14466/cpickle_write

[issue5683] Speed up cPickle's pickling generally

Antoine Pitrou added the comment: Any updates? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.p

[issue5683] Speed up cPickle's pickling generally

Changes by Antoine Pitrou : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue5683] Speed up cPickle's pickling generally

New submission from Collin Winter : This patch simplifies cPickle's complicated internal buffering system. The new version uses a single buffer closer to the Pickler object, flushing to a file object only when necessary. This avoids the overhead of several indirection layers for what are frequent