marshal.dumps quadratic growth and marshal.dump not allowing file-like objects
I'm stuck on a problem where I want to use marshal for serialization (yes, yes, I know (c)Pickle is normally recommended here). I favor marshal for speed for the types of data I use. However it seems that marshal.dumps() for large objects has a quadratic performance issue which I'm assuming is that it grows its memory buffer in constant increments. This causes a nasty slowdown for marshaling large objects. I thought I would get around this by passing a cStringIO.StringIO object to marshal.dump() instead but I quickly learned this is not supported (only true file objects are supported). Any ideas about how to get around the marshal quadratic issue? Any hope for a fix for that on the horizon? Thanks for any information. -- http://mail.python.org/mailman/listinfo/python-list
Re: marshal.dumps quadratic growth and marshal.dump not allowing file-like objects
On Jun 15, 3:16 am, John Machin <[EMAIL PROTECTED]> wrote: > But that change went into the svn trunk on 11-May-2008; perhaps the OP > is using a production release which would have the previous version, > which is merely "newsize = size + 1024;". > > Do people really generate 32MB pyc files, or is stopping doubling at > 32MB just a safety valve in case someone/something runs amok? Indeed. I (the OP) am using a production release which has the 1k linear growth. I am seeing the problems with ~5MB and ~10MB sizes. Apparently this will be improved greatly in Python 2.6, at least up to the 32MB limit. Thanks all for responding. -- http://mail.python.org/mailman/listinfo/python-list
