Antoine Pitrou pitrou.net> writes:
> Does cPickle bytecode have some kind of NOP instruction?
> You could keep track of which PUTs weren't necessary and zero them out at the
> end. It would be much cheaper than writing a whole other "optimized" stream.
For a large file, I'm not sure it is much fa
Collin Winter google.com> writes:
> I don't think it's possible in general to remove any PUTs if the
> pickle is being written to a file-like object. It is possible to reuse
> a single Pickler to pickle multiple objects: this causes the Pickler's
> memo dict to be shared between the objects being
Alexandre Vassalotti peadrop.com> writes:
>
> On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin gmail.com>
> wrote:
> > This wouldn't help our use case, your code needs the entire pickle
> > stream to be in memory, which in our case would be about 475mb, this
Collin Winter google.com> writes:
> I should add that, adding the necessary bookkeeping to remove only
> unused PUTs (instead of the current all-or-nothing scheme) should not
> be hard. I'd watch out for a further performance/memory hit; the
> pickling benchmarks in the benchmark suite should help
Alexandre Vassalotti peadrop.com> writes:
> Just put your code on bugs.python.org and I will take a look.
>
Thanks, I'll put it in there.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscr
We were having performance problems unpickling a large pickle file, we were
getting 170s running time (which was fine), but 1100mb memory usage. Memory
usage ought to have been about 300mb, this was happening because of memory
fragmentation, due to many unnecessary "puts" in the pickle stream.
We
