Dan Gindikin gmail.com> writes:
>
> Antoine Pitrou pitrou.net> writes:
> > Does cPickle bytecode have some kind of NOP instruction?
> > You could keep track of which PUTs weren't necessary and zero them out at
> > the
> > end. It would be much cheaper than writing a whole other "optimized" stre
Antoine Pitrou pitrou.net> writes:
> Does cPickle bytecode have some kind of NOP instruction?
> You could keep track of which PUTs weren't necessary and zero them out at the
> end. It would be much cheaper than writing a whole other "optimized" stream.
For a large file, I'm not sure it is much fa
Collin Winter google.com> writes:
> I don't think it's possible in general to remove any PUTs if the
> pickle is being written to a file-like object. It is possible to reuse
> a single Pickler to pickle multiple objects: this causes the Pickler's
> memo dict to be shared between the objects being
Collin Winter google.com> writes:
>
> I don't think it's possible in general to remove any PUTs if the
> pickle is being written to a file-like object.
Does cPickle bytecode have some kind of NOP instruction?
You could keep track of which PUTs weren't necessary and zero them out at the
end. It w
On Fri, Apr 23, 2010 at 1:53 PM, Alexandre Vassalotti
wrote:
> On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin wrote:
>> This wouldn't help our use case, your code needs the entire pickle
>> stream to be in memory, which in our case would be about 475mb, this
>> is on top of the 300mb+ data structu
Alexandre Vassalotti peadrop.com> writes:
>
> On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin gmail.com>
> wrote:
> > This wouldn't help our use case, your code needs the entire pickle
> > stream to be in memory, which in our case would be about 475mb, this
> > is on top of the 300mb+ data struc
On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin wrote:
> This wouldn't help our use case, your code needs the entire pickle
> stream to be in memory, which in our case would be about 475mb, this
> is on top of the 300mb+ data structures that generated the pickle
> stream.
>
In that case, the best w
Collin Winter google.com> writes:
> I should add that, adding the necessary bookkeeping to remove only
> unused PUTs (instead of the current all-or-nothing scheme) should not
> be hard. I'd watch out for a further performance/memory hit; the
> pickling benchmarks in the benchmark suite should help
On Fri, Apr 23, 2010 at 3:07 PM, Collin Winter wrote:
> I should add that, adding the necessary bookkeeping to remove only
> unused PUTs (instead of the current all-or-nothing scheme) should not
> be hard. I'd watch out for a further performance/memory hit; the
> pickling benchmarks in the benchma
On Fri, Apr 23, 2010 at 11:53 AM, Collin Winter wrote:
> On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti
> wrote:
>> On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti
>> wrote:
>>> Collin Winter wrote a simple optimization pass for cPickle in Unladen
>>> Swallow [1]. The code reads th
On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti
wrote:
> On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti
> wrote:
>> Collin Winter wrote a simple optimization pass for cPickle in Unladen
>> Swallow [1]. The code reads through the stream and remove all the
>> unnecessary PUTs in-place
Alexandre Vassalotti peadrop.com> writes:
> Just put your code on bugs.python.org and I will take a look.
>
Thanks, I'll put it in there.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscr
On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti
wrote:
> Collin Winter wrote a simple optimization pass for cPickle in Unladen
> Swallow [1]. The code reads through the stream and remove all the
> unnecessary PUTs in-place.
>
I just noticed the code removes *all* PUT opcodes, regardless if
On Fri, Apr 23, 2010 at 2:11 PM, Dan Gindikin wrote:
> We were having performance problems unpickling a large pickle file, we were
> getting 170s running time (which was fine), but 1100mb memory usage. Memory
> usage ought to have been about 300mb, this was happening because of memory
> fragmentat
On Fri, Apr 23, 2010 at 11:11, Dan Gindikin wrote:
> We were having performance problems unpickling a large pickle file, we were
> getting 170s running time (which was fine), but 1100mb memory usage. Memory
> usage ought to have been about 300mb, this was happening because of memory
> fragmentati
15 matches
Mail list logo
