Mike Coleman wrote:
Andrew, this is on an (intel) x86_64 box with 64GB of RAM. I don't
recall the maker or details of the architecture off the top of my
head, but it would be something "off the rack" from Dell or maybe HP.
There were other users on the box at the time, but nothing heavy or
that
4631 should be a release blocker. I'll have a bit of time on Monday
and Tuesday to wrap it up.
Jeremy
On Fri, Dec 19, 2008 at 5:28 PM, Barry Warsaw wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> I'd like to get Python 3.0.1 out before the end of the year. There are no
> showstopp
On Sat, Dec 20, 2008 at 5:02 PM, Arfrever Frehtes Taifersar Arahesis
wrote:
> 2008-12-20 23:46:15 Benjamin Peterson napisał(a):
>> On Sat, Dec 20, 2008 at 4:28 PM, Arfrever Frehtes Taifersar Arahesis
>> wrote:
>> > Python 2.6.1 documentation currently isn't available for download at:
>> > http://
Mike Coleman gmail.com> writes:
>
> The 45G was the VM size of the process (resident size was similar).
Can you reproduce it with a smaller working set? Something between 1 and 2GB,
possibly randomly-generated, and post both the generation script and the
problematic script on the bug tracker?
On Sat, Dec 20, 2008 at 5:40 PM, Alexandre Vassalotti
wrote:
> Could you give us more information about the dictionary. For example,
> how many objects does it contain? Is 45GB the actual size of the
> dictionary or of the Python process?
The 45G was the VM size of the process (resident size was
Mike Coleman gmail.com> writes:
>
> Just to clarify, I didn't gc.disable() to address this problem, but
> rather because it destroys performance during the creation of the huge
> dict. I don't have a specific number, but I think disabling gc
> reduced construction from something like 70 minutes
On Sat, Dec 20, 2008 at 4:11 PM, Tim Peters wrote:
> [Lots of answers]
Thanks. Wish I could have offered something useful.
--
Cheers,
Leif
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscrib
Re "held" and "intern_it": Haha! That's evil and extremely evil,
respectively. :-)
I will add these to the Python wiki if they're not already there...
Mike
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/pyth
Tim, I left out some details that I believe probably rule out the
"swapped out" theory. The machine in question has 64GB RAM, but only
16GB swap. I'd prefer more swap, but in any case only around ~400MB
of the swap was actually in use during my program's entire run.
Furthermore, during my program
Aahz wrote:
> On Sat, Dec 20, 2008, Nick Coghlan wrote:
>> It turns out that _PyBuiltin_Init doesn't call PyType_Ready on any of
>> the builtin types - they're left to have it called implicitly when an
>> operation using them needs tp_dict filled in.
>
> This seems like a release blocker for 3.0.1
s...@pobox.com wrote:
>
> Steve> Unfortunately there are doubtless programs out there that do rely
> Steve> on actions being taken at shutdown.
>
> Indeed. I believe any code which calls atexit.register.
>
> Steve> Maybe os.exit() could be more widely advertised, though ...
>
> Tha
Tim Peters wrote:
> If that is the case here, there's no evident general solution. If you
> have millions of objects still alive at exit, refcount-based
> reclamation has to visit all of them, and if they've been swapped out
> to disk it can take a very long time to swap them all back into memory
[Sorry, for the previous garbage post.]
> On Fri, Dec 19, 2008 at 6:29 PM, Mike Coleman wrote:
> I have a program that creates a huge (45GB) defaultdict. (The keys
> are short strings, the values are short lists of pairs (string, int).)
> Nothing but possibly the strings and ints is shared.
Cou
On Fri, Dec 19, 2008 at 6:29 PM, Mike Coleman wrote:
> I have a program that creates a huge (45GB) defaultdict. (The keys
> are short strings, the values are short lists of pairs (string, int).)
> Nothing but possibly the strings and ints is shared.
>
> That is, after executing the final stat
Steve Holden holdenweb.com> writes:
> I believe the OP engendered a certain amount of confusion by describing
> object deallocation as being performed by the garbage collector. So he
> perhaps didn't understand that even decref'ing all the objects only
> referenced by the dict will take a huge amo
On Sat, Dec 20, 2008 at 13:45, Fabio Zadrozny wrote:
> It appears that this bug was already reported:
> http://bugs.python.org/issue4705
>
> Any chance that it gets in the next 3.0.x bugfix release?
>
> Just as a note, if I do: sys.stdout._line_buffering = True, it also
> works, but doesn't seem
2008-12-20 23:46:15 Benjamin Peterson napisał(a):
> On Sat, Dec 20, 2008 at 4:28 PM, Arfrever Frehtes Taifersar Arahesis
> wrote:
> > Python 2.6.1 documentation currently isn't available for download at:
> > http://docs.python.org/ftp/python/doc/
>
> It is avaiable here, though:
>
> http://www.p
On Sat, Dec 20, 2008 at 4:37 PM, Steve Holden wrote:
> It might be helpful if
>
> http://www.python.org/download/releases/2.6.1/license/
>
> said it was also the official license for the 2.6.1 release (though I
> don't suppose it matters that it's still called the 2.5 license, since
> that's its
On Sat, Dec 20, 2008 at 4:28 PM, Arfrever Frehtes Taifersar Arahesis
wrote:
> Python 2.6.1 documentation currently isn't available for download at:
> http://docs.python.org/ftp/python/doc/
It is avaiable here, though:
http://www.python.org/ftp/python/doc/current/
>
> Additionally please include
Antoine Pitrou wrote:
> Leif Walsh gmail.com> writes:
>> It might be a semantic change that I'm looking for here, but it seems
>> to me that if you turn off the garbage collector, you should be able
>> to expect that either it also won't run on exit, or it should have a
>> way of letting you tell
It might be helpful if
http://www.python.org/download/releases/2.6.1/license/
said it was also the official license for the 2.6.1 release (though I
don't suppose it matters that it's still called the 2.5 license, since
that's its origin).
Another detail to go into the release manage PEP?
rega
Python 2.6.1 documentation currently isn't available for download at:
http://docs.python.org/ftp/python/doc/
Additionally please include version numbers in documentation
archives (e.g. python-docs-html-2.6.1.tar.bz2).
--
Arfrever Frehtes Taifersar Arahesis
signature.asc
Description: This is a
>> I will try next week to see if I can come up with a smaller,
>> submittable example. Thanks.
>
> These long exit times are usually caused by the garbage collection
> of objects. This can be a very time consuming task.
I doubt that. The long exit times are usually caused by a bad
malloc implem
On Sat, Dec 20, 2008 at 2:50 PM, M.-A. Lemburg wrote:
> If you want a really fast exit, try this:
>
> import os
> os.kill(os.getpid(), 9)
>
> But you better know what you're doing if you take this approach...
This would work, but I think os._exit(EX_OK) is probably just as fast,
and allows you to
> Any opinions?
I would use a different marshal implementation. Instead of defining
a stream format for marshal, make marshal dump its graph of objects
along with the actual memory layout. On load, copying can
be avoided; just a few pointers need to be updated. The resulting
marshal files would be
It appears that this bug was already reported: http://bugs.python.org/issue4705
Any chance that it gets in the next 3.0.x bugfix release?
Just as a note, if I do: sys.stdout._line_buffering = True, it also
works, but doesn't seem right as it's accessing an internal attribute.
Note 2: the solutio
Leif Walsh gmail.com> writes:
>
> It might be a semantic change that I'm looking for here, but it seems
> to me that if you turn off the garbage collector, you should be able
> to expect that either it also won't run on exit, or it should have a
> way of letting you tell it not to run on exit.
[
[Leif Walsh]
> ...
> It might be a semantic change that I'm looking for here, but it seems
> to me that if you turn off the garbage collector, you should be able
> to expect that either it also won't run on exit,
It won't then, but "the garbage collector" is the gc module, and that
only performs /
[Mike Coleman]
>> ... Regarding interning, I thought this only worked with strings.
Implementation details. Recent versions of CPython also, e.g.,
"intern" the empty tuple, and very small integers.
>> Is there some way to intern integers? I'm probably creating 300M
>> integers more or less unif
(@Skip, Michael, Tim)
On Sat, Dec 20, 2008 at 3:26 PM, wrote:
> Because useful side effects are sometimes performed as a result of this
> activity (flushing disk buffers, closing database connections, etc).
Of course they are. But what about the case given above:
On Sat, Dec 20, 2008 at 5:55
On 2008-12-20 21:20, Leif Walsh wrote:
> On Sat, Dec 20, 2008 at 3:04 PM, M.-A. Lemburg wrote:
>> These long exit times are usually caused by the garbage collection
>> of objects. This can be a very time consuming task.
>
> In that case, the question would be "why is the interpreter collecting
>
[M.-A. Lemburg]
>> These long exit times are usually caused by the garbage collection
>> of objects. This can be a very time consuming task.
[Leif Walsh]
> In that case, the question would be "why is the interpreter collecting
> garbage when it knows we're trying to exit anyway?".
Because user-de
Leif> In that case, the question would be "why is the interpreter
Leif> collecting garbage when it knows we're trying to exit anyway?".
Because useful side effects are sometimes performed as a result of this
activity (flushing disk buffers, closing database connections, etc).
Skip
__
Leif Walsh wrote:
On Sat, Dec 20, 2008 at 3:04 PM, M.-A. Lemburg wrote:
These long exit times are usually caused by the garbage collection
of objects. This can be a very time consuming task.
In that case, the question would be "why is the interpreter collecting
garbage when it knows w
On Sat, Dec 20, 2008 at 3:04 PM, M.-A. Lemburg wrote:
> These long exit times are usually caused by the garbage collection
> of objects. This can be a very time consuming task.
In that case, the question would be "why is the interpreter collecting
garbage when it knows we're trying to exit anyway
On 2008-12-20 17:57, Mike Coleman wrote:
> On Sat, Dec 20, 2008 at 4:02 AM, Kristján Valur Jónsson
> wrote:
>> Can you distill the program into something reproducible?
>> Maybe with something slightly less than 45Gb but still exhibiting some
>> degradation of exit performance?
>> I can try to poi
Steven D'Aprano pearwood.info> writes:
>
> In November 2007, a similar problem was reported on the comp.lang.python
> newsgroup. 370MB was large enough to demonstrate the problem. I don't
> know if a bug was ever reported.
Do you still reproduce it on trunk?
I've tried your scripts on my machi
Hi fellow snakemen and lizard ladies,
We have been recently done lots of Python work on Nokia Series 60 phones and
even managed to roll out some commercial Python based applications. In the
future we plan to create some iPhone Python apps also.
Python runs fine in phones - after it has been launc
You can always try poor-man's profiling, which is surprisingly useful in the
face of massive performance problems.
Just attach a debugger to the program, and when it suffering from a performance
problem, break the execution on a regular basis. You are statistically very
likely to get a callstack
Mike Coleman wrote:
... Regarding interning, I thought this only worked with strings.
Is there some way to intern integers? I'm probably creating 300M
integers more or less uniformly distributed across range(1)?
held = list(range(1))
...
troublesome_dict[string] = held[number_to_h
Andrew, this is on an (intel) x86_64 box with 64GB of RAM. I don't
recall the maker or details of the architecture off the top of my
head, but it would be something "off the rack" from Dell or maybe HP.
There were other users on the box at the time, but nothing heavy or
that gave me any reason to
On Sat, Dec 20, 2008 at 4:02 AM, Kristján Valur Jónsson
wrote:
> Can you distill the program into something reproducible?
> Maybe with something slightly less than 45Gb but still exhibiting some
> degradation of exit performance?
> I can try to point our commercial profiling tools at it and see w
On Sat, Dec 20, 2008, Nick Coghlan wrote:
>
> It turns out that _PyBuiltin_Init doesn't call PyType_Ready on any of
> the builtin types - they're left to have it called implicitly when an
> operation using them needs tp_dict filled in.
This seems like a release blocker for 3.0.1 to me
--
Aahz (a
Steve> Unfortunately there are doubtless programs out there that do rely
Steve> on actions being taken at shutdown.
Indeed. I believe any code which calls atexit.register.
Steve> Maybe os.exit() could be more widely advertised, though ...
That would be os._exit(). Calling it avoid
Barry Warsaw schrieb:
> I'd like to get Python 3.0.1 out before the end of the year. There
> are no showstoppers, but I haven't yet looked at the deferred blockers
> or the buildbots.
>
> Do you think we can get 3.0.1 out on December 24th? Or should we wait
> until after Christmas and get it out
Benjamin Peterson schrieb:
> On Fri, Dec 19, 2008 at 12:55 PM, Tarek Ziadé wrote:
>> Hello
>>
>> I would like to request a commit access to work specifically on
>> distutils maintenance.
>
> +1
>
> We are currently without an active distutils maintainer, and many
> stale distutil tickets are in
Andrew MacIntyre wrote:
> Mike Coleman wrote:
>> I have a program that creates a huge (45GB) defaultdict. (The keys
>> are short strings, the values are short lists of pairs (string, int).)
>> Nothing but possibly the strings and ints is shared.
>>
>> The program takes around 10 minutes to run, b
Mike Coleman wrote:
I have a program that creates a huge (45GB) defaultdict. (The keys
are short strings, the values are short lists of pairs (string, int).)
Nothing but possibly the strings and ints is shared.
The program takes around 10 minutes to run, but longer than 20 minutes
to exit (I g
On Sat, 20 Dec 2008 09:02:38 pm Kristján Valur Jónsson wrote:
> Can you distill the program into something reproducible?
> Maybe with something slightly less than 45Gb but still exhibiting
> some degradation of exit performance? I can try to point our
> commercial profiling tools at it and see wha
Can you distill the program into something reproducible?
Maybe with something slightly less than 45Gb but still exhibiting some
degradation of exit performance?
I can try to point our commercial profiling tools at it and see what it is
doing.
K
-Original Message-
From: python-dev-bounces
50 matches
Mail list logo
