Re: [Python-Dev] And the winner is...

2009-03-31 Thread Mike Coleman
On Tue, Mar 31, 2009 at 12:31 AM, Stephen J. Turnbull wrote: > I also just wrote a long post about the comparison of bzr to hg > responding to a comment on baz...@canonical.com.  I won't recap it > here but it might be of interest. I found the post interesting. Here's a link to the start of the

Re: [Python-Dev] And the winner is...

2009-03-31 Thread Mike Coleman
On Mon, Mar 30, 2009 at 9:54 PM, Guido van Rossum wrote: > Yeah, I also think I'll just stop developing Python now and suggest > that you all switch to Java, which has clearly won the mindshare war > for languages. :-) Heh. :-) Guess I should have said "mindshare among people whose technical op

Re: [Python-Dev] And the winner is...

2009-03-30 Thread Mike Coleman
Just for curiosity's sake, could someone outline the five (or so) most significant pluses of hg relative to git? (My personal feeling is that any of the three is a huge improvement over subversion. I think git probably should have been written in Python with some stuff in C where necessary, and

Re: [Python-Dev] error in doc for fcntl module

2009-01-08 Thread Mike Coleman
7;re doing anyways; the examples are meant to briefly show the call > format. > > On Wed, Jan 7, 2009 at 1:31 PM, Mike Coleman wrote: >> In the doc page for the fcntl module, the example below is given. >> This seems like an error, or at least very misleading, as the normal >>

[Python-Dev] error in doc for fcntl module

2009-01-07 Thread Mike Coleman
In the doc page for the fcntl module, the example below is given. This seems like an error, or at least very misleading, as the normal usage is to get the flags (F_GETFL), set or unset the bits you want to change, then set the flags (F_SETFL). A reader might think that the example below merely set

Re: [Python-Dev] suggest change to "Failed to find the necessary bits to build these modules" message

2008-12-23 Thread Mike Coleman
Done: http://bugs.python.org/issue4731 On Tue, Dec 23, 2008 at 12:13 PM, Brett Cannon wrote: > On Tue, Dec 23, 2008 at 09:59, Mike Coleman wrote: >> I was thrown by the "Failed to find the necessary bits to build these >> modules" message at the end of newer Python

[Python-Dev] suggest change to "Failed to find the necessary bits to build these modules" message

2008-12-23 Thread Mike Coleman
I was thrown by the "Failed to find the necessary bits to build these modules" message at the end of newer Python builds, and thought that this indicated that the Python executable itself was not built. That's arguably stupidity on my part, but I wonder if others will not trip on this, too. Would

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-23 Thread Mike Coleman
On Sat, Dec 20, 2008 at 6:22 PM, Mike Coleman wrote: > Re "held" and "intern_it": Haha! That's evil and extremely evil, > respectively. :-) P.S. I tried the "held" idea out (interning integers in a list), and unfortunately it didn't make that m

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
I unfortunately don't have time to work out how obmalloc works myself, but I wonder if any of the constants in that file might need to scale somehow with memory size. That is, is it possible that some of them that work okay with 1G RAM won't work well with (say) 128G or 1024G (coming soon enough)?

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
2008/12/22 Ivan Krstić : > On Dec 22, 2008, at 6:28 PM, Mike Coleman wrote: >> >> For (2), yes, 100% CPU usage. > > 100% _user_ CPU usage? (I'm trying to make sure we're not chasing some > particular degeneration of kmalloc/vmalloc and friends.) Yes, user. No n

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
On Mon, Dec 22, 2008 at 2:22 PM, Adam Olsen wrote: > To make sure that's the correct line please recompile python without > optimizations. GCC happily reorders and merges different parts of a > function. > > Adding a counter in C and recompiling would be a lot faster than using > a gdb hook. Oka

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
On Mon, Dec 22, 2008 at 2:54 PM, Ivan Krstić wrote: > It's still not clear to me, from reading the whole thread, precisely what > you're seeing. A self-contained test case, preferably with generated random > data, would be great, and save everyone a lot of investigation time. I'm still working on

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
On Mon, Dec 22, 2008 at 2:38 PM, "Martin v. Löwis" wrote: >> Or perhaps there's a smarter way to manage the list of >> arena/free pool info. > > If that code is the real problem (in a reproducible test case), > then this approach is the only acceptable solution. Disabling > long-running code is no

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
On Mon, Dec 22, 2008 at 6:20 AM, M.-A. Lemburg wrote: > BTW: Rather than using a huge in-memory dict, I'd suggest to either > use an on-disk dictionary such as the ones found in mxBeeBase or > a database. I really want this to work in-memory. I have 64G RAM, and I'm only trying to use 45G of it

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-22 Thread Mike Coleman
_Main (argc=, argv=0x7af691a8) at Modules/main.c:523 #12 0x0030fea1d8b4 in __libc_start_main () from /lib64/libc.so.6 #13 0x00411799 in _start () On Sun, Dec 21, 2008 at 12:44 PM, Adam Olsen wrote: > On Sat, Dec 20, 2008 at 6:09 PM, Mike Coleman wrote: >> O

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-20 Thread Mike Coleman
On Sat, Dec 20, 2008 at 5:40 PM, Alexandre Vassalotti wrote: > Could you give us more information about the dictionary. For example, > how many objects does it contain? Is 45GB the actual size of the > dictionary or of the Python process? The 45G was the VM size of the process (resident size was

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-20 Thread Mike Coleman
Re "held" and "intern_it": Haha! That's evil and extremely evil, respectively. :-) I will add these to the Python wiki if they're not already there... Mike ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/pyth

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-20 Thread Mike Coleman
Tim, I left out some details that I believe probably rule out the "swapped out" theory. The machine in question has 64GB RAM, but only 16GB swap. I'd prefer more swap, but in any case only around ~400MB of the swap was actually in use during my program's entire run. Furthermore, during my program

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-20 Thread Mike Coleman
On Sat, Dec 20, 2008 at 2:50 PM, M.-A. Lemburg wrote: > If you want a really fast exit, try this: > > import os > os.kill(os.getpid(), 9) > > But you better know what you're doing if you take this approach... This would work, but I think os._exit(EX_OK) is probably just as fast, and allows you to

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-20 Thread Mike Coleman
ught this only worked with strings. Is there some way to intern integers? I'm probably creating 300M integers more or less uniformly distributed across range(1). Mike On Sat, Dec 20, 2008 at 4:08 AM, Andrew MacIntyre wrote: > Mike Coleman wrote: >> >> I have a progra

Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-20 Thread Mike Coleman
On Sat, Dec 20, 2008 at 4:02 AM, Kristján Valur Jónsson wrote: > Can you distill the program into something reproducible? > Maybe with something slightly less than 45Gb but still exhibiting some > degradation of exit performance? > I can try to point our commercial profiling tools at it and see w

[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

2008-12-19 Thread Mike Coleman
I have a program that creates a huge (45GB) defaultdict. (The keys are short strings, the values are short lists of pairs (string, int).) Nothing but possibly the strings and ints is shared. The program takes around 10 minutes to run, but longer than 20 minutes to exit (I gave up at that point).