[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-19 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 Andi Kleen changed: What|Removed |Added Status|UNCONFIRMED |RESOLVED Resolution|

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-17 Thread ak at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #15 from ak at gcc dot gnu.org 2011-10-17 14:43:45 UTC --- Author: ak Date: Mon Oct 17 14:43:37 2011 New Revision: 180093 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=180093 Log: Use MADV_DONTNEED for freeing in garbage collec

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #14 from Andi Kleen 2011-10-08 21:10:13 UTC --- Thanks for the review. Fixed the accounting I'll leave the xmalloc_failed hook out for now: it would need a retry path which is somewhat complicated. If it's needed would probably just

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #13 from Jakub Jelinek 2011-10-08 20:14:14 UTC --- In the 0001-initial-madvise.patch patch I think if you subtract the size MADV_DONTNEEDed from G.bytes_mapped, then you should add it again in alloc_page (i.e. replace + p->unmapp

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 Andi Kleen changed: What|Removed |Added Attachment #25445|0 |1 is obsolete|

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-08 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #11 from Andi Kleen 2011-10-08 16:47:54 UTC --- Created attachment 25445 --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=25445 patchkit I tested this patchkit which implements most of the ideas from this bug, unfortunately still

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-07 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #10 from Andi Kleen 2011-10-07 14:44:10 UTC --- To track the pattern you can simply use strace or ftrace (I did ftrace) I checked the kernel code now and if the madvise is big enough it won't split up the 2MB page. So doing it aggres

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-07 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #9 from Jakub Jelinek 2011-10-07 10:01:32 UTC --- (In reply to comment #2) > Perhaps on Linux, at least with recentish kernel, we could change > release_pages > into keeping the pages in the G.free_pages chain, but call madvise > MAD

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-07 Thread rguenth at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #8 from Richard Guenther 2011-10-07 08:25:37 UTC --- (In reply to comment #4) > Right now alloc_page will (usually) allocate in GGC_QUIRE_SIZE * G.pagesize > chunks, but release_pages will munmap individual pages immediately during >

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #7 from Andi Kleen 2011-10-07 05:50:40 UTC --- *** Bug 50511 has been marked as a duplicate of this bug. ***

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #6 from Andi Kleen 2011-10-07 05:47:54 UTC --- *** Bug 50302 has been marked as a duplicate of this bug. ***

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #5 from Andi Kleen 2011-10-06 21:46:32 UTC --- If it's a 2MB page then madvise MADV_DONTNEED will split it if it's not 2MB aligned. It would be good to optimize the freeing pattern so that this happens rarely. I will try to do some n

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #4 from Jakub Jelinek 2011-10-06 21:41:41 UTC --- Right now alloc_page will (usually) allocate in GGC_QUIRE_SIZE * G.pagesize chunks, but release_pages will munmap individual pages immediately during collection (well, using one munmap

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread andi-gcc at firstfloor dot org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #3 from Andi Kleen 2011-10-06 21:31:56 UTC --- I would prefer to free in 2MB chunks if possible I was experimenting with increasing the quire size from 1 to 2MB so that a modern kernel with transparent huge pages can always get a hug

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread jakub at gcc dot gnu.org
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 Jakub Jelinek changed: What|Removed |Added CC||jakub at gcc dot gnu.org --- Comment #2 f

[Bug other/50636] GC in large LTO builds cause excessive fragmentation in memory map

2011-10-06 Thread hubicka at ucw dot cz
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50636 --- Comment #1 from Jan Hubicka 2011-10-06 19:20:45 UTC --- > When doing a very large LTO build I fail with "out of virtual memory" > > Some investigation showed the problem was not actually running out of > memory, but gcc excessively fragment