Hi Santiago-- On Sat 2017-01-21 12:40:06 -0500, Santiago Vila wrote: > I have a cron job monitoring "Committed_AS" in /proc/meminfo every time > my autobuilders are running, that way I know how much memory each > package requires. > > Well, building gpgme1.0 allocates 100 GB, 200 GB and sometimes 500 GB.
yikes! I've never noticed that, but i can replicate it now that i'm looking for it. I'm a bit surprised a cronjob would have caught it, because i only see the effect for a few seconds at most. but it's definitely a significant spike in Committed_AS. > Then try to build this package in such machine, and it will fail. > > I have detected only one other package with a problem like this. > If you are curious: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848589 > > Every other package which I tried builds fine with only 20 GB of RAM, > so this is clearly an anomaly. I would have guessed that this happens in gpgme due to the C++ library build which is now part of libgpgme -- i know that g++ and the linker can both be pretty beastly. and snap-align is also C++. however, i did a bit more inspection, and i see these commitments only during the test suite. in particular, on a system with 8GiB of RAM, my RAM commitments moved from a normal level of: Committed_AS: 7360592 kB to: Committed_AS: 108890120 kB during the execution of tests/gpg/t-thread-keylist-verify. this test makes 200 threads, 100 of which do a gpg verification operation, and the other 100 do a keylist operation; that's when the RAM allocation spikes. Interestingly, just before that happens is tests/gpg/t-thread-keylist, which doesn't seem to allocate nearly as much RAM despite being of the same general form (it just does keylist without the simultaneous verify, though). Each thread does spawn an additional gpg process, so there's certainly memory committments there. But even with 200 concurrent processes, that amounts to 0.5GiB of RAM commitments per process, which seems large to me. I'm open for suggestions for how to address this. We could just drop THREAD_COUNT from 100 to 10 to reduce the RAM consumption, but that would mean minimizing some of the concurrency that the test suite is (rightly) trying to judge itself against. I'd rather not remove those kinds of checks if we don't need to. Any suggestions about what the right thing to do is? --dkg
signature.asc
Description: PGP signature
