On Sun, Feb 13, 2011 at 12:44:52PM +0000, Enrico Zini wrote:
> Alternatively, we can of course tweak XAPIAN_FLUSH_THRESHOLD to be
> proportional to the available memory, with an upper bound. I do not have
> a suitable system, however, to use for testing the tweak. Or better, I
> do have a NSLU2 I could use, for example, but I have no idea what kind
> of level of performance I should consider tolerable and what not.
I did some simple testing on my laptop. It has 6GB of RAM, but the RAM
used measurements should still be relevant. In fact, an excess of RAM
means that the resident sizes below should be natural, rather than due to
VM pressure actively swapping out pages.
The laptop has wheezy, and was updated fairly recently.
I ran this command (with the sudo password cached), with the python script
as shipped, and also patched with the attached patch:
time sudo update-apt-xapian-index -f
I reran and ignored the initial numbers, so these are warm cache cases
(otherwise the second variant to run get the advantage of a warm cache).
I watched the command in top and noted the peak resident size.
For the script as shipped, the memory usage peaked at 228M and it took
about 35 seconds.
Patched, the memory usage peaked at 152M and it took 1m15 (just over
twice as long).
If I patch it to set the threshold to 100, then it's 145M and takes
much longer (8m22, but I didn't have the patience to retry it). So
it doesn't seem useful to go much lower than 1000, as you save little
memory, but take much longer.
I think it would be good to do something about this for wheezy. My
suggestion would be to apply the patch, but probably tweaked to not reduce
the threshold if the system has "lots" of memory. And if there's less
than a certain amount (256MB perhaps) then disable apt-xapian-index.
I'm happy to experiment with tuning the threshold more if you want to go
this route. 1000 seems in the right area, but I didn't try anything
closer than 100 or 10000 (the default).
Thoughts?
Cheers,
Olly
--- /usr/sbin/update-apt-xapian-index 2011-12-29 02:45:19.000000000 +1300
+++ update-apt-xapian-index 2012-06-01 14:22:08.000000000 +1200
@@ -30,6 +30,9 @@
from optparse import OptionParser
import sys
import warnings
+import os
+
+os.environ['XAPIAN_FLUSH_THRESHOLD'] = '100'
VERSION="0.45"
