On Wed, Oct 6, 2010 at 9:57 PM, Burton-West, Tom <tburt...@umich.edu> wrote:
> Hi Mike,
>
>>.Do you use multiple threads for indexing? Large RAM buffer size is
>>>also good, but I think perf peaks out mabye around 512 MB (at least
>>>based on past tests)?
>
> We are using Solr, I'm not sure if Solr uses multiple threads for indexing.
> We have 30 "producers" each sending documents to 1 of 12 Solr shards on a
> round robin basis. So each shard will get multiple requests.
OK, I believe if your clients are sending multiple requests to Solr,
that it will in fact use multiple threads for adding the docs.
But, do you periodically call commit? Because I believe Solr still
forces a close/open of the IW for commit (which is a holdover from
long ago when Lucene didn't have a .commit method), so that can really
slow down your indexing eg if there's a large running merge...
>>>Believe it or not, merging is typically compute bound. It's costly to
>>>decode & re-encode all the vInts.
>
> Sounds like we need to do some monitoring during merging to see what the cpu
> use is and also the io wait during large merges.
And please post back!
>>>Larger merge factor is good because it means the postings are copied
>>>fewer times, but, it's bad beacuse you could risk running out of
>>>descriptors, and, if the OS doesn't have enough RAM, you'll start to
>>>thin out the readahead that the OS can do (which makes the merge less
>>>efficient since the disk heads are seeking more).
>
> Is there a way to estimate the amount of RAM for the readahead? Once we
> start the re-indexing we will be running 12 shards on a 16 processor box with
> 144 GB of memory.
The "worst" part of merging is merging the postings, when we read from
3 (.tis, .frq, .prx) files simultaneously across all of the segments
being merged. So eg if your MF is 10, then this means the OS has to
buffer readahead for 30 files (competing with other was the OS uses
memory, eg buffer cache for say searches that are running, virtual
memory for processes, etc.).
I'm sure it's rather OS specific, how "well" the OS will read ahead in
this case. You could test, instead, forcing this readahead in lucene,
by changing the private static constant MERGE_READ_BUFFER_SIZE in
IndexWriter eg to something largish (4 MB? 8 MB?). A large enough
readahead (either by the OS "being smart" or by Lucene forcing it)
reduces the amortized disk head seek cost, making the merge more CPU
bound and less IO bound.
Just speculating...: it could also be possible that disabling CFS
improves the OS's readahead opto, since we are then truly opening a
file and doing a single read from start to finish.
>>>Do you do any deleting?
> Deletes would happen as a byproduct of updating a record. This shouldn't
> happen too frequently during re-indexing, but we update records when a
> document gets re-scanned and re-OCR'd. This would probably amount to a few
> thousand.
OK that's interesting.
I asked because there is a potentially powerful merge opto that we
could implement (haven't yet) which is to bulk-copy the postings data
for a single term, in the case where there are no (or, not many)
deletions. This should be a very large speedup for merging (and then
likely merging would in fact become IO bound).
>>>Do you use stored fields and/or term vectors? If so, try to make
>>>your docs "uniform" if possible, ie add the same fields in the same
>>>order. This enables lucene to use bulk byte copy merging under the hood.
>
> We use 4 or 5 stored fields. They are very small compared to our huge OCR
> field. Since we construct our Solr documents programattically, I'm fairly
> certain that they are always in the same order. I'll have to look at the
> code when I get back to make sure.
OK but that "or" makes me nervous ;) Ie, for the 4 case, if you still
add that 5th field, say with the empty string as its value, in the
precise same order that they are added in the 5 fields case, then
you'll guarantee the bulk copy opto applies during merge.
And btw it's absurd that apps need to take such steps to ensure this
brittle opto "applies". Really Lucene should be more deterministic in
how it assigns field names to numbers:
https://issues.apache.org/jira/browse/LUCENE-1737
Once we do that opto (which is sorta tricky because Lucene assumes in
various places now that field name -> number mapping is "dense") then
you no longer have to resort to such silliness.
> We aren't using term vectors now, but we plan to add them as well as a number
> of fields based on MARC (cataloging) metadata in the future.
OK same caveat (on field order) applies...
Mike
