Hmm, interesting. I'll have to look closer...
On Sun, Jan 20, 2013 at 3:50 PM, Walter Underwood <wun...@wunderwood.org> wrote: > I routinely see hit rates over 75% on the document cache. Perhaps yours is > too small. Mine is set at 10240 entries. > > wunder > > On Jan 20, 2013, at 8:08 AM, Erick Erickson wrote: > >> About your question about document cache: Typically the document cache >> has a pretty low hit-ratio. I've rarely, if ever, seen it get hit very >> often. And remember that this cache is only hit when assembling the >> response for a few documents (your page size). >> >> Bottom line: I wouldn't worry about this cache much. It's quite useful >> for processing a particular query faster, but not really intended for >> cross-query use. >> >> Really, I think you're getting the cart before the horse here. Run it >> up the flagpole and try it. Rely on the OS to do its job >> (http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html). >> Find a bottleneck _then_ tune. Premature optimization and all >> that.... >> >> Several tens of millions of docs isn't that large unless the text >> fields are enormous. >> >> Best >> Erick >> >> On Sat, Jan 19, 2013 at 2:32 PM, Isaac Hebsh <isaac.he...@gmail.com> wrote: >>> Ok. Thank you everyone for your helpful answers. >>> I understand that fieldValueCache is not used for resolving queries. >>> Is there any cache that can help this basic scenario (a lot of different >>> queries, on a small set of fields)? >>> Does Lucene's FieldCache help (implicitly)? >>> How can I use RAM to reduce I/O in this type of queries? >>> >>> >>> On Fri, Jan 18, 2013 at 4:09 PM, Tomás Fernández Löbbe < >>> tomasflo...@gmail.com> wrote: >>> >>>> No, the fieldValueCache is not used for resolving queries. Only for >>>> multi-token faceting and apparently for the stats component too. The >>>> document cache maintains in memory the stored content of the fields you are >>>> retrieving or highlighting on. It'll hit if the same document matches the >>>> query multiple times and the same fields are requested, but as Eirck said, >>>> it is important for cases when multiple components in the same request need >>>> to access the same data. >>>> >>>> I think soft committing every 10 minutes is totally fine, but you should >>>> hard commit more often if you are going to be using transaction log. >>>> openSearcher=false will essentially tell Solr not to open a new searcher >>>> after the (hard) commit, so you won't see the new indexed data and caches >>>> wont be flushed. openSearcher=false makes sense when you are using >>>> hard-commits together with soft-commits, as the "soft-commit" is dealing >>>> with opening/closing searchers, you don't need hard commits to do it. >>>> >>>> Tomás >>>> >>>> >>>> On Fri, Jan 18, 2013 at 2:20 AM, Isaac Hebsh <isaac.he...@gmail.com> >>>> wrote: >>>> >>>>> Unfortunately, it seems ( >>>>> http://lucene.472066.n3.nabble.com/Nrt-and-caching-td3993612.html) that >>>>> these caches are not per-segment. In this case, I want to (soft) commit >>>>> less frequently. Am I right? >>>>> >>>>> Tomás, as the fieldValueCache is very similar to lucene's FieldCache, I >>>>> guess it has a big contribution to standard (not only faceted) queries >>>>> time. SolrWiki claims that it primarily used by faceting. What that says >>>>> about complex textual queries? >>>>> >>>>> documentCache: >>>>> Erick, After a query processing is finished, doesn't some documents stay >>>> in >>>>> the documentCache? can't I use it to accelerate queries that should >>>>> retrieve stored fields of documents? In this case, a big documentCache >>>> can >>>>> hold more documents.. >>>>> >>>>> About commit frequency: >>>>> HardCommit: "openSearch=false" seems as a nice solution. Where can I read >>>>> about this? (found nothing but one unexplained sentence in SolrWiki). >>>>> SoftCommit: In my case, the required index freshness is 10 minutes. The >>>>> plan to soft commit every 10 minutes is similar to storing all of the >>>>> documents in a queue (outside to Solr), an indexing a bulk every 10 >>>>> minutes. >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> On Fri, Jan 18, 2013 at 2:15 AM, Tomás Fernández Löbbe < >>>>> tomasflo...@gmail.com> wrote: >>>>> >>>>>> I think fieldValueCache is not per segment, only fieldCache is. >>>> However, >>>>>> unless I'm missing something, this cache is only used for faceting on >>>>>> multivalued fields >>>>>> >>>>>> >>>>>> On Thu, Jan 17, 2013 at 8:58 PM, Erick Erickson < >>>> erickerick...@gmail.com >>>>>>> wrote: >>>>>> >>>>>>> filterCache: This is bounded by 1M * (maxDoc) / 8 * (num filters in >>>>>>> cache). Notice the /8. This reflects the fact that the filters are >>>>>>> represented by a bitset on the _internal_ Lucene ID. UniqueId has no >>>>>>> bearing here whatsoever. This is, in a nutshell, why warming is >>>>>>> required, the internal Lucene IDs may change. Note also that it's >>>>>>> maxDoc, the internal arrays have "holes" for deleted documents. >>>>>>> >>>>>>> Note this is an _upper_ bound, if there are only a few docs that >>>>>>> match, the size will be (num of matching docs) * sizeof(int)). >>>>>>> >>>>>>> fieldValueCache. I don't think so, although I'm a bit fuzzy on this. >>>>>>> It depends on whether these are "per-segment" caches or not. Any "per >>>>>>> segment" cache is still valid. >>>>>>> >>>>>>> Think of documentCache as intended to hold the stored fields while >>>>>>> various components operate on it, thus avoiding repeatedly fetching >>>>>>> the data from disk. It's _usually_ not too big a worry. >>>>>>> >>>>>>> About hard-commits once a day. That's _extremely_ long. Think instead >>>>>>> of committing more frequently with openSearcher=false. If nothing >>>>>>> else, you transaction log will grow lots and lots and lots. I'm >>>>>>> thinking on the order of 15 minutes, or possibly even much less. With >>>>>>> softCommits happening more often, maybe every 15 seconds. In fact, >>>> I'd >>>>>>> start out with soft commits every 15 seconds and hard commits >>>>>>> (openSearcher=false) every 5 minutes. The problem with hard commits >>>>>>> being once a day is that, if for any reason the server is >>>> interrupted, >>>>>>> on startup Solr will try to replay the entire transaction log to >>>>>>> assure index integrity. Not to mention that your tlog will be huge. >>>>>>> Not to mention that there is some memory usage for each document in >>>>>>> the tlog. Hard commits roll over the tlog, flush the in-memory tlog >>>>>>> pointers, close index segments, etc. >>>>>>> >>>>>>> Best >>>>>>> Erick >>>>>>> >>>>>>> On Thu, Jan 17, 2013 at 1:29 PM, Isaac Hebsh <isaac.he...@gmail.com> >>>>>>> wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am going to build a big Solr (4.0?) index, which holds some >>>> dozens >>>>> of >>>>>>>> millions of documents. Each document has some dozens of fields, and >>>>> one >>>>>>> big >>>>>>>> textual field. >>>>>>>> The queries on the index are non-trivial, and a little-bit long >>>>> (might >>>>>> be >>>>>>>> hundreds of terms). No query is identical to another. >>>>>>>> >>>>>>>> Now, I want to analyze the cache performance (before setting up the >>>>>> whole >>>>>>>> environment), in order to estimate how much RAM will I need. >>>>>>>> >>>>>>>> filterCache: >>>>>>>> In my scenariom, every query has some filters. let's say that each >>>>>> filter >>>>>>>> matches 1M documents, out of 10M. Does the estimated memory usage >>>>>> should >>>>>>> be >>>>>>>> 1M * sizeof(uniqueId) * num-of-filters-in-cache? >>>>>>>> >>>>>>>> fieldValueCache: >>>>>>>> Due to the difference between queries, I guess that fieldValueCache >>>>> is >>>>>>> the >>>>>>>> most important factor on query performance. Here comes a generic >>>>>>> question: >>>>>>>> I'm indexing new documents to the index constantly. Soft commits >>>> will >>>>>> be >>>>>>>> performed every 10 mins. Does it say that the cache is meaningless, >>>>>> after >>>>>>>> every 10 minutes? >>>>>>>> >>>>>>>> documentCache: >>>>>>>> enableLazyFieldLoading will be enabled, and "fl" contains a very >>>>> small >>>>>>> set >>>>>>>> of fields. BUT, I need to return highlighting on about (possibly) >>>> 20 >>>>>>>> fields. Does the highlighting component use the documentCache? I >>>>> guess >>>>>>> that >>>>>>>> highlighting requires the whole field to be loaded into the >>>>>>> documentCache. >>>>>>>> Will it happen only for fields that matched a term from the query? >>>>>>>> >>>>>>>> And one more question: I'm planning to hard-commit once a day. >>>>> Should I >>>>>>>> prepare to a significant RAM usage growth between hard-commits? >>>>>>> (consider a >>>>>>>> lot of new documents in this period...) >>>>>>>> Does this RAM comes from the same pool as the caches? An >>>> OutOfMemory >>>>>>>> exception can happen is this scenario? >>>>>>>> >>>>>>>> Thanks a lot. >>>>>>> >>>>>> >>>>> >>>> > > -- > Walter Underwood > wun...@wunderwood.org > > >
