Ok. Thank you everyone for your helpful answers. I understand that fieldValueCache is not used for resolving queries. Is there any cache that can help this basic scenario (a lot of different queries, on a small set of fields)? Does Lucene's FieldCache help (implicitly)? How can I use RAM to reduce I/O in this type of queries?
On Fri, Jan 18, 2013 at 4:09 PM, Tomás Fernández Löbbe < tomasflo...@gmail.com> wrote: > No, the fieldValueCache is not used for resolving queries. Only for > multi-token faceting and apparently for the stats component too. The > document cache maintains in memory the stored content of the fields you are > retrieving or highlighting on. It'll hit if the same document matches the > query multiple times and the same fields are requested, but as Eirck said, > it is important for cases when multiple components in the same request need > to access the same data. > > I think soft committing every 10 minutes is totally fine, but you should > hard commit more often if you are going to be using transaction log. > openSearcher=false will essentially tell Solr not to open a new searcher > after the (hard) commit, so you won't see the new indexed data and caches > wont be flushed. openSearcher=false makes sense when you are using > hard-commits together with soft-commits, as the "soft-commit" is dealing > with opening/closing searchers, you don't need hard commits to do it. > > Tomás > > > On Fri, Jan 18, 2013 at 2:20 AM, Isaac Hebsh <isaac.he...@gmail.com> > wrote: > > > Unfortunately, it seems ( > > http://lucene.472066.n3.nabble.com/Nrt-and-caching-td3993612.html) that > > these caches are not per-segment. In this case, I want to (soft) commit > > less frequently. Am I right? > > > > Tomás, as the fieldValueCache is very similar to lucene's FieldCache, I > > guess it has a big contribution to standard (not only faceted) queries > > time. SolrWiki claims that it primarily used by faceting. What that says > > about complex textual queries? > > > > documentCache: > > Erick, After a query processing is finished, doesn't some documents stay > in > > the documentCache? can't I use it to accelerate queries that should > > retrieve stored fields of documents? In this case, a big documentCache > can > > hold more documents.. > > > > About commit frequency: > > HardCommit: "openSearch=false" seems as a nice solution. Where can I read > > about this? (found nothing but one unexplained sentence in SolrWiki). > > SoftCommit: In my case, the required index freshness is 10 minutes. The > > plan to soft commit every 10 minutes is similar to storing all of the > > documents in a queue (outside to Solr), an indexing a bulk every 10 > > minutes. > > > > Thanks. > > > > > > On Fri, Jan 18, 2013 at 2:15 AM, Tomás Fernández Löbbe < > > tomasflo...@gmail.com> wrote: > > > > > I think fieldValueCache is not per segment, only fieldCache is. > However, > > > unless I'm missing something, this cache is only used for faceting on > > > multivalued fields > > > > > > > > > On Thu, Jan 17, 2013 at 8:58 PM, Erick Erickson < > erickerick...@gmail.com > > > >wrote: > > > > > > > filterCache: This is bounded by 1M * (maxDoc) / 8 * (num filters in > > > > cache). Notice the /8. This reflects the fact that the filters are > > > > represented by a bitset on the _internal_ Lucene ID. UniqueId has no > > > > bearing here whatsoever. This is, in a nutshell, why warming is > > > > required, the internal Lucene IDs may change. Note also that it's > > > > maxDoc, the internal arrays have "holes" for deleted documents. > > > > > > > > Note this is an _upper_ bound, if there are only a few docs that > > > > match, the size will be (num of matching docs) * sizeof(int)). > > > > > > > > fieldValueCache. I don't think so, although I'm a bit fuzzy on this. > > > > It depends on whether these are "per-segment" caches or not. Any "per > > > > segment" cache is still valid. > > > > > > > > Think of documentCache as intended to hold the stored fields while > > > > various components operate on it, thus avoiding repeatedly fetching > > > > the data from disk. It's _usually_ not too big a worry. > > > > > > > > About hard-commits once a day. That's _extremely_ long. Think instead > > > > of committing more frequently with openSearcher=false. If nothing > > > > else, you transaction log will grow lots and lots and lots. I'm > > > > thinking on the order of 15 minutes, or possibly even much less. With > > > > softCommits happening more often, maybe every 15 seconds. In fact, > I'd > > > > start out with soft commits every 15 seconds and hard commits > > > > (openSearcher=false) every 5 minutes. The problem with hard commits > > > > being once a day is that, if for any reason the server is > interrupted, > > > > on startup Solr will try to replay the entire transaction log to > > > > assure index integrity. Not to mention that your tlog will be huge. > > > > Not to mention that there is some memory usage for each document in > > > > the tlog. Hard commits roll over the tlog, flush the in-memory tlog > > > > pointers, close index segments, etc. > > > > > > > > Best > > > > Erick > > > > > > > > On Thu, Jan 17, 2013 at 1:29 PM, Isaac Hebsh <isaac.he...@gmail.com> > > > > wrote: > > > > > Hi, > > > > > > > > > > I am going to build a big Solr (4.0?) index, which holds some > dozens > > of > > > > > millions of documents. Each document has some dozens of fields, and > > one > > > > big > > > > > textual field. > > > > > The queries on the index are non-trivial, and a little-bit long > > (might > > > be > > > > > hundreds of terms). No query is identical to another. > > > > > > > > > > Now, I want to analyze the cache performance (before setting up the > > > whole > > > > > environment), in order to estimate how much RAM will I need. > > > > > > > > > > filterCache: > > > > > In my scenariom, every query has some filters. let's say that each > > > filter > > > > > matches 1M documents, out of 10M. Does the estimated memory usage > > > should > > > > be > > > > > 1M * sizeof(uniqueId) * num-of-filters-in-cache? > > > > > > > > > > fieldValueCache: > > > > > Due to the difference between queries, I guess that fieldValueCache > > is > > > > the > > > > > most important factor on query performance. Here comes a generic > > > > question: > > > > > I'm indexing new documents to the index constantly. Soft commits > will > > > be > > > > > performed every 10 mins. Does it say that the cache is meaningless, > > > after > > > > > every 10 minutes? > > > > > > > > > > documentCache: > > > > > enableLazyFieldLoading will be enabled, and "fl" contains a very > > small > > > > set > > > > > of fields. BUT, I need to return highlighting on about (possibly) > 20 > > > > > fields. Does the highlighting component use the documentCache? I > > guess > > > > that > > > > > highlighting requires the whole field to be loaded into the > > > > documentCache. > > > > > Will it happen only for fields that matched a term from the query? > > > > > > > > > > And one more question: I'm planning to hard-commit once a day. > > Should I > > > > > prepare to a significant RAM usage growth between hard-commits? > > > > (consider a > > > > > lot of new documents in this period...) > > > > > Does this RAM comes from the same pool as the caches? An > OutOfMemory > > > > > exception can happen is this scenario? > > > > > > > > > > Thanks a lot. > > > > > > > > > >
