My 2 cents (rounded): Quote: "the size of our index data is more than 30GB every year now”
- is it the size of *data* or the size of *index*? This is super important! You can have petabytes of data, growing terabytes a year, and your index files will grow only few gigabytes a year at most. Note also that Lucene index files are immutable: it means that, for example, if your index files total size is 25Gb in a filesystem, then having at least 25Gb+2Gb of free RAM available (for index files + for OS) will be beneficial (as already mentioned in this thread). However, caching of index files in a RAM won’t reduce search performance from minutes of response time to milliseconds. If you really have timeouts (and I believe you use at least 60 seconds timeout settings for SolrJ) then possible reasons could be: 1. “Shared VM” such as Amazon shared nodes, sometimes they just stop for few minutes 2. Garbage collection in Java 3. Sophisticated Solr query such as faceting and aggregations, with inadequately configured field cache and other caches Having 100Gb index files in a filesystem cannot cause more than a few milliseconds response times for trivial queries such as “text:Solr”! (Exception: faceting) You need to isolate (troubleshoot) your timeouts, and you mentioned it only happens during new queries to the new searcher after replication from Master to Slave. Which means Case #3: improperly configured cache parameters. You need warm-up query. New Solr searcher will become available after internal caches warmed up (prepopulated with data). Memory estimate example: suppose you configured Solr such a way that it will use field cache for SKU field. Suppose SKU field is 64 bytes in average (UTF8 will take 2 bytes per character), and you have 100 millions of documents. Then you will need 6,400,000,000 bytes for just this instance of a field cache, more than 4Gb! This is basic formula. If you have few such fields, then you will need ton of memory, and you need few minutes to warm-up field cache. Calculate it properly: 8Gb or 24Gb? Consider sharding / SolrCloud if you need huge memory just for field cache. And you will be forced to consider it if you gave more that 2 billions documents (am I right? Lucene internal limitation, Integer.MAX_INT) Thanks, -- Fuad Efendi (416) 993-2060 http://www.tokenizer.ca Search Relevancy and Recommender Systems On November 2, 2016 at 1:11:10 PM, Erick Erickson (erickerick...@gmail.com) wrote: You need to move to SolrCloud when it's time to shard ;)..... More seriously, at some point simply adding more memory will not be adequate. Either your JVM heap will to grow to a point where you start encountering GC pauses or the time to serve requests will increase unacceptably. "when?" you ask? well unfortunately there are no guidelines that can be guaranteed, here's a long blog on the subject: https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ The short form is you need to stress-test your index and query patterns. Now, I've seen 20M docs strain a 32G Java heap. I've seen 300M docs give very nice response times with 12G of memory. It Depends (tm). Whether to put Solr on bare metal or not: There's inevitably some penalty for a VM. That said there are lots of places that use VMs successfully. Again, stress testing is the key. And finally, using docValues for any field that sorts, facets or groups will reduce the JVM requirements significantly, albeit by using OS memory space, see Uwe's excellent blog: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Best, Erick On Tue, Nov 1, 2016 at 10:23 PM, Kent Mu <solr.st...@gmail.com> wrote: > Thanks, I got it, Erick! > > the size of our index data is more than 30GB every year now, and it is > still growing up, and actually our solr now is running on a virtual > machine. so I wonder if we need to deploy solr in a physical machine, or I > can just upgrade the physical memory of our Virtual machines? > > Best, > Kent > > 2016-11-02 11:33 GMT+08:00 Erick Erickson <erickerick...@gmail.com>: > >> Kent: OK, I see now. Then a minor pedantic point... >> >> It'll avoid confusion if you use master and slaves >> rather than master and replicas when talking about >> non-cloud setups. >> >> The equivalent in SolrCloud is leader and replicas. >> >> No big deal either way, just FYI. >> >> Best, >> Erick >> >> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <solr.st...@gmail.com> wrote: >> > Thanks a lot for your reply, Shawn! >> > >> > no other applications on the server, I agree with you that we need to >> > upgrade physical memory, and allocate the reasonable jvm size, so that >> the >> > operating system have spare memory available to cache the index. >> > >> > actually, we have nearly 100 million of data every year now, and it is >> > still growing, and actually our solr now is running on a virtual machine. >> > so I wonder if we need to deploy solr in a physical machine. >> > >> > Best Regards! >> > Kent >> > >> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <apa...@elyograg.org>: >> > >> >> On 11/1/2016 1:07 AM, Kent Mu wrote: >> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to >> >> > connect to solr server, our deployment is one master with 10 replicas. >> >> > we index data to the master, and search data from the replicas via >> >> > load balancing. the error stack is as below: *Timeout occured while >> >> > waiting response from server at: >> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview >> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>* >> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured >> >> > while waiting response from server at: >> >> >> >> This shows that you are connecting to port 80. It is relatively rare to >> >> run Solr on port 80, though it is possible. Do you have an intermediate >> >> layer, like a proxy or a load balancer? If so, you'll need to ensure >> >> that there's not a problem there. If it works normally when replication >> >> isn't happening, that's probably not a worry. >> >> >> >> > It takes place not often. after analysis, we find that only when the >> >> > replicas Synchronous Data from master solr server. it seem that when >> >> > the replicas block search requests when synchronizing data from >> >> > master, is that true? >> >> >> >> Solr should be able to continue serving requests while replication >> >> happens. I have never heard of this happening before, and I never ran >> >> into it when I was using replication a long time ago on version 1.4.x. >> >> I think it is more likely that you've got a memory issue than a bug. If >> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need to >> >> upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is >> >> the latest at the moment, and release plans are underway for 6.3 right >> now. >> >> >> >> > I wonder if it is because that our solr server hardware configuration >> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we set >> >> > is Xms512m, Xmx7168m. >> >> >> >> The following assumes that there is no other software on the server, >> >> like a database, or an application server, web server, etc. If there >> >> is, any issues are likely to be a result of extreme memory starvation, >> >> and possibly swapping. Additional physical memory is definitely needed >> >> if there is other software on the server beyond basic OS tools. >> >> >> >> If the total index data that is on your server is larger than about 1.5 >> >> to 2GB, chances are excellent that you do not have enough free memory to >> >> cache that data effectively, which can lead to major performance >> >> issues. You've only left about 1GB of memory in the system for that >> >> purpose, and that memory must also run the entire operating system, >> >> which can take a significant percentage of 1GB. With a large index, I >> >> would strongly recommend adding memory to this server. >> >> >> >> https://wiki.apache.org/solr/SolrPerformanceProblems >> >> >> >> As mentioned in that wiki page, for good performance Solr absolutely >> >> requires that the operating system have spare memory available to cache >> >> the index. In general, allocating almost all your memory to the Java >> >> heap is a bad idea with Solr. >> >> >> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is >> >> probably not necessary, unless you are doing *incredibly* memory-hungry >> >> queries, such as grouping, faceting, or sorting on many fields. If you >> >> can reduce the heap size, there would be more memory available for >> caching. >> >> >> >> Indexing can sometimes cause very large merges to happen, and a full >> >> index optimize would rewrite the entire index. Replication copies the >> >> changed index files, and if the size of the changes is significant, >> >> additional memory can be required for good performance. See the special >> >> note on the wiki page above about optimizes. >> >> >> >> Thanks, >> >> Shawn >> >> >> >> >>
