Hi, I had three quesions/doubts regarding Solr and SolrCloud functionality. Can anyone help clarify these? I know these are bit long, please bear with me.
[A] Replication related - As I understand before SolrCloud, under a classic master/slave replication setup, every 'X' minutes slaves will pull/poll the updated index (index segments added and deleted/merged away ). And when a client explicitly issues a 'commit' only master solr closes/finalizes current index segment and creates a new current index segment. As port of this index segment merges as well as 'fsync' ensuring data is on the disk also happens. I read documentation regarding replication on SolrCloud but unfortunately it is still not very clear to me. Say I have solr cloud setup of 3 solr servers with just a single shard. Let's call them L (the leader) and F1 and F2, the followers. Case 1: We are not using autoCommits, and explictly issue 'commit' via Client. How does replication happen now? Does the each update to leader L that goes into tlog get replicated to followers F1, and F2 (wher they also put update in tlog ) before client sees response from leader L? What happens when client issues a 'commit'? Does the creation of new segment, merging of index segments if required, and fsync happen on all three solrs or that just happens on leader L and followers F1, F2 simply sync the post commit state of index. More-over does leader L wait for fsync in followers F1, F2, before responding sucessfully to Client? If yes does it sequentially updates F1 and then F2 or is the process concurrent/parallel via threads. Case 2: We use autoCommit every 'X' minutes and do not issue 'commit' via Client. Is this setup similar to classic master slave in terms of data/index updates? As in since autoCommit happens every 'X' minutes replication will happen after commit, every 'X' minutes followers get updated index. But does simple updates, the ones that go int tlog get replicated immediately to follower's tlog . Another thing I noticed in Solr Admin UI, is that replication is set to afterCommit, what are other possible settings for this knob. And what behaviour do we get out of them. [B] Load balancing related - In traditional master/slave setup we use load balancer to distribute load search query load equally over slaves. In case one of the slave solr is running on 'beefier' machine (say more RAM or CPU or both) than others, then load balancers allow distributing load by weights so that we can distribute load proportional to percieved machine capacity. With solr cloud setup, lets take an example, 2 shards, 3 replicas per shard, totaling to 6 solr servers are running and say we have Servers S1L1, S1F1, S1F2 hosting replicas of shard1 and servers S2L1, S2F1, S2F2 hosting replicas of shard2. S1L1 and S2L2 happen to be leaders of their respective shard. And lets say S1F2, and S2F1 happen to be twice as big machines as others (twice the RAM and CPU). Ideally speaking in such case we would want S2F1 and S1F2 to handle twice the search query load as their peers. That is if 100 search queries come we know each shard will receive these 100 queries. So we want S1L1, and S1F1 to handle 25 queries each, and S1F2 to handle 50 queries. Similarly we would want S2L1 and S2F2 to handle 25 queries and S2F1 to handle 50 queries. As far as I understand, this is not possible via smart client provided in SolrJ. All solr servers will handle 33% of the query load. Alternative is to use dumb client and load balancer over all servers. But even then I guess we won't get correct/desired distribution of queries. Say we put following weights for each server 1 - S1L1 1 - S1F1 2 - S1F2 1 - S1L1 2 - S1F1 1 - S1F2 Now 1/4 of total number of requests go to S1F2 directly, plus now it recieves 1/6 ( 1/2 * 1/3 ) of request that went to some server on shard 2. This totals up to 10/24 of request load, not half as we would expect. One way could be to chose weight y and x such that y/(2*(y + 2x)) + 1/6 = 1/2 . It seems too much of trouble to get them ( y = 4 and x = 1 ). Every time we add/remove/upgrade servers we need to recalculate new weights. A simpler alternative it appears would be that each solr node register its 'query_weight' with zoo-keeper on joining the cluster. This 'query_weight' could be a property similar to 'solr.solr.home' or 'zkHosts' that we specify with startup commandline for solr server. And all smart clients and solr servers, to honour that weight when they distribute load. Is there such a feature planned for Solr Cloud? [C] GC/Memory usage related - From the documentation and videos available on internet, it appears that solr perform well if index fits into the memory and stord fields fit in the memory. Holding just index in memory has more degrading impact on solr performance and if we don't have enough memory to hold index solr is still slower, and the moment java process hits swap space solr will slow to a crawl. My question is what the 'memory' being talked about is? Is it the Java Heap we specify via Xmx and Xms options. Or is it the free memory, or buffered, or cached memory as available from output free command on *nix systems. And how do we know if our index and stored fields will fit the memory. For example say the data directory for the core/collection occupies 200MB on disk ( 150,000 live documents and 180,000 max documents per Solr UI) , then is a 8GB machine with solr being configured with Xmx 4G going to be sufficient? Are there any guidlines as to configuring the java heap and total RAM, given an index size and the expected query rate ( queries per minute/second). On production system I observed via gc logs that minor collections happen at rate of 20 per minute, full gc happens every seven to ten minutes, are these too high or low given direct search query load on that solr node is about 2500 request per minute. What kind of GC behaviour I should expect from an healthy and fast/optimal solr node in solr-cloud setup. Is the answer it depends on your specific response time and throughput requirements or is there some kind of rule of thumb that can followed irrespective of the situation. Or should I see if any improvements can be made via regular measure, tweak , measure cycles. Thanks, Himanshu
