Re: Whole RAM consumed while Indexing.

2015-03-15 Thread Nitin Solanki
Hi Erick, You are saying correct. Something, **"overlapping searchers" warning messages** are coming in logs. **numDocs numbers** are changing when documents are adding at the time of indexing. Any help? On Sat, Mar 14, 2015 at 11:24 PM, Erick Erickson wrote: > First, the soft commit

Re: Using the collections API to create a new collection

2015-03-15 Thread Mike Thomsen
Thanks. I think I found the final problem that I was facing in this ticket: https://issues.apache.org/jira/browse/SOLR-5306 On Sun, Mar 15, 2015 at 11:53 AM, Erick Erickson wrote: > Yes, "configs" is the same as configsets, I tend to use them > interchangeably. > > You're still missing the poin

Solr load balancing

2015-03-15 Thread SolrUser2604
I am using solr 4.10.1 on a 40 server cloud. Recently, due to an increase in concurrent user queries, I am dealing with problems of load balancing. What is the best way to load-balance distributed solr servers? Round-robin? Does it make sense to create a "special shard" that doesn't have an inde

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
Its for both. I am facing some problem, and I want to get to the root of it by understanding what happens when we issue an update. The problem I am facing is that sometimes, old transaction logs are not getting deleted for my solr cloud setup for one or two replicas, no matter how many times I do

Re: Solr tlog and soft commit

2015-03-15 Thread Erick Erickson
I have to ask what problem you're trying to solve, or is this just for background? Your understanding looks fine to me. Erick On Sun, Mar 15, 2015 at 9:44 AM, vidit.asthana wrote: > Thanks Eric. Its super helpful! > > So here's my understanding so far: > > 1. On update, write the doc to tlog(whi

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
Thanks Eric. Its super helpful! So here's my understanding so far: 1. On update, write the doc to tlog(which will be used only for recovery) 2. As soon as the docs size becomes greater than ramBufferSize, flush it to the latest segment inside the index directory. 3. Upto this point, even though t

Re: Solr tlog and soft commit

2015-03-15 Thread Yonik Seeley
On Sun, Mar 15, 2015 at 12:09 PM, Erick Erickson wrote: > 1> Well, probably not. Hate to be confusing here, but if your ramBufferSizeMB > setting is exceeded, then internal buffers will be flushed to the > currently open segment in the > index directory. It's even more confusing though... if you

Re: Solr tlog and soft commit

2015-03-15 Thread Erick Erickson
1> Well, probably not. Hate to be confusing here, but if your ramBufferSizeMB setting is exceeded, then internal buffers will be flushed to the currently open segment in the index directory. You still won't be able to search it since no commits happened. You really have little control over when thi

Re: Using the collections API to create a new collection

2015-03-15 Thread Erick Erickson
Yes, "configs" is the same as configsets, I tend to use them interchangeably. You're still missing the point. Once the files are up in Zookeeper, that's where they live. They do NOT then live on the nodes hosting the replicas. So, assuming that when you write bq: Our ZK configuration data is unde

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
Thanks for reply Yonik. I am very new to solrcloud and trying to understand how the update requests are handled and what exactly happens at file system level. 1. So lets say I send an update request, and I don't issue any type of commit(neither hard nor soft), so will the document ever touch index

Re: Solr tlog and soft commit

2015-03-15 Thread Yonik Seeley
Your basic assumptions about the underlying mechanisms are incorrect. The size of the index has nothing to do with the transaction logs... and transaction logs are never "written to index" except in recovery. You would see the same index size behavior w/o transaction logs, and it has to do with som

Re: Using the collections API to create a new collection

2015-03-15 Thread Mike Thomsen
I tried that with upconfig, and it created it under /configs. Our ZK configuration data is under /dev-local-solr/configs. Not sure how to specify that. Also, is "configs" the same thing as "configsets" for the version of solr that I'm using? Thanks, Mike On Sat, Mar 14, 2015 at 6:38 PM, Anshum

Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
I want to know what all thing gets written to index from tlog directory whenever a soft commit is issued. I have a test SolrCloud setup and I can see that even if I disable the hardcommit, and if I only issue soft commits, then also index directory keeps increasing little by little, so I am presu

Re: backport Heliosearch features to Solr

2015-03-15 Thread Damien Kamerman
Sounds like 64bit is OK. Would be work re-testing that G1GC assert trip with the latest JDK. On 14 March 2015 at 01:46, Shawn Heisey wrote: > On 3/12/2015 5:11 PM, Markus Jelsma wrote: > > Hello - i would assume off-heap would out perform any heap based data > structure. G1 is only useful if you