Re: Solr tlog and soft commit

2015-03-17 Thread Shalin Shekhar Mangar
When you issue an atomic update request, Solr needs to lookup the stored fields of the last document that you have added, process it according to your atomic update and then index that document again (replacing the old one). If you send an atomic update request before the old document was committe

Re: Solr tlog and soft commit

2015-03-17 Thread vidit.asthana
Should I open a JIRA, in case there is no explanation of why all of a sudden transaction logs start piling up for some shard/replica? I have provided very detailed explanation in a different thread: http://lucene.472066.n3.nabble.com/Transaction-logs-not-getting-deleted-td4184635.html Also can s

Re: Solr tlog and soft commit

2015-03-16 Thread vidit.asthana
Can someone please reply to these questions? Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-tlog-and-soft-commit-tp4193105p4193311.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
Its for both. I am facing some problem, and I want to get to the root of it by understanding what happens when we issue an update. The problem I am facing is that sometimes, old transaction logs are not getting deleted for my solr cloud setup for one or two replicas, no matter how many times I do

Re: Solr tlog and soft commit

2015-03-15 Thread Erick Erickson
I have to ask what problem you're trying to solve, or is this just for background? Your understanding looks fine to me. Erick On Sun, Mar 15, 2015 at 9:44 AM, vidit.asthana wrote: > Thanks Eric. Its super helpful! > > So here's my understanding so far: > > 1. On update, write the doc to tlog(whi

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
Thanks Eric. Its super helpful! So here's my understanding so far: 1. On update, write the doc to tlog(which will be used only for recovery) 2. As soon as the docs size becomes greater than ramBufferSize, flush it to the latest segment inside the index directory. 3. Upto this point, even though t

Re: Solr tlog and soft commit

2015-03-15 Thread Yonik Seeley
On Sun, Mar 15, 2015 at 12:09 PM, Erick Erickson wrote: > 1> Well, probably not. Hate to be confusing here, but if your ramBufferSizeMB > setting is exceeded, then internal buffers will be flushed to the > currently open segment in the > index directory. It's even more confusing though... if you

Re: Solr tlog and soft commit

2015-03-15 Thread Erick Erickson
1> Well, probably not. Hate to be confusing here, but if your ramBufferSizeMB setting is exceeded, then internal buffers will be flushed to the currently open segment in the index directory. You still won't be able to search it since no commits happened. You really have little control over when thi

Re: Solr tlog and soft commit

2015-03-15 Thread vidit.asthana
Thanks for reply Yonik. I am very new to solrcloud and trying to understand how the update requests are handled and what exactly happens at file system level. 1. So lets say I send an update request, and I don't issue any type of commit(neither hard nor soft), so will the document ever touch index

Re: Solr tlog and soft commit

2015-03-15 Thread Yonik Seeley
Your basic assumptions about the underlying mechanisms are incorrect. The size of the index has nothing to do with the transaction logs... and transaction logs are never "written to index" except in recovery. You would see the same index size behavior w/o transaction logs, and it has to do with som