Thanks for your reply, Erick. 1. Unfortunately, we got those incidents after a long time, and relevant log files have been already rolled, so I couldn't find commit failure messages, but since I found OOM messages in other logs, I can guess it was the root cause. 2. Just to be sure I'm right, every soft commit, a new searcher is opened and all caches are warmed up again. Doesn't it impact performance (memory, IO, CPU)? Is there a best practice? 3. I think I used the wrong term. We saw cases where the tlog files were huge (more than 1 GB). We tried to change some settings in restart Solr, but it took a lot of time to load, I guess because of these tlog files. So again, is there a way Solr can do it async and not suspend the loading? 4. We can't upgrade to version greater than 7.3.1 currently. The question is there something in commit/tlog management that was improved in 7.3.1?
Thanks again. -----Original Message----- From: Erick Erickson <erickerick...@gmail.com> Sent: Wednesday, July 3, 2019 6:42 PM To: solr-user@lucene.apache.org Subject: Re: tlog/commit questions External Email: Don’t open links/attachments from untrusted senders Let’s take this a piece at a time. 1. commit failures are very rare, in fact the only time I’ve seen them is when running out of disk space, OOMs, pulling the plug, etc. Look in your log files, is there any evidence of same? 2. OOM messages. To support Real Time Get, internal structures are kept for all docs that have been indexed but no searcher has been opened to make visible. So you’re collecting up to 30 minutes of updates. This _may_ be relevant to your OOM problem. So I’d recommend dropping your soft commit interval to maybe 5 minutes. 3. Tlogs shouldn’t replay much. They only replay when Solr quits abnormally, OOM, kill -9, pull the plug etc. When Solr is shut down gracefully, i.e. “bin/solr stop” etc, it should commit before closing and should _not_ replay anything from the tlog. Of course you should stop indexing while shutting down Solr… 4. There are lots of improvements in Solr 7x. Go to the latest Solr version (7.7.2) rather than 7.3.1. That said, TMP has been around for a long, long time. The low-level process of merging segments hasn’t been changed. One thing that _has_ changed is that TMP will now respect the max segment size (5G) when optimizing or doing an expungeDeletes. And I strongly recommend that you do neither of those unless you demonstrate need, just mentioning in case you already do that. 7.4-: https://urldefense.proofpoint.com/v2/url?u=https-3A__lucidworks.com_post_segment-2Dmerging-2Ddeleted-2Ddocuments-2Doptimize-2Dmay-2Dbad_&d=DwIFaQ&c=TxO9TIZxM1NIgbR_44vEiALc2o8uaxixBRc1BtwrN08&r=N8Ef6xGR2eDgjA8I5q1SOErZhf616XiV4IPj4Ncf1w0&m=z9ADXLG0NJrFBm2Dxo17ipIMtPEHqjs1V_5liNAIwfk&s=6KFrZbKSYKabZXETQpnsK0CC2JkWue9WuURs6YcjO8M&e= 7.5+: https://urldefense.proofpoint.com/v2/url?u=https-3A__lucidworks.com_post_solr-2Dand-2Doptimizing-2Dyour-2Dindex-2Dtake-2Dii_&d=DwIFaQ&c=TxO9TIZxM1NIgbR_44vEiALc2o8uaxixBRc1BtwrN08&r=N8Ef6xGR2eDgjA8I5q1SOErZhf616XiV4IPj4Ncf1w0&m=z9ADXLG0NJrFBm2Dxo17ipIMtPEHqjs1V_5liNAIwfk&s=XBru20LMKde85b29wQx2BkeLns6RyxHp3ZRzqdMMRlw&e= All in all, I’d recommend getting to the bottom of your OOM issues. Absent abnormal termination, tlog replay really shouldn’t be happening. Are you totally sure that it was TLOG replay and not a full sync from the leader? Best, Erick > On Jul 3, 2019, at 12:36 AM, Avi Steiner <astei...@varonis.com> wrote: > > Hi > > We had some cases with customers (Solr 5.3.1, one search node, one shard) > with huge tlog files (more than 1 GB). > > Our settings: > > <updateHandler class="solr.DirectUpdateHandler2"> > <autoCommit> > <maxDocs>10000</maxDocs> > <maxTime>30000</maxTime> <!-- 30 seconds --> > <openSearcher>false</openSearcher> <!-- don't > open a new searcher --> > </autoCommit> > > <autoSoftCommit> > <maxTime>1800000</maxTime> <!-- 30 > minutes --> </autoSoftCommit> > > <updateLog> > <str name="dir">${solr.data.dir:}</str> > </updateLog> > </updateHandler> > > I don't have enough logs so I don't know if commit failed or not. I just > remember there were OOM messages. > > As you may know, during restart, Solr tries to replay from tlog. It may take > a lot of time. I tried to move the files to other location, started Solr and > only after the core was loaded, I moved tlog back to their original location. > They were cleared after a while. > > So I have few questions: > > 1. Do you have any idea for commit failures? > 2. Should we decrease the maxTime for hard commit or any other settings? > 3. Is there any way to replay tlog asynchronously (or disable it, so we > will be able to call it programmatically from our code in a separate thread), > so Solr will be loaded more quickly? > 4. Is there any improvement in Solr 7.3.1? > > Thanks in advance > > Avi > > > ________________________________ > This email and any attachments thereto may contain private, confidential, and > privileged material for the sole use of the intended recipient. Any review, > copying, or distribution of this email (or any attachments thereto) by others > is strictly prohibited. If you are not the intended recipient, please contact > the sender immediately and permanently delete the original and any copies of > this email and any attachments thereto. ________________________________ This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.
