Hi experts:
Need some help and suggestion about an issue I am facing
Solr info:
- Solr 8.7
- Solr cloud with tlog replica; replica size is 3 for my Solr collection
Issue:
- before issuing collection reload; I observed a new tlog file are created
after every commit; and those tlog files are deleted after a while (may be
after index are merged?)
- then I issued a collection reload using collection API on my collection at
20:15
- after leader replica is reloaded; no new tlog file are created; instead
latest tlog file is growing, and no tlog file is deleted after reload. Below
under "files under tlog directory" section is a snapshot of the tlog files
under tlog directory of the leader replica. Again, I issued collection reload
at 20:15, and after that tlog.0000000000000000883 is growing
- I looked into log file and find error log entries below under "log entries"
section, and the log entry repeats continuously for every auto commit after
reload. I hope this log entry can provide some information for the issue.
Please help and suggestion what I may do incorrectly. Or this is a known issue,
is there a way I can fix or work-around it?
Thank you so much!
--Michael Hu
--- beginning for files under tlog directory ---
-rw-r--r-- 1 solr solr 47527321 Mar 4 20:14 tlog.0000000000000000877
-rw-r--r-- 1 solr solr 42614907 Mar 4 20:14 tlog.0000000000000000878
-rw-r--r-- 1 solr solr 37524663 Mar 4 20:14 tlog.0000000000000000879
-rw-r--r-- 1 solr solr 44067997 Mar 4 20:14 tlog.0000000000000000880
-rw-r--r-- 1 solr solr 33209784 Mar 4 20:15 tlog.0000000000000000881
-rw-r--r-- 1 solr solr 55435186 Mar 4 20:15 tlog.0000000000000000882
-rw-r--r-- 1 solr solr 2179991713 Mar 4 20:29 tlog.0000000000000000883
--- end for files under tlog directory ---
--- beginning for log entries ---
2021-03-04 20:15:38.251 ERROR (commitScheduler-4327-thread-1) [c:mycollection
s:myshard r:core_node10 x:mycolletion_myshard_replica_t7] o.a.s.u.CommitTracker
auto commit error...:
org.apache.solr.common.SolrException: java.nio.channels.ClosedChannelException
at
org.apache.solr.update.TransactionLog.writeCommit(TransactionLog.java:503)
at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:835)
at org.apache.solr.update.UpdateLog.preCommit(UpdateLog.java:819)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:673)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:273)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.channels.ClosedChannelException
at
java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150)
at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:266)
at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
at java.base/java.nio.channels.Channels.writeFully(Channels.java:97)
at java.base/java.nio.channels.Channels$1.write(Channels.java:172)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:193)
at
org.apache.solr.update.TransactionLog.writeCommit(TransactionLog.java:498)
... 10 more
--- end for log entries ---
