Hello,
Our main Solr text search collection broke down last night (search was still
working fine), every indexing action timed out with the Solr master spending
most of its time in Java regex. One shard has only one replica left for queries
and it stays like that. I have copied both shard's leader to local to see what
is going on.
One shard is fine but the other has a replica with has about 600MB of data to
replay and it is extremely slow. Using the VisualVM sampler i find that the
replayer is also spending almost all time in dealing with Java regex (stack
trace below). Is this to be expected? And what is it actually doing? Where do
the TokenFilters come from?
I had a old but clean collection on the same cluster and started indexing to it
to see what is going on but it too timed out due to Java regex. This is weird,
because locally i have no problem indexing a million records in a 8.5.1
collection, and the broken down cluster has been running fine for over a month.
A note, this index uses PreAnalyzedField, so i would expect no analysis or
whatsoever, certainly no regex.
Thanks,
Markus
"replayUpdatesExecutor-3-thread-1-processing-n:127.0.1.1:8983_solr
x:sitesearch_shard2_replica_t2 c:sitesearch s:shard2 r:core_node4" #222 prio=5
os_prio=0 cpu=239207,44ms elapsed=239,50s tid=0x00007ffde0057000 nid=0x24f5
runnable [0x00007ffeedd0f000]
java.lang.Thread.State: RUNNABLE
at
java.util.regex.Pattern$GroupTail.match(java.base@11.0.7/Pattern.java:4863)
at
java.util.regex.Pattern$CharPropertyGreedy.match(java.base@11.0.7/Pattern.java:4306)
at
java.util.regex.Pattern$GroupHead.match(java.base@11.0.7/Pattern.java:4804)
at
java.util.regex.Pattern$CharPropertyGreedy.match(java.base@11.0.7/Pattern.java:4306)
at
java.util.regex.Pattern$Start.match(java.base@11.0.7/Pattern.java:3619)
at java.util.regex.Matcher.search(java.base@11.0.7/Matcher.java:1729)
at java.util.regex.Matcher.find(java.base@11.0.7/Matcher.java:746)
at
org.apache.lucene.analysis.pattern.PatternReplaceFilter.incrementToken(PatternReplaceFilter.java:71)
at
org.apache.lucene.analysis.miscellaneous.TrimFilter.incrementToken(TrimFilter.java:42)
at
org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:49)
at
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:812)
at
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:442)
at
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406)
at
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
at
org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:979)
at
org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:345)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:292)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:239)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:259)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:489)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$0(DistributedUpdateProcessor.java:339)
at
org.apache.solr.update.processor.DistributedUpdateProcessor$$Lambda$631/0x0000000840670c40.apply(Unknown
Source)
at
org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50)
- locked <0x00000000a7df5620> (a org.apache.solr.update.VersionBucket)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:339)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:225)
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at
org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$1(UpdateLog.java:2025)
at
org.apache.solr.update.UpdateLog$LogReplayer$$Lambda$629/0x0000000840672c40.run(Unknown
Source)
