[
https://issues.apache.org/jira/browse/SOLR-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029273#comment-17029273
]
Jörn Franke commented on SOLR-14202:
------------------------------------
Some excerpt from the logfiles when this happens. What strange is that it opens
searcher ec395c2 at one point in time and later it registers it as new
searcher. Not sure if this is normal/expected.
2020-02-03 16:07:58.649 INFO (commitScheduler-23-thread-1) [ ]
o.a.s.s.SolrIndexSearcher Opening [Searcher@*c35a51d*[c_test_shard1_replica_n1]
main]
2020-02-03 16:07:58.651 INFO (commitScheduler-23-thread-1) [ ]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-02-03 16:07:58.652 INFO (searcherExecutor-17-thread-1) [ ]
o.a.s.c.QuerySenderListener QuerySenderListener sending requests to
Searcher@c35a51d[c_test_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1q(8.4.1):C59:[diagnostics={os=Linux,
java.vendor=Amazon.com Inc., java.version=11.0.4,
java.vm.version=11.0.4+11-LTS, lucene.version=8.4.1, os.arch=amd64,
java.runtime.version=11.0.4+11-LTS, source=flush,
os.version=3.10.0-1062.9.1.el7.x86_64,
[..]
2020-02-03 16:07:58.660 INFO (commitScheduler-22-thread-1) [ ]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-02-03 16:07:58.660 INFO (Thread-18) [ ] o.a.s.u.DirectUpdateHandler2 start
commit\{_version_=1657532400040673280,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-02-03 16:07:58.660 INFO (Thread-18) [ ] o.a.s.u.SolrIndexWriter Calling
setCommitData with IW:org.apache.solr.update.SolrIndexWriter@84c04e1
commitCommandVersion:1657532400040673280
2020-02-03 16:07:58.735 INFO (Thread-18) [ ] o.a.s.s.SolrIndexSearcher Opening
[Searcher@*ec395c2*[c_test_shard1_replica_n1] main]
2020-02-03 16:07:58.735 INFO (Thread-18) [ ] o.a.s.u.DirectUpdateHandler2
end_commit_flush
2020-02-03 16:07:58.736 INFO (searcherExecutor-17-thread-1) [ ]
o.a.s.c.QuerySenderListener QuerySenderListener sending requests to
Searcher@*ec395c2*[c_test_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1q(8.4.1):C59:[diagnostics={os=Linux,
java.vendor=Amazon.com Inc., java.version=11.0.4,
java.vm.version=11.0.4+11-LTS, lucene.version=8.4.1, os.arch=amd64,
java.runtime.version=11.0.4+11-LTS, source=flush,
os.version=3.10.0-1062.9.1.el7.x86_64,
[..]
2020-02-03 16:07:58.736 INFO (searcherExecutor-17-thread-1) [ ]
o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-02-03 16:07:58.739 INFO (searcherExecutor-17-thread-1) [ ]
o.a.s.c.SolrCore [c_test_shard1_replica_n1] Registered new searcher
Searcher@*ec395c2*[c_test_shard1_replica_n1]
main\{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1q(8.4.1):C59:[diagnostics={os=Linux,
java.vendor=Amazon.com Inc., java.version=11.0.4,
java.vm.version=11.0.4+11-LTS, lucene.version=8.4.1, os.arch=amd64,
java.runtime.version=11.0.4+11-LTS, source=flush,
os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1580745897231}]:[attributes=\{Lucene50StoredFieldsFormat.mode=BEST_SPEED}])
Uninverting(_29(8.4.1):c76:[diagnostics=\{os=Linux, java.version=11.0.4,
os.arch=amd64, java.runtime.version=11.0.4+11-LTS, source=merge,
os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Amazon.com Inc.,
java.vm.version=11.0.4+11-LTS, lucene.version=8.4.1, mergeMaxNumSegments=-1,
mergeFactor=10,
timestamp=1580746003291}]:[attributes=\{Lucene50StoredFieldsFormat.mode=BEST_SPEED}])
> UpdateProcessor/also in DIH with a ScriptTransformer that does Atomic Updates
> leaks searchers
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-14202
> URL: https://issues.apache.org/jira/browse/SOLR-14202
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 8.3, 8.4
> Reporter: Jörn Franke
> Priority: Major
> Attachments: eoe.zip, eoedihleak.zip
>
>
> The data directory of a collection is growing and growing. It seems that old
> segments are not deleted. They are only deleting during start of Solr.
> How to reproduce. Have any collection (e.g. the example collection) and start
> indexing documents. Even during the indexing the data directory is growing
> significantly - much more than expected (several magnitudes). if certain
> documents are updated (without significantly increasing the amount of data)
> the index data directory grows again several magnitudes. Even for small
> collections the needed space explodes.
> This reduces significantly if Solr is stopped and then started. During
> startup (not shutdown) Solr purges all those segments if not needed (*
> sometimes some but not a significant amount is deleted during shutdown). This
> is of course not a good workaround for normal operations.
> It does not seem to have a affect on queries (their performance do not seem
> to change).
> The configs have not changed before the upgrade and after (e.g. from Solr 8.2
> to 8.3 to 8.4, not cross major versions), so I assume it could be related to
> Solr 8.4. It may have been also in Solr 8.3 (not sure), but not in 8.2.
>
> IndexConfig is pretty much default: Lock type: native, autoCommit: 15000,
> openSearcher=false, autoSoftCommit -1 (reproducible with autoCommit 5000).
> Nevertheless, it did not happen in previous versions of Solr and the config
> did not change.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
