[jira] [Commented] (SOLR-599) Lightweight SolrJ client
[ https://issues.apache.org/jira/browse/SOLR-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042487#comment-17042487 ] Robert Muir commented on SOLR-599: -- Can the lightweight solrj client have lightweight *tests* too, using mocks instead of flaky tests that very slowly spin up multiple jetty servers? > Lightweight SolrJ client > > > Key: SOLR-599 > URL: https://issues.apache.org/jira/browse/SOLR-599 > Project: Solr > Issue Type: Improvement > Components: clients - java, SolrJ >Reporter: Shalin Shekhar Mangar >Priority: Minor > Fix For: 4.9, 6.0 > > Attachments: SOLR-599-fix-for-SolrJ-on-GAE.patch, SOLR-599.patch, > SOLR-599.patch > > > SolrJ provides a SolrServer implementation backed by commons-httpclient which > introduces many dependency jars (commons-codec, commons-io and > commons-logging). Apart from that SolrJ also uses StAX API for XML parsing > which introduces dependencies like stax-api, stax and stax-utils. > This enhancement will add a SolrServer implementation backed by > java.net.HttpUrlConnection and will use BinaryResponseParser as the default > response parser. Using this basic implementation out of the box would require > no dependencies on either commons-httpclient or StAX. The only dependency > would be on solr-commons making this a very lightweight and distribution > friendly Java client for Solr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] s1monw commented on issue #1274: LUCENE-9164: Prevent IW from closing gracefully if threads are still modifying
s1monw commented on issue #1274: LUCENE-9164: Prevent IW from closing gracefully if threads are still modifying URL: https://github.com/apache/lucene-solr/pull/1274#issuecomment-589987566 @dnhatn I did change some parts slightly to wait for threads after all resources are closed like merge schedulers etc. I am still a bit nervous about this change but tests are passing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dnhatn edited a comment on issue #1274: LUCENE-9164: Prevent IW from closing gracefully if threads are still modifying
dnhatn edited a comment on issue #1274: LUCENE-9164: Prevent IW from closing gracefully if threads are still modifying URL: https://github.com/apache/lucene-solr/pull/1274#issuecomment-589706428 @s1monw There are two related test failures: testHangDuringRollback and testCloseWithThreads. We might need to look at them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14278) data loss during live shard split if leader dies
Yonik Seeley created SOLR-14278: --- Summary: data loss during live shard split if leader dies Key: SOLR-14278 URL: https://issues.apache.org/jira/browse/SOLR-14278 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Reporter: Yonik Seeley While trying to develop better tests for shared storage (SOLR-13101), I ran across another failure for normal replica types as well (one of the first things I do when a test fails for shared storage is to try and validate that normal NRT replicas succeed.) The PR I'll open has a test adapted from the one in SOLR-13813 for master. Scenario: - indexing is happening during shard split - leader is killed shortly after (before the split has finished) and never brought back up - there are often some missing documents at the end While it's possible that the simulated killing of the node in the unit test is imperfect, I haven't reproduced a failure if I comment out the split command and just kill the leader. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] yonik opened a new pull request #1277: SOLR-14278: draft test for data loss when leader dies during live split
yonik opened a new pull request #1277: SOLR-14278: draft test for data loss when leader dies during live split URL: https://github.com/apache/lucene-solr/pull/1277 Draft patch of the simplest test to show an issue (not for committing yet). See SOLR-14278 for details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9241) fix most memory-hungry tests
Robert Muir created LUCENE-9241: --- Summary: fix most memory-hungry tests Key: LUCENE-9241 URL: https://issues.apache.org/jira/browse/LUCENE-9241 Project: Lucene - Core Issue Type: Improvement Reporter: Robert Muir Currently each test jvm has Xmx of 512M. With a modern macbook pro this is 4GB which is pretty crazy. On the other hand, if we fix a few edge cases, tests can work with lower heaps such as 128M. This can save many gigabytes (also it finds interesting memory waste/issues). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9241) fix most memory-hungry tests
[ https://issues.apache.org/jira/browse/LUCENE-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-9241: Attachment: LUCENE-9241.patch > fix most memory-hungry tests > > > Key: LUCENE-9241 > URL: https://issues.apache.org/jira/browse/LUCENE-9241 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9241.patch > > > Currently each test jvm has Xmx of 512M. With a modern macbook pro this is > 4GB which is pretty crazy. > On the other hand, if we fix a few edge cases, tests can work with lower > heaps such as 128M. This can save many gigabytes (also it finds interesting > memory waste/issues). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9241) fix most memory-hungry tests
[ https://issues.apache.org/jira/browse/LUCENE-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042782#comment-17042782 ] Robert Muir commented on LUCENE-9241: - Attached patch. [~dawid.weiss] I didn't yet modify the gradle build, I figured lets just clean up the memory hungry tests first. It is almost possible to run with 64MB heap with the patch, but we'd need to use OfflineSorter for the kuromoji/nori dictionary builds, which is more involved. > fix most memory-hungry tests > > > Key: LUCENE-9241 > URL: https://issues.apache.org/jira/browse/LUCENE-9241 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Robert Muir >Priority: Major > Attachments: LUCENE-9241.patch > > > Currently each test jvm has Xmx of 512M. With a modern macbook pro this is > 4GB which is pretty crazy. > On the other hand, if we fix a few edge cases, tests can work with lower > heaps such as 128M. This can save many gigabytes (also it finds interesting > memory waste/issues). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9241) fix most memory-hungry tests
[
https://issues.apache.org/jira/browse/LUCENE-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042785#comment-17042785
]
Robert Muir commented on LUCENE-9241:
-
There are a few "real" code changes here to review:
* {{kuromoji}} adopts {{nori}}'s in-memory representation of the in-memory
connection cost matrix. Instead of a 2-D heap {{short[][]}}, it uses a direct
buffer. I think this is a better representation.
* {{RunAutomaton}} uses a {{FixedBitSet}} instead of a {{boolean[]}} for the
accept states. This is only checked once per "word" by subclasses (see e.g.
https://github.com/apache/lucene-solr/blob/bed694ec8811c67b8ba4b4c8943e60eda281850a/lucene/core/src/java/org/apache/lucene/util/automaton/ByteRunAutomaton.java#L44
), and it just adds some shift/mask there. Probably helps to not be so
wasteful. On the other hand this isn't the heaviest part of this data structure
when tableized, but its at least a little less stupid?
* "write-time" data structures of {{kuromoji}}/{{nori}} are a little more
efficient on the connection costs and per-term metadata. This only impacts
tests or "regenerate" type tasks, but we shouldn't be so wasteful anyway: these
classes have been recently moved into the public API.
> fix most memory-hungry tests
>
>
> Key: LUCENE-9241
> URL: https://issues.apache.org/jira/browse/LUCENE-9241
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9241.patch
>
>
> Currently each test jvm has Xmx of 512M. With a modern macbook pro this is
> 4GB which is pretty crazy.
> On the other hand, if we fix a few edge cases, tests can work with lower
> heaps such as 128M. This can save many gigabytes (also it finds interesting
> memory waste/issues).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
