[jira] [Commented] (LUCENE-9322) Discussing a unified vectors format API
[
https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220027#comment-17220027
]
Tomoko Uchida commented on LUCENE-9322:
---
I encountered a test failure in TestVectorValues that can be reproduced with
this seed on my Linux PC (Fedora) and Java 11:
{code}
$ ./gradlew :lucene:core:test --tests
"org.apache.lucene.index.TestVectorValues" -Ptests.seed=BA5BA0B8B98813F2
org.apache.lucene.index.TestVectorValues > testSortedIndex FAILED
java.lang.AssertionError: expected:<3> but was:<2>
at
__randomizedtesting.SeedInfo.seed([BA5BA0B8B98813F2:CDC4C64E81716A26]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at
org.apache.lucene.index.TestVectorValues.testSortedIndex(TestVectorValues.java:583)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
> Discussing a unified vectors
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1993: .gitignore clean up
dweiss commented on a change in pull request #1993: URL: https://github.com/apache/lucene-solr/pull/1993#discussion_r511344950 ## File path: .gitignore ## @@ -1,43 +1,27 @@ -# . -/eclipse-build -/maven-build -/classes -build -/idea-build -dist -lib -test-lib -*~ -.#* -/build.properties -/.idea -lucene/**/*.iml -parent.iml -*.ipr -*.iws -/*.iml -.project -.classpath -.settings -/.caches -/prj.el -bin -/bin -/bin.* -pom.xml -/nbproject -/nb-build -.pydevproject -__pycache__ -/dev-tools/scripts/scripts.iml -.DS_Store - -build/ +# Gradle +#Ignore the generated local settings file. +/gradle.properties .gradle/ -.idea/ +build/ -# Ignore the generated local settings file. -gradle.properties +# IntelliJ IDEA +/.idea/ +#IntelliJ creates this folder, ignore. Review comment: This is caused by the fact that it's a composite build and we don't configure it with such extra care as we do with other projects (we don't override intellij's defaults). I don't think this matters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta opened a new pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta opened a new pull request #2023: URL: https://github.com/apache/lucene-solr/pull/2023 "sandbox" module shares package names o.a.l.codecs, o.a.l.document, and o.a.l.search. We have two choices to clean up this conflicts: - Move them under `o.a.l.sandbox` and relax classes/methods visibility to public/protected to allow classes in sandbox module access those (currently package-private) classes/methods. - Move them to core module to keep package-private visibility in core module as is. Here, I took the first option for all classes for the first step to resolve the conflicts. But that may be a rather wild approach; some classes probably should be moved to core (option 2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511364893
##
File path: lucene/core/src/java/org/apache/lucene/document/RangeFieldQuery.java
##
@@ -41,7 +41,7 @@
/**
* Query class for searching {@code RangeField} types by a defined {@link
Relation}.
*/
-abstract class RangeFieldQuery extends Query {
+public abstract class RangeFieldQuery extends Query {
Review comment:
This is used in o.a.l.sandbox.document.LatLonBoundingBox.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511365628
##
File path: lucene/core/src/java/org/apache/lucene/document/RangeFieldQuery.java
##
@@ -57,7 +57,7 @@
final int bytesPerDim;
/** Used by {@code RangeFieldQuery} to check how each internal or leaf node
relates to the query. */
- enum QueryType {
+ public enum QueryType {
Review comment:
This is used in o.a.l.sandbox.document.LatLonBoundingBox.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511367381
##
File path: lucene/core/src/java/org/apache/lucene/document/RangeFieldQuery.java
##
@@ -228,7 +228,7 @@ boolean matches(byte[] queryPackedValue, byte[]
packedValue, int numDims, int by
* @param ranges encoded range values; this is done by the {@code
RangeField} implementation
* @param queryType the query relation
*/
- RangeFieldQuery(String field, final byte[] ranges, final int numDims, final
QueryType queryType) {
+ protected RangeFieldQuery(String field, final byte[] ranges, final int
numDims, final QueryType queryType) {
Review comment:
This is used in o.a.l.sandbox.document.LatLonBoundingBox.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511367972
##
File path: lucene/core/src/java/org/apache/lucene/search/ExactPhraseMatcher.java
##
@@ -33,7 +33,8 @@
import org.apache.lucene.search.similarities.Similarity.SimScorer;
import org.apache.lucene.util.PriorityQueue;
-final class ExactPhraseMatcher extends PhraseMatcher {
+/** Expert: Find exact phrases */
+public final class ExactPhraseMatcher extends PhraseMatcher {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511368151
##
File path: lucene/core/src/java/org/apache/lucene/search/ExactPhraseMatcher.java
##
@@ -50,7 +51,8 @@ public PostingsAndPosition(PostingsEnum postings, int offset)
{
private final DocIdSetIterator approximation;
private final ImpactsDISI impactsApproximation;
- ExactPhraseMatcher(PhraseQuery.PostingsAndFreq[] postings, ScoreMode
scoreMode, SimScorer scorer, float matchCost) {
+ /** Expert: Creates ExactPhraseMatcher instance */
+ public ExactPhraseMatcher(PhraseQuery.PostingsAndFreq[] postings, ScoreMode
scoreMode, SimScorer scorer, float matchCost) {
Review comment:
Same avobe.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511368625
##
File path: lucene/core/src/java/org/apache/lucene/search/HitQueue.java
##
@@ -19,7 +19,8 @@
import org.apache.lucene.util.PriorityQueue;
-final class HitQueue extends PriorityQueue {
+/** Expert: Priority queue containing hit docs */
+public final class HitQueue extends PriorityQueue {
Review comment:
This is used in o.a.l.sandbox.search.LargeNumHitsTopDocsCollector.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511368688
##
File path: lucene/core/src/java/org/apache/lucene/search/HitQueue.java
##
@@ -59,7 +60,7 @@
* @param prePopulate
* specifies whether to pre-populate the queue with sentinel values.
*/
- HitQueue(int size, boolean prePopulate) {
+ public HitQueue(int size, boolean prePopulate) {
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511369392
##
File path: lucene/core/src/java/org/apache/lucene/search/MultiPhraseQuery.java
##
@@ -410,7 +410,7 @@ private boolean termArraysEquals(Term[][] termArrays1,
Term[][] termArrays2) {
*
* Note: positions are merged during freq()
*/
- static class UnionPostingsEnum extends PostingsEnum {
+ public static class UnionPostingsEnum extends PostingsEnum {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
##
File path: lucene/core/src/java/org/apache/lucene/search/MultiPhraseQuery.java
##
@@ -423,7 +423,7 @@ private boolean termArraysEquals(Term[][] termArrays1,
Term[][] termArrays2) {
/** list of subs (unordered) */
final PostingsEnum[] subs;
-UnionPostingsEnum(Collection subs) {
+public UnionPostingsEnum(Collection subs) {
Review comment:
Same avobe.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511369770
##
File path: lucene/core/src/java/org/apache/lucene/search/MultiPhraseQuery.java
##
@@ -575,17 +575,17 @@ private void growArray() {
}
}
- // Slower version of UnionPostingsEnum that delegates offsets and positions,
for
- // use by MatchesIterator
- static class UnionFullPostingsEnum extends UnionPostingsEnum {
+ /** Slower version of UnionPostingsEnum that delegates offsets and
positions, for
+ use by MatchesIterator */
+ public static class UnionFullPostingsEnum extends UnionPostingsEnum {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
##
File path: lucene/core/src/java/org/apache/lucene/search/MultiPhraseQuery.java
##
@@ -575,17 +575,17 @@ private void growArray() {
}
}
- // Slower version of UnionPostingsEnum that delegates offsets and positions,
for
- // use by MatchesIterator
- static class UnionFullPostingsEnum extends UnionPostingsEnum {
+ /** Slower version of UnionPostingsEnum that delegates offsets and
positions, for
+ use by MatchesIterator */
+ public static class UnionFullPostingsEnum extends UnionPostingsEnum {
int freq = -1;
boolean started = false;
final PriorityQueue posQueue;
final Collection subs;
-UnionFullPostingsEnum(List subs) {
+public UnionFullPostingsEnum(List subs) {
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511370101
##
File path: lucene/core/src/java/org/apache/lucene/search/Multiset.java
##
@@ -29,13 +29,13 @@
* Iteration order is not specified.
* @lucene.internal
*/
-final class Multiset extends AbstractCollection {
+public final class Multiset extends AbstractCollection {
Review comment:
This is used in o.a.l.sandbox.search.CoveringQuery.
##
File path: lucene/core/src/java/org/apache/lucene/search/Multiset.java
##
@@ -29,13 +29,13 @@
* Iteration order is not specified.
* @lucene.internal
*/
-final class Multiset extends AbstractCollection {
+public final class Multiset extends AbstractCollection {
private final Map map = new HashMap<>();
private int size;
/** Create an empty {@link Multiset}. */
- Multiset() {
+ public Multiset() {
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511370707
##
File path: lucene/core/src/java/org/apache/lucene/search/PhraseMatcher.java
##
@@ -26,7 +26,7 @@
* relevant document, then call {@link #reset()}. Clients can then call
* {@link #nextMatch()} to iterate over the matches
*/
-abstract class PhraseMatcher {
+public abstract class PhraseMatcher {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511371028
##
File path: lucene/core/src/java/org/apache/lucene/search/PhraseQuery.java
##
@@ -304,13 +304,15 @@ public void visit(QueryVisitor visitor) {
v.consumeTerms(this, terms);
}
- static class PostingsAndFreq implements Comparable {
+ /** Term postings and position information for phrase matching */
+ public static class PostingsAndFreq implements Comparable {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511371410
##
File path: lucene/core/src/java/org/apache/lucene/search/PhraseQuery.java
##
@@ -413,7 +415,7 @@ public boolean equals(Object obj) {
* This is for use by {@link TwoPhaseIterator#matchCost} implementations.
* @param termsEnum The term is the term at which this TermsEnum is
positioned.
*/
- static float termPositionsCost(TermsEnum termsEnum) throws IOException {
+ public static float termPositionsCost(TermsEnum termsEnum) throws
IOException {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511371777
##
File path: lucene/core/src/java/org/apache/lucene/search/PhraseWeight.java
##
@@ -23,13 +23,15 @@
import org.apache.lucene.search.similarities.Similarity;
import org.apache.lucene.search.similarities.Similarity.SimScorer;
-abstract class PhraseWeight extends Weight {
+/** Expert: Weight class for phrase matching */
+public abstract class PhraseWeight extends Weight {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511372160
##
File path:
lucene/core/src/java/org/apache/lucene/search/SloppyPhraseMatcher.java
##
@@ -53,7 +53,7 @@
* would get same score as "g f"~2, although "c b"~2 could be matched twice.
* We may want to fix this in the future (currently not, for performance
reasons).
*/
-final class SloppyPhraseMatcher extends PhraseMatcher {
+public final class SloppyPhraseMatcher extends PhraseMatcher {
Review comment:
This is used in o.a.l.sandbox.search.PhraseWildcardQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511372528
##
File path:
lucene/core/src/java/org/apache/lucene/search/SloppyPhraseMatcher.java
##
@@ -81,7 +81,7 @@
private boolean positioned;
private int matchLength;
- SloppyPhraseMatcher(PhraseQuery.PostingsAndFreq[] postings, int slop,
ScoreMode scoreMode, SimScorer scorer, float matchCost, boolean
captureLeadMatch) {
+ public SloppyPhraseMatcher(PhraseQuery.PostingsAndFreq[] postings, int slop,
ScoreMode scoreMode, SimScorer scorer, float matchCost, boolean
captureLeadMatch) {
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511372816
##
File path: lucene/core/src/java/org/apache/lucene/search/TermScorer.java
##
@@ -25,7 +25,7 @@
/** Expert: A Scorer for documents matching a Term.
*/
-final class TermScorer extends Scorer {
+public final class TermScorer extends Scorer {
Review comment:
This is used in o.a.l.sandbox.search.BM25FQuery.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511373161
##
File path: lucene/core/src/java/org/apache/lucene/search/TermScorer.java
##
@@ -35,7 +35,7 @@
/**
* Construct a {@link TermScorer} that will iterate all documents.
*/
- TermScorer(Weight weight, PostingsEnum postingsEnum, LeafSimScorer
docScorer) {
+ public TermScorer(Weight weight, PostingsEnum postingsEnum, LeafSimScorer
docScorer) {
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511373437
##
File path: lucene/core/src/java/org/apache/lucene/search/TermScorer.java
##
@@ -60,7 +60,8 @@ public int docID() {
return postingsEnum.docID();
}
- final int freq() throws IOException {
+ /** Returns term frequency in the current document. */
+ public final int freq() throws IOException {
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023: URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511373874 ## File path: lucene/core/src/java/org/apache/lucene/search/TopDocsCollector.java ## @@ -35,7 +35,7 @@ /** This is used in case topDocs() is called with illegal parameters, or there * simply aren't (enough) results. */ - protected static final TopDocs EMPTY_TOPDOCS = new TopDocs(new TotalHits(0, TotalHits.Relation.EQUAL_TO), new ScoreDoc[0]); + public static final TopDocs EMPTY_TOPDOCS = new TopDocs(new TotalHits(0, TotalHits.Relation.EQUAL_TO), new ScoreDoc[0]); Review comment: This is used in o.a.l.sandbox.search.LargeNumHitsTopDocsCollector. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #2023: LUCENE-9319: Clean up package name conflicts for sandbox module
mocobeta commented on a change in pull request #2023:
URL: https://github.com/apache/lucene-solr/pull/2023#discussion_r511374261
##
File path:
lucene/core/src/java/org/apache/lucene/search/TopScoreDocCollector.java
##
@@ -38,9 +38,10 @@
*/
public abstract class TopScoreDocCollector extends TopDocsCollector {
- abstract static class ScorerLeafCollector implements LeafCollector {
+ /** Scorable leaf collector */
+ public abstract static class ScorerLeafCollector implements LeafCollector {
Review comment:
This is used in o.a.l.sandbox.search.LargeNumHitsTopDocsCollector.
##
File path:
lucene/core/src/java/org/apache/lucene/search/TopScoreDocCollector.java
##
@@ -38,9 +38,10 @@
*/
public abstract class TopScoreDocCollector extends TopDocsCollector {
- abstract static class ScorerLeafCollector implements LeafCollector {
+ /** Scorable leaf collector */
+ public abstract static class ScorerLeafCollector implements LeafCollector {
-Scorable scorer;
+protected Scorable scorer;
Review comment:
Same above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9319) Clean up Sandbox project by retiring/delete functionality or move it to Lucene core
[
https://issues.apache.org/jira/browse/LUCENE-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220042#comment-17220042
]
Tomoko Uchida commented on LUCENE-9319:
---
I opened [https://github.com/apache/lucene-solr/pull/2023.]
In general, we have two options to resolve the split package issue for sandbox
module.
* Move them under {{o.a.l.sandbox}} (i.e., {{o.a.l.sandbox.codecs}},
{{o.a.l.sandbox.document}}, and {{o.a.l.sandbox.search}}) and relax
classes/methods visibility to public/protected to allow classes in sandbox
module access those (currently package-private) classes/methods.
* Move them to core module to keep package-private visibility in core module
as is.
Here I took the first option for all classes for the first step to resolve the
conflicts. But that may be a rather wild approach; some classes probably should
be moved to core (option 2).
> Clean up Sandbox project by retiring/delete functionality or move it to
> Lucene core
> ---
>
> Key: LUCENE-9319
> URL: https://issues.apache.org/jira/browse/LUCENE-9319
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/other
>Affects Versions: master (9.0)
>Reporter: David Ryan
>Priority: Major
> Labels: build, features
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> To allow Lucene to be modularised with Java module system there are a few
> preparatory tasks to be completed prior to this being possible. These are
> detailed by Uwe on the mailing list here:
> [http://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/%3c0a5e01d60ff2$563f9c80$02bed580$@thetaphi.de%3e]
>
> The lucene-sandbox currently shares package names with lucene-core which is
> not allowed in the Java module system. There are two ways to deal with this.
> Either prefix all packages with "sandbox" or retire the lucene-sandbox all
> together. As per the email:
> {quote}Cleanup sandbox to prefix all classes there with “sandbox” package and
> where needed remove package-private access. If it’s needed for internal
> access, WTF: Just move the stuff to core! We have a new version 9.0, so
> either retire/delete Sandbox stuff or make it part of Lucene core.
> {quote}
> The suggested way forward is to move sandbox code to core.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14942) Reduce leader election time on node shutdown
[ https://issues.apache.org/jira/browse/SOLR-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220064#comment-17220064 ] ASF subversion and git services commented on SOLR-14942: Commit 706f284c467becb5f002c05455808ee31aee3465 in lucene-solr's branch refs/heads/master from Shalin Shekhar Mangar [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=706f284 ] SOLR-14942: Reduce leader election time on node shutdown (#2004) The shutdown process waits for all replicas/cores to be closed before removing the election node of the leader. This can take some time due to index flush or merge activities on the leader cores and delays new leaders from being elected. Moreover, jetty stops accepting new requests on receiving SIGTERM which means that even though a leader technically exists, no new indexing requests can be processed by the node. This commit waits for all in-flight indexing requests to complete, removes election nodes (thus triggering leader election) and then closes all replicas. Co-authored-by: Cao Manh Dat > Reduce leader election time on node shutdown > > > Key: SOLR-14942 > URL: https://issues.apache.org/jira/browse/SOLR-14942 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 7.7.3, 8.6.3 >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > The credit for this issue and investigation belongs to [~caomanhdat]. I am > merely reporting the issue and creating PRs based on his work. > The shutdown process waits for all replicas/cores to be closed before > removing the election node of the leader. This can take some time due to > index flush or merge activities on the leader cores and delays new leaders > from being elected. > This process happens at CoreContainer.shutdown(): > # zkController.preClose(): remove current node from live_node and change > states of all cores in this node to DOWN state. Assuming that the current > node hosting a leader of a shard, the shard becomes leaderless after calling > this method, since the state of the leader is DOWN now. The leader election > process is not triggered for the shard since the election node is still > on-hold by the current node. > # Waiting for all cores to be loaded (if there are any). > # SolrCores.close(): close all cores. > # zkController.close(): this is where all ephemeral nodes are removed from ZK > which include election nodes created by this node. Therefore other replicas > in the shard can take part in the leader election from now. > Note that CoreContainer.shutdown() is invoked when Jetty/Solr nodes receive > SIGTERM signal. > On receiving SIGTERM, Jetty will also stop accepting new connections and new > requests. This is a very important factor, since even if the leader replica > is ACTIVE and its node in live_nodes, the shard will be considered as > leaderless if no-one can index to that shard. Therefore shards become > leaderless as soon as the node (which contains shard’s leader) receives > SIGTERM. > Therefore the longer time step 1, 2 and 3 needed to finish, the longer shards > remain leaderless. The time needed for step 3 scales with the number of cores > so the more cores a node has, the worse. This time is spent in > IndexWriter.close() where the system will > # Flush all pending updates to disk > # Waiting for all merge finish (this most likely is the meaty part) > The shutdown process is proposed to changed to: > # Wait for all in-flight indexing requests and replication requests to > complete > # Remove election nodes > # Close all replicas/cores > This ensures that index flush or merges do not block new leader elections > anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] shalinmangar merged pull request #2004: SOLR-14942: Reduce leader election time on node shutdown
shalinmangar merged pull request #2004: URL: https://github.com/apache/lucene-solr/pull/2004 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9322) Discussing a unified vectors format API
[ https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220065#comment-17220065 ] Michael Sokolov commented on LUCENE-9322: - Ooh, a fun failure - ah I see it's because we got an index with multiple segments, but we only expect one. I'll add a forceMerge(1) here and in a few other tests that have similar assumptions > Discussing a unified vectors format API > --- > > Key: LUCENE-9322 > URL: https://issues.apache.org/jira/browse/LUCENE-9322 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Julie Tibshirani >Priority: Major > Fix For: master (9.0) > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Two different approximate nearest neighbor approaches are currently being > developed, one based on HNSW (LUCENE-9004) and another based on coarse > quantization ([#LUCENE-9136]). Each prototype proposes to add a new format to > handle vectors. In LUCENE-9136 we discussed the possibility of a unified API > that could support both approaches. The two ANN strategies give different > trade-offs in terms of speed, memory, and complexity, and it’s likely that > we’ll want to support both. Vector search is also an active research area, > and it would be great to be able to prototype and incorporate new approaches > without introducing more formats. > To me it seems like a good time to begin discussing a unified API. The > prototype for coarse quantization > ([https://github.com/apache/lucene-solr/pull/1314]) could be ready to commit > soon (this depends on everyone's feedback of course). The approach is simple > and shows solid search performance, as seen > [here|https://github.com/apache/lucene-solr/pull/1314#issuecomment-608645326]. > I think this API discussion is an important step in moving that > implementation forward. > The goals of the API would be > # Support for storing and retrieving individual float vectors. > # Support for approximate nearest neighbor search -- given a query vector, > return the indexed vectors that are closest to it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9581) Clarify discardCompoundToken behavior in the JapaneseTokenizer
[
https://issues.apache.org/jira/browse/LUCENE-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220078#comment-17220078
]
Tomoko Uchida commented on LUCENE-9581:
---
{quote}As we have discussed in LUCENE-9123, we want to change the default
behavior of the current search mode that the tokenization results will be the
same with `discardCompoundToken=true`.
{quote}
I am in favor of that... we don't need compound tokens for the purpose of
searching. Also, personally I would like to provide a synonym filter friendly
default configuration to users (I saw every new user who wants to use synonym
(graph) filter with JapaneseTokenizer encounters the issue that was described
in LUCENE-9123).
> Clarify discardCompoundToken behavior in the JapaneseTokenizer
> --
>
> Key: LUCENE-9581
> URL: https://issues.apache.org/jira/browse/LUCENE-9581
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-9581.patch, LUCENE-9581.patch
>
>
> At first sight, the discardCompoundToken option added in LUCENE-9123 seems
> redundant with the NORMAL mode of the Japanese tokenizer. When set to true,
> the current behavior is to disable the decomposition for compounds, that's
> exactly what the NORMAL mode does.
> So I wonder if the right semantic of the option would be to keep only the
> decomposition of the compound or if it's really needed. If the goal is to
> make the output compatible with a graph token filter, the current workaround
> to set the mode to NORMAL should be enough.
> That's consistent with the mode that should be used to preserve positions in
> the index since we don't handle position length on the indexing side.
> Am I missing something regarding the new option ? Is there a compelling case
> where it differs from the NORMAL mode ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14067) Move StatelessScriptUpdateProcessor to a contrib
[ https://issues.apache.org/jira/browse/SOLR-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220082#comment-17220082 ] David Eric Pugh commented on SOLR-14067: Thanks [~tflobbe], I found that just having StatelessScriptUpdateProcessor extend ScriptUpdateProcessor worked, I gave it a test with the techproducts example.I did put in a deprecation annotation. > Move StatelessScriptUpdateProcessor to a contrib > > > Key: SOLR-14067 > URL: https://issues.apache.org/jira/browse/SOLR-14067 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: David Eric Pugh >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Move server-side scripting out of core and into a new contrib. This is > better for security. > Former description: > > We should eliminate all scripting capabilities within Solr. Let us start with > the StatelessScriptUpdateProcessor deprecation/removal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14067) Move StatelessScriptUpdateProcessor to a contrib
[ https://issues.apache.org/jira/browse/SOLR-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220083#comment-17220083 ] David Eric Pugh commented on SOLR-14067: [~dsmiley] I added a notice to the `major-changes-in-solr-9.adoc`.I looked at the wiki page, but didn't quite know how to convey "The original plan was deprecation but now we moved to it's one contrib in Solr 9 instead". Maybe just delete the row?At somepoint I'd like to make it a proper package, but I think that is another ticket for another day. > Move StatelessScriptUpdateProcessor to a contrib > > > Key: SOLR-14067 > URL: https://issues.apache.org/jira/browse/SOLR-14067 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Assignee: David Eric Pugh >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Move server-side scripting out of core and into a new contrib. This is > better for security. > Former description: > > We should eliminate all scripting capabilities within Solr. Let us start with > the StatelessScriptUpdateProcessor deprecation/removal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9322) Discussing a unified vectors format API
[ https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220065#comment-17220065 ] Michael Sokolov edited comment on LUCENE-9322 at 10/24/20, 12:47 PM: - Ooh, a fun failure - ah I see it's because we got an index with multiple segments, but we only expect one. I'll add a forceMerge(1) here and in a few other tests that have similar assumptions. OK, I added forceMerge calls and also switched to using getOnlyLeafReader which throws an exception when there are multiple readers. pushed a fix: https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=commit;h=56eac7c435c31abd8fce090e96e4bf09b6d1cad6 was (Author: sokolov): Ooh, a fun failure - ah I see it's because we got an index with multiple segments, but we only expect one. I'll add a forceMerge(1) here and in a few other tests that have similar assumptions > Discussing a unified vectors format API > --- > > Key: LUCENE-9322 > URL: https://issues.apache.org/jira/browse/LUCENE-9322 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Julie Tibshirani >Priority: Major > Fix For: master (9.0) > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Two different approximate nearest neighbor approaches are currently being > developed, one based on HNSW (LUCENE-9004) and another based on coarse > quantization ([#LUCENE-9136]). Each prototype proposes to add a new format to > handle vectors. In LUCENE-9136 we discussed the possibility of a unified API > that could support both approaches. The two ANN strategies give different > trade-offs in terms of speed, memory, and complexity, and it’s likely that > we’ll want to support both. Vector search is also an active research area, > and it would be great to be able to prototype and incorporate new approaches > without introducing more formats. > To me it seems like a good time to begin discussing a unified API. The > prototype for coarse quantization > ([https://github.com/apache/lucene-solr/pull/1314]) could be ready to commit > soon (this depends on everyone's feedback of course). The approach is simple > and shows solid search performance, as seen > [here|https://github.com/apache/lucene-solr/pull/1314#issuecomment-608645326]. > I think this API discussion is an important step in moving that > implementation forward. > The goals of the API would be > # Support for storing and retrieving individual float vectors. > # Support for approximate nearest neighbor search -- given a query vector, > return the indexed vectors that are closest to it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] epugh commented on pull request #2016: SOLR-14067 v2 Move Stateless Scripting Update Process to /contrib
epugh commented on pull request #2016: URL: https://github.com/apache/lucene-solr/pull/2016#issuecomment-715911331 @dweiss I'm hoping you can steer me in the right direction. I removed the `contrib/scripting-update-processor/src/java/org/apache/solr/update/processor/package-info.java` because Eclipse flagged that there was already one in `core/src/java/org/apache/solr/update/processor/`, and instead added a `package.html`, following another example. That is making the precommit fail. Thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14945) Improve compressed HTTP responses handling in SolrJ
[ https://issues.apache.org/jira/browse/SOLR-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220095#comment-17220095 ] Erick Erickson commented on SOLR-14945: --- This is explained more fully in SOLR-14844. The short form is we're setting the min gzip size to a magic number 23 in JettySolrRunnner. That corresponds to the number in the Jetty issues Samuel linked above. If Jetty changes this in future, we'll go back around this loop so we need something more robust. > Improve compressed HTTP responses handling in SolrJ > --- > > Key: SOLR-14945 > URL: https://issues.apache.org/jira/browse/SOLR-14945 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: Samuel García Martínez >Priority: Major > > After upgrading Jetty version to 9.4.32.v20200930 the suite > BasicHttpSolrClientTest started to fail because of how the > compression/decompression is handled on the client side. > Jetty changed its behaviour with this upgrade as part of the so for empty > responses (Content-Length: 0) with Accept-Encoding: gzip it's still returning > a Content-Encoding: gzip but with no gzip header bytes in the response. > Jetty's relevant issue: https://github.com/eclipse/jetty.project/issues/4824 > Jetty's relevant code changes: > https://github.com/eclipse/jetty.project/commit/d58da0f7d2e30c732f52a38feaba2974e299bf70#diff-e175032f6f83c7d41fb7bc1a66187f5aR232 > Interceptors should be removed and BasicHttpSolrClient should use Apache > HttpClient's provided processors decompress the HTTP responses. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14844) Upgrade Jetty to 9.4.32.v20200930
[ https://issues.apache.org/jira/browse/SOLR-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220096#comment-17220096 ] Erick Erickson commented on SOLR-14844: --- [~samuelgmartinez] Why not just change the min GZIP compression from 0 to 23 in JettySolrRunner? When I make that change, the test passes on 8x just as it does on master without any other changes. 8x precommit fails because in the new mock writer httpResp.getWriter() is a forbidden API call, which is what started me down this path. I'm totally uncomfortable with setting the minimum gzip size to the magic number 23 as a fix for 8x, all that can be said for it is "it matches what's on master". I see what you mean though and a more robust fix in SOLR-14945 makes sense, thanks for raising that JIRA. [~mdrob] in SOLR-14264, you changed the gzip compression minimum size from 0 to 23 on master, but not 8x. Do you know any reason it shouldn't be changed in 8x too or was that omission just being cautious? I'm running the full suite on 8x now, so maybe it'll be obvious soon. > Upgrade Jetty to 9.4.32.v20200930 > - > > Key: SOLR-14844 > URL: https://issues.apache.org/jira/browse/SOLR-14844 > Project: Solr > Issue Type: Improvement >Affects Versions: 8.6 >Reporter: Cassandra Targett >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-14844-master.patch, SOLR-14884-8x.patch > > Time Spent: 20m > Remaining Estimate: 0h > > A CVE was found in Jetty 9.4.27-9.4.29 that has some security scanning tools > raising red flags > ([https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17638]). > Here's the Jetty issue: > [https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984]. It's fixed in > 9.4.30+, so we should upgrade to that for 8.7 > -It has a simple mitigation (raise Jetty's responseHeaderSize to higher than > requestHeaderSize), but I don't know how Solr uses Jetty well enough to a) > know if this problem is even exploitable in Solr, or b) if the workaround > suggested is even possible in Solr.- > In normal Solr installs, w/o jetty optimizations, this issue is largely > mitigated in 8.6.3: see SOLR-14896 (and linked bug fixes) for details. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14469) Removed deprecated code in solr/core (master only)
[ https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220116#comment-17220116 ] Erick Erickson commented on SOLR-14469: --- Who knows? Maybe I'll get to this as winter closes in... Java9 has "since" and "forRemoval" parameters. WDYT about requiring those tags and failing as part of validating source patterns if not present? Currently, to know whether the intent is to remove something if the comments aren't in place in the javadocs, you have to go to the JIRA and see what version the annotation was added in and then guess. Mechanically, that's easy with Git annotate on a one-by-one basis, but I don't see any good way to get a list of all the deprecations that could be removed in version X. Although until we resolve the reference impl, it'd be a lot of fiddly work that might have to be repeated... > Removed deprecated code in solr/core (master only) > -- > > Key: SOLR-14469 > URL: https://issues.apache.org/jira/browse/SOLR-14469 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I'm currently working on getting all the warnings out of the code, so this is > something of a placeholder for a week or two. > There will be sub-tasks, please create them when you start working on a > project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14469) Removed deprecated code in solr/core (master only)
[ https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220120#comment-17220120 ] Erick Erickson commented on SOLR-14469: --- See Smiley's DeprecationLog class too? > Removed deprecated code in solr/core (master only) > -- > > Key: SOLR-14469 > URL: https://issues.apache.org/jira/browse/SOLR-14469 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I'm currently working on getting all the warnings out of the code, so this is > something of a placeholder for a week or two. > There will be sub-tasks, please create them when you start working on a > project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14067) Move StatelessScriptUpdateProcessor to a contrib
[
https://issues.apache.org/jira/browse/SOLR-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220124#comment-17220124
]
David Smiley commented on SOLR-14067:
-
RE package: That will happen in SOLR-14688 or some new sub-task ticket of that.
I think we shouldn't name this contrib "scripting-update-processor"; I think it
should just be "scripting" to make room for other scripting functionality. The
URP may be the only component there on inception, but it's easy to imagine
other things being added. XSLT belongs there. WDYT?
I disagree on use of the "@Deprecation" annotation (although the
{{\@deprecated}} javadoc tag is fine) because the user will get a generic
warning about the functionality being deprecated (implying you shouldn't use
it). Instead, use the new DeprecationLog utility in the constructor to say
something more helpful. Or just don't bother, as you wish.
> Move StatelessScriptUpdateProcessor to a contrib
>
>
> Key: SOLR-14067
> URL: https://issues.apache.org/jira/browse/SOLR-14067
> Project: Solr
> Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: David Eric Pugh
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Move server-side scripting out of core and into a new contrib. This is
> better for security.
> Former description:
>
> We should eliminate all scripting capabilities within Solr. Let us start with
> the StatelessScriptUpdateProcessor deprecation/removal.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley merged pull request #1993: .gitignore clean up
dsmiley merged pull request #1993: URL: https://github.com/apache/lucene-solr/pull/1993 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14942) Reduce leader election time on node shutdown
[ https://issues.apache.org/jira/browse/SOLR-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220133#comment-17220133 ] ASF subversion and git services commented on SOLR-14942: Commit b6d06bb309d121901ab6ce1d1935b4067ce610fe in lucene-solr's branch refs/heads/branch_8x from Shalin Shekhar Mangar [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b6d06bb ] SOLR-14942: Reduce leader election time on node shutdown (#2004) The shutdown process waits for all replicas/cores to be closed before removing the election node of the leader. This can take some time due to index flush or merge activities on the leader cores and delays new leaders from being elected. Moreover, jetty stops accepting new requests on receiving SIGTERM which means that even though a leader technically exists, no new indexing requests can be processed by the node. This commit waits for all in-flight indexing requests to complete, removes election nodes (thus triggering leader election) and then closes all replicas. Co-authored-by: Cao Manh Dat (cherry picked from commit 706f284c467becb5f002c05455808ee31aee3465) > Reduce leader election time on node shutdown > > > Key: SOLR-14942 > URL: https://issues.apache.org/jira/browse/SOLR-14942 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 7.7.3, 8.6.3 >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > The credit for this issue and investigation belongs to [~caomanhdat]. I am > merely reporting the issue and creating PRs based on his work. > The shutdown process waits for all replicas/cores to be closed before > removing the election node of the leader. This can take some time due to > index flush or merge activities on the leader cores and delays new leaders > from being elected. > This process happens at CoreContainer.shutdown(): > # zkController.preClose(): remove current node from live_node and change > states of all cores in this node to DOWN state. Assuming that the current > node hosting a leader of a shard, the shard becomes leaderless after calling > this method, since the state of the leader is DOWN now. The leader election > process is not triggered for the shard since the election node is still > on-hold by the current node. > # Waiting for all cores to be loaded (if there are any). > # SolrCores.close(): close all cores. > # zkController.close(): this is where all ephemeral nodes are removed from ZK > which include election nodes created by this node. Therefore other replicas > in the shard can take part in the leader election from now. > Note that CoreContainer.shutdown() is invoked when Jetty/Solr nodes receive > SIGTERM signal. > On receiving SIGTERM, Jetty will also stop accepting new connections and new > requests. This is a very important factor, since even if the leader replica > is ACTIVE and its node in live_nodes, the shard will be considered as > leaderless if no-one can index to that shard. Therefore shards become > leaderless as soon as the node (which contains shard’s leader) receives > SIGTERM. > Therefore the longer time step 1, 2 and 3 needed to finish, the longer shards > remain leaderless. The time needed for step 3 scales with the number of cores > so the more cores a node has, the worse. This time is spent in > IndexWriter.close() where the system will > # Flush all pending updates to disk > # Waiting for all merge finish (this most likely is the meaty part) > The shutdown process is proposed to changed to: > # Wait for all in-flight indexing requests and replication requests to > complete > # Remove election nodes > # Close all replicas/cores > This ensures that index flush or merges do not block new leader elections > anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14844) Upgrade Jetty to 9.4.32.v20200930
[ https://issues.apache.org/jira/browse/SOLR-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220201#comment-17220201 ] Erick Erickson commented on SOLR-14844: --- All the tests pass with just changing the JettySolrRunner in 8x to have the magic 23 for min size, so I'll check this in on Monday afternoon absent objections. A couple of PRs shortly. > Upgrade Jetty to 9.4.32.v20200930 > - > > Key: SOLR-14844 > URL: https://issues.apache.org/jira/browse/SOLR-14844 > Project: Solr > Issue Type: Improvement >Affects Versions: 8.6 >Reporter: Cassandra Targett >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-14844-master.patch, SOLR-14884-8x.patch > > Time Spent: 20m > Remaining Estimate: 0h > > A CVE was found in Jetty 9.4.27-9.4.29 that has some security scanning tools > raising red flags > ([https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17638]). > Here's the Jetty issue: > [https://bugs.eclipse.org/bugs/show_bug.cgi?id=564984]. It's fixed in > 9.4.30+, so we should upgrade to that for 8.7 > -It has a simple mitigation (raise Jetty's responseHeaderSize to higher than > requestHeaderSize), but I don't know how Solr uses Jetty well enough to a) > know if this problem is even exploitable in Solr, or b) if the workaround > suggested is even possible in Solr.- > In normal Solr installs, w/o jetty optimizations, this issue is largely > mitigated in 8.6.3: see SOLR-14896 (and linked bug fixes) for details. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson commented on pull request #2003: SOLR-14844: Upgrade Jetty to 9.4.32.v20200930
ErickErickson commented on pull request #2003: URL: https://github.com/apache/lucene-solr/pull/2003#issuecomment-716047640 New one momentarily. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson closed pull request #2003: SOLR-14844: Upgrade Jetty to 9.4.32.v20200930
ErickErickson closed pull request #2003: URL: https://github.com/apache/lucene-solr/pull/2003 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson closed pull request #2021: SOLR-14844: upgrade jetty to 9.4.32.v20200930
ErickErickson closed pull request #2021: URL: https://github.com/apache/lucene-solr/pull/2021 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson commented on pull request #2021: SOLR-14844: upgrade jetty to 9.4.32.v20200930
ErickErickson commented on pull request #2021: URL: https://github.com/apache/lucene-solr/pull/2021#issuecomment-716047701 new one momentarily This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson opened a new pull request #2024: SOLR-14844: Upgrade Jetty to 9.4.32.v20200930 master
ErickErickson opened a new pull request #2024: URL: https://github.com/apache/lucene-solr/pull/2024 Fixes the checksum issue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ErickErickson opened a new pull request #2025: SOLR-14844: Upgrade Jetty to 9.4.32.v20200930 8x
ErickErickson opened a new pull request #2025: URL: https://github.com/apache/lucene-solr/pull/2025 fixes the test failure by setting min compression in JettySolrRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14963) Child "rows" param should apply per level
David Smiley created SOLR-14963:
---
Summary: Child "rows" param should apply per level
Key: SOLR-14963
URL: https://issues.apache.org/jira/browse/SOLR-14963
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: David Smiley
The {{[child rows=10]}} doc transformer "rows" param _should_ apply per parent,
and it's documented this way: "The maximum number of child documents to be
returned per parent document.". However, it is instead implemented as an
overall limit as the child documents are processed in a depth-first order way.
The implementation ought to change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #2016: SOLR-14067 v2 Move Stateless Scripting Update Process to /contrib
tflobbe commented on a change in pull request #2016:
URL: https://github.com/apache/lucene-solr/pull/2016#discussion_r511515809
##
File path:
solr/contrib/scripting-update-processor/src/java/org/apache/solr/update/processor/StatelessScriptUpdateProcessorFactory.java
##
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.update.processor;
+
+/**
+ * @deprecated use {@link ScriptUpdateProcessorFactory} instead. This class
will be removed in an upcoming release.
+ */
+@Deprecated
+public class StatelessScriptUpdateProcessorFactory extends
ScriptUpdateProcessorFactory {
Review comment:
Just an idea, but you could implement the constructor that logs the
deprecation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on pull request #2017: SOLR-14957: Add Prometheus Exporter to docker PATH. Fix classpath issues.
dsmiley commented on pull request #2017: URL: https://github.com/apache/lucene-solr/pull/2017#issuecomment-716091259 +1 LGTM; I see you are putting SLF4J JARs on the classpath, so I assume that was the issue you encountered? What's in conf? Oh the configuration... yes I ran into this issue and I solved it by setting the CWD from Docker to prometheus-exporter's dir. Please be sure to Q/A both manual execution (i.e. as if from a distribution) and via Docker. I thought you might have also added the env var overrides while you were at it but maybe you want to do that on another issue? You could do it here; IMO it fits with the scope of making prometheus-exporter easier to use via Docker than is today (possible but some quirks). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
