[jira] [Commented] (LUCENE-9283) DelimitedBoostTokenFilter can fail testRandomChains
[
https://issues.apache.org/jira/browse/LUCENE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062418#comment-17062418
]
Alan Woodward commented on LUCENE-9283:
---
Straightforward diff here:
{code:java}
diff --git
a/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestRandomChains.java
b/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestRandomChains.java
index be81c0acab2..f1ac4426a58 100644
---
a/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestRandomChains.java
+++
b/lucene/analysis/common/src/test/org/apache/lucene/analysis/core/TestRandomChains.java
@@ -62,6 +62,7 @@ import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.ValidatingTokenFilter;
+import org.apache.lucene.analysis.boost.DelimitedBoostTokenFilter;
import org.apache.lucene.analysis.charfilter.NormalizeCharMap;
import org.apache.lucene.analysis.cjk.CJKBigramFilter;
import org.apache.lucene.analysis.commongrams.CommonGramsFilter;
@@ -198,6 +199,8 @@ public class TestRandomChains extends
BaseTokenStreamTestCase {
WordDelimiterGraphFilter.class,
// requires a special encoded token value, so it may fail with
random data:
DelimitedTermFrequencyTokenFilter.class,
+ // requires a special encoded token value, so it may fail with
random data:
+ DelimitedBoostTokenFilter.class,
// clones of core's filters:
org.apache.lucene.analysis.core.StopFilter.class,
org.apache.lucene.analysis.core.LowerCaseFilter.class)) { {code}
> DelimitedBoostTokenFilter can fail testRandomChains
> ---
>
> Key: LUCENE-9283
> URL: https://issues.apache.org/jira/browse/LUCENE-9283
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
>
> DelimitedBoostTokenFilter expects tokens of the form `token` or
> `token|number` and throws a NumberFormatException if the `number` part can't
> be parsed. This can cause test failures when we build random chains and
> throw random data through them.
> We can either exclude DelimiteBoostTokenFilter when building a random
> analyzer, or add a flag to ignore badly-formed tokens. I lean towards doing
> the former, as I don't really want to make leniency the default here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9284) BKDWriter refactor: Extract methods that serialise the tree into its own class
Ignacio Vera created LUCENE-9284:
Summary: BKDWriter refactor: Extract methods that serialise the
tree into its own class
Key: LUCENE-9284
URL: https://issues.apache.org/jira/browse/LUCENE-9284
Project: Lucene - Core
Issue Type: Improvement
Reporter: Ignacio Vera
Currently the class {{BKDWriter}} contains the logic to build the tree and
serialise the tree. This makes the class difficult to understand and difficult
to maintain. In this issue it is only proposed to move the methods the
serialise the tree into the index to its own class called {{BKDIndexWriter}}.
This class contains just two methods, one to write the leaf nodes and one to
write the inner nodes.
While working on this, I wonder if the indexWriter should be an interface so we
can get rid of {{SimpleTextBKDWriter}} and just use a different serialisation
class for that case. I left that for some further improvement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase opened a new pull request #1363: LUCENE-9284: Refactor out logic that serialises the BKD tree
iverase opened a new pull request #1363: LUCENE-9284: Refactor out logic that serialises the BKD tree URL: https://github.com/apache/lucene-solr/pull/1363 Split BKDwriter into the logic that build the Kd-tree and the logic that serialises the tree into the index. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] iverase commented on a change in pull request #1363: LUCENE-9284: Refactor out logic that serialises the BKD tree
iverase commented on a change in pull request #1363: LUCENE-9284: Refactor out
logic that serialises the BKD tree
URL: https://github.com/apache/lucene-solr/pull/1363#discussion_r394954911
##
File path: lucene/core/src/java/org/apache/lucene/util/bkd/BKDIndexWriter.java
##
@@ -0,0 +1,474 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.util.bkd;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.function.IntFunction;
+
+import org.apache.lucene.codecs.CodecUtil;
+import org.apache.lucene.store.ByteBuffersDataOutput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefBuilder;
+
+/** Utility class providing methods to serialize a KD-tree.
+ *
+ * @lucene.experimental */
+public class BKDIndexWriter {
+
+ public static final String CODEC_NAME = "BKD";
+ public static final int VERSION_START = 4; // version used by Lucene 7.0
+ //public static final int VERSION_CURRENT = VERSION_START;
+ public static final int VERSION_LEAF_STORES_BOUNDS = 5;
+ public static final int VERSION_SELECTIVE_INDEXING = 6;
+ public static final int VERSION_LOW_CARDINALITY_LEAVES = 7;
+ public static final int VERSION_CURRENT = VERSION_LOW_CARDINALITY_LEAVES;
+
+ /** How many dimensions we are storing at the leaf (data) nodes */
+ private final int numDataDims;
+
+ /** How many dimensions we are indexing in the internal nodes */
+ private final int numIndexDims;
+
+ /** How many bytes each value in each dimension takes. */
+ private final int bytesPerDim;
+
+ /** numDataDims * bytesPerDim */
+ private final int packedBytesLength;
+
+ /** numIndexDims * bytesPerDim */
+ private final int packedIndexBytesLength;
+
+ private final int maxPointsInLeafNode;
+
+ private final byte[] scratch;
+
+ // Reused when writing leaf blocks
+ private final ByteBuffersDataOutput scratchOut =
ByteBuffersDataOutput.newResettableInstance();
+
+ BKDIndexWriter(int numDataDims, int numIndexDims, int bytesPerDim, int
maxPointsInLeafNode) {
+this.numDataDims = numDataDims;
+this.numIndexDims = numIndexDims;
+this.bytesPerDim = bytesPerDim;
+this.packedBytesLength = numDataDims * bytesPerDim;
+this.packedIndexBytesLength = numIndexDims * bytesPerDim;
+this.maxPointsInLeafNode = maxPointsInLeafNode;
+this.scratch = new byte[packedBytesLength];
+ }
+
+ /** writes a leaf block in the provided DataOutput */
+ public void writeLeafBlock(DataOutput out, int[] docIDs, int start, int
count,
+ int[] commonPrefixes, byte[] packedValue,
+ int sortedDim, IntFunction
packedValues, int leafCardinality) throws IOException {
+assert count > 0 : "count must be bigger than 0";
+assert count <= maxPointsInLeafNode: "maxPointsInLeafNode=" +
maxPointsInLeafNode + " > count=" + count;
+assert scratchOut.size() == 0;
+// Write docIDs first, as their own chunk, so that at intersect time we
can add all docIDs w/o
+// loading the values:
+writeLeafBlockDocs(scratchOut, docIDs, start, count);
+// Write common prefixes:
+writeCommonPrefixes(scratchOut, commonPrefixes, packedValue);
+// Write point values:
+writeLeafBlockPackedValues(scratchOut, commonPrefixes, count, sortedDim,
packedValues, leafCardinality);
+scratchOut.copyTo(out);
Review comment:
I wonder why we are using this ByteBuffersDataOutput?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9283) DelimitedBoostTokenFilter can fail testRandomChains
[ https://issues.apache.org/jira/browse/LUCENE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062549#comment-17062549 ] David Smiley commented on LUCENE-9283: -- +1 > DelimitedBoostTokenFilter can fail testRandomChains > --- > > Key: LUCENE-9283 > URL: https://issues.apache.org/jira/browse/LUCENE-9283 > Project: Lucene - Core > Issue Type: Bug >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > > DelimitedBoostTokenFilter expects tokens of the form `token` or > `token|number` and throws a NumberFormatException if the `number` part can't > be parsed. This can cause test failures when we build random chains and > throw random data through them. > We can either exclude DelimiteBoostTokenFilter when building a random > analyzer, or add a flag to ignore badly-formed tokens. I lean towards doing > the former, as I don't really want to make leniency the default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14348) Split TestJsonFacets to multiple Test Classes
[
https://issues.apache.org/jira/browse/SOLR-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N reassigned SOLR-14348:
---
Assignee: Munendra S N
> Split TestJsonFacets to multiple Test Classes
> -
>
> Key: SOLR-14348
> URL: https://issues.apache.org/jira/browse/SOLR-14348
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14348.patch
>
>
> {{TestJsonFacets}} has parameterized testing. It runs each tests for each
> facet.method. There are error cases which doesn't actually need it. Also,
> facet.method is applicable only to term facets.
> There are few Range facet tests which are present and runs repeatedly without
> any change(facet.method as no effect). Also, splitting would help when we
> introduce facet.method to range which would be different to term facets
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit
mikemccand commented on a change in pull request #1361: LUCENE-8118: Throw
exception if DWPT grows beyond it's maximum ram limit
URL: https://github.com/apache/lucene-solr/pull/1361#discussion_r395028755
##
File path:
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java
##
@@ -219,6 +222,9 @@ final void testPoint(String message) {
/** Anything that will add N docs to the index should reserve first to
* make sure it's allowed. */
private void reserveOneDoc() {
+if (numDocsInRAM != 0 && hardMaxBytesPerDWPT < bytesUsed()) { // TODO
should we check this after each field as well to also preempt within one doc?
Review comment:
Could you reverse the conditional to `bytesUsed() > hardMaxBytesPerDWPT`? I
think it's more quickly readable in that order?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit
mikemccand commented on a change in pull request #1361: LUCENE-8118: Throw
exception if DWPT grows beyond it's maximum ram limit
URL: https://github.com/apache/lucene-solr/pull/1361#discussion_r395006886
##
File path:
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java
##
@@ -219,6 +222,9 @@ final void testPoint(String message) {
/** Anything that will add N docs to the index should reserve first to
* make sure it's allowed. */
private void reserveOneDoc() {
+if (numDocsInRAM != 0 && hardMaxBytesPerDWPT < bytesUsed()) { // TODO
should we check this after each field as well to also preempt within one doc?
Review comment:
Or even per-token while inverting? I think we already detect this today and
throw aborting exception? But we could fix that to instead throw a
non-aborting exception, so the one document is deleted, but IW/DWPT are not
aborted. But we should do that later ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit
mikemccand commented on a change in pull request #1361: LUCENE-8118: Throw
exception if DWPT grows beyond it's maximum ram limit
URL: https://github.com/apache/lucene-solr/pull/1361#discussion_r395029178
##
File path:
lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThread.java
##
@@ -219,6 +222,9 @@ final void testPoint(String message) {
/** Anything that will add N docs to the index should reserve first to
* make sure it's allowed. */
private void reserveOneDoc() {
+if (numDocsInRAM != 0 && hardMaxBytesPerDWPT < bytesUsed()) { // TODO
should we check this after each field as well to also preempt within one doc?
+ throw new IllegalArgumentException("RAM used by a single
DocumentsWriterPerThread can not exceed: " + hardMaxBytesPerDWPT / 1024 / 1024
+ "MB");
Review comment:
s/`can not`/`cannot`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N reassigned SOLR-14344:
---
Assignee: Munendra S N
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N updated SOLR-14344:
Status: Patch Available (was: Open)
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N updated SOLR-14344:
Attachment: SOLR-14344.patch
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062567#comment-17062567
]
Munendra S N commented on SOLR-14344:
-
[^SOLR-14344.patch]
This removes deprecated exceptions and its usage
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14331) Support expand component on multiple fields in single request
[ https://issues.apache.org/jira/browse/SOLR-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Munendra S N updated SOLR-14331: Summary: Support expand component on multiple fields in single request (was: Support expand component on multiple fields in single field) > Support expand component on multiple fields in single request > - > > Key: SOLR-14331 > URL: https://issues.apache.org/jira/browse/SOLR-14331 > Project: Solr > Issue Type: Wish > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Munendra S N >Priority: Major > > SOLR-14073 adds support for multiple collapse and SOLR-14329 adds support to > choose collapse group for expand with multiple collapse. > Evaluate if it is possible to support expand on multiple collapse > groups/fields. Each field will contain different block in the response -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9283) DelimitedBoostTokenFilter can fail testRandomChains
[ https://issues.apache.org/jira/browse/LUCENE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062570#comment-17062570 ] ASF subversion and git services commented on LUCENE-9283: - Commit e4ab8410f0e481c9c0ba691bf9dbc194a7b6f7b1 in lucene-solr's branch refs/heads/branch_8x from Alan Woodward [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e4ab841 ] LUCENE-9283: Exclude DelimitedBoostTokenFilter from TestRandomChains > DelimitedBoostTokenFilter can fail testRandomChains > --- > > Key: LUCENE-9283 > URL: https://issues.apache.org/jira/browse/LUCENE-9283 > Project: Lucene - Core > Issue Type: Bug >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > > DelimitedBoostTokenFilter expects tokens of the form `token` or > `token|number` and throws a NumberFormatException if the `number` part can't > be parsed. This can cause test failures when we build random chains and > throw random data through them. > We can either exclude DelimiteBoostTokenFilter when building a random > analyzer, or add a flag to ignore badly-formed tokens. I lean towards doing > the former, as I don't really want to make leniency the default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9283) DelimitedBoostTokenFilter can fail testRandomChains
[ https://issues.apache.org/jira/browse/LUCENE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062573#comment-17062573 ] ASF subversion and git services commented on LUCENE-9283: - Commit 126e4a61b8679ac79ada5e0ad394622032ddc119 in lucene-solr's branch refs/heads/master from Alan Woodward [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=126e4a6 ] LUCENE-9283: Exclude DelimitedBoostTokenFilter from TestRandomChains > DelimitedBoostTokenFilter can fail testRandomChains > --- > > Key: LUCENE-9283 > URL: https://issues.apache.org/jira/browse/LUCENE-9283 > Project: Lucene - Core > Issue Type: Bug >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > > DelimitedBoostTokenFilter expects tokens of the form `token` or > `token|number` and throws a NumberFormatException if the `number` part can't > be parsed. This can cause test failures when we build random chains and > throw random data through them. > We can either exclude DelimiteBoostTokenFilter when building a random > analyzer, or add a flag to ignore badly-formed tokens. I lean towards doing > the former, as I don't really want to make leniency the default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9283) DelimitedBoostTokenFilter can fail testRandomChains
[ https://issues.apache.org/jira/browse/LUCENE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward resolved LUCENE-9283. --- Fix Version/s: 8.6 master (9.0) Resolution: Fixed > DelimitedBoostTokenFilter can fail testRandomChains > --- > > Key: LUCENE-9283 > URL: https://issues.apache.org/jira/browse/LUCENE-9283 > Project: Lucene - Core > Issue Type: Bug >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Fix For: master (9.0), 8.6 > > > DelimitedBoostTokenFilter expects tokens of the form `token` or > `token|number` and throws a NumberFormatException if the `number` part can't > be parsed. This can cause test failures when we build random chains and > throw random data through them. > We can either exclude DelimiteBoostTokenFilter when building a random > analyzer, or add a flag to ignore badly-formed tokens. I lean towards doing > the former, as I don't really want to make leniency the default here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11725) json.facet's stddev() function should be changed to use the "Corrected sample stddev" formula
[
https://issues.apache.org/jira/browse/SOLR-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062583#comment-17062583
]
ASF subversion and git services commented on SOLR-11725:
Commit e9d6c24fb786291fe1e0750acb371774bd09cc3d in lucene-solr's branch
refs/heads/master from Munendra S N
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e9d6c24 ]
SOLR-11725: use corrected sample formula to calc stdDev in JSON facets
* Both stdDev and variance uses corrected sample formula to compute
the values. This is similar to StatsComponent
> json.facet's stddev() function should be changed to use the "Corrected sample
> stddev" formula
> -
>
> Key: SOLR-11725
> URL: https://issues.apache.org/jira/browse/SOLR-11725
> Project: Solr
> Issue Type: Sub-task
> Components: Facet Module
>Reporter: Chris M. Hostetter
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-11725.patch, SOLR-11725.patch, SOLR-11725.patch
>
>
> While working on some equivalence tests/demonstrations for
> {{facet.pivot+stats.field}} vs {{json.facet}} I noticed that the {{stddev}}
> calculations done between the two code paths can be measurably different, and
> realized this is due to them using very different code...
> * {{json.facet=foo:stddev(foo)}}
> ** {{StddevAgg.java}}
> ** {{Math.sqrt((sumSq/count)-Math.pow(sum/count, 2))}}
> * {{stats.field=\{!stddev=true\}foo}}
> ** {{StatsValuesFactory.java}}
> ** {{Math.sqrt(((count * sumOfSquares) - (sum * sum)) / (count * (count -
> 1.0D)))}}
> Since I"m not really a math guy, I consulting with a bunch of smart math/stat
> nerds I know online to help me sanity check if these equations (some how)
> reduced to eachother (In which case the discrepancies I was seeing in my
> results might have just been due to the order of intermediate operation
> execution & floating point rounding differences).
> They confirmed that the two bits of code are _not_ equivalent to each other,
> and explained that the code JSON Faceting is using is equivalent to the
> "Uncorrected sample stddev" formula, while StatsComponent's code is
> equivalent to the the "Corrected sample stddev" formula...
> https://en.wikipedia.org/wiki/Standard_deviation#Uncorrected_sample_standard_deviation
> When I told them that stuff like this is why no one likes mathematicians and
> pressed them to explain which one was the "most canonical" (or "most
> generally applicable" or "best") definition of stddev, I was told that:
> # This is something statisticians frequently disagree on
> # Practically speaking the diff between the calculations doesn't tend to
> differ significantly when count is "very large"
> # _"Corrected sample stddev" is more appropriate when comparing two
> distributions_
> Given that:
> * the primary usage of computing the stddev of a field/function against a
> Solr result set (or against a sub-set of results defined by a facet
> constraint) is probably to compare that distribution to a different Solr
> result set (or to compare N sub-sets of results defined by N facet
> constraints)
> * the size of the sets of documents (values) can be relatively small when
> computing stats over facet constraint sub-sets
> ...it seems like {{StddevAgg.java}} should be updated to use the "Corrected
> sample stddev" equation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14012) Different data type for unique aggregation and countDistinct
[ https://issues.apache.org/jira/browse/SOLR-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062584#comment-17062584 ] ASF subversion and git services commented on SOLR-14012: Commit 78e670f19ec253fb5e9dc3980986feb501762995 in lucene-solr's branch refs/heads/master from Munendra S N [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=78e670f ] SOLR-14012: return long from unique and hll even for standalone > Different data type for unique aggregation and countDistinct > > > Key: SOLR-14012 > URL: https://issues.apache.org/jira/browse/SOLR-14012 > Project: Solr > Issue Type: Sub-task >Reporter: Munendra S N >Assignee: Munendra S N >Priority: Major > Attachments: SOLR-14012.patch, SOLR-14012.patch, SOLR-14012.patch > > > countDistinct value is long but unique aggregation's value is either long or > int depending on shard count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14012) Different data type for unique aggregation and countDistinct
[ https://issues.apache.org/jira/browse/SOLR-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Munendra S N updated SOLR-14012: Fix Version/s: master (9.0) Resolution: Fixed Status: Resolved (was: Patch Available) > Different data type for unique aggregation and countDistinct > > > Key: SOLR-14012 > URL: https://issues.apache.org/jira/browse/SOLR-14012 > Project: Solr > Issue Type: Sub-task >Reporter: Munendra S N >Assignee: Munendra S N >Priority: Major > Fix For: master (9.0) > > Attachments: SOLR-14012.patch, SOLR-14012.patch, SOLR-14012.patch > > > countDistinct value is long but unique aggregation's value is either long or > int depending on shard count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-11725) json.facet's stddev() function should be changed to use the "Corrected sample stddev" formula
[
https://issues.apache.org/jira/browse/SOLR-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N updated SOLR-11725:
Fix Version/s: master (9.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> json.facet's stddev() function should be changed to use the "Corrected sample
> stddev" formula
> -
>
> Key: SOLR-11725
> URL: https://issues.apache.org/jira/browse/SOLR-11725
> Project: Solr
> Issue Type: Sub-task
> Components: Facet Module
>Reporter: Chris M. Hostetter
>Assignee: Munendra S N
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-11725.patch, SOLR-11725.patch, SOLR-11725.patch
>
>
> While working on some equivalence tests/demonstrations for
> {{facet.pivot+stats.field}} vs {{json.facet}} I noticed that the {{stddev}}
> calculations done between the two code paths can be measurably different, and
> realized this is due to them using very different code...
> * {{json.facet=foo:stddev(foo)}}
> ** {{StddevAgg.java}}
> ** {{Math.sqrt((sumSq/count)-Math.pow(sum/count, 2))}}
> * {{stats.field=\{!stddev=true\}foo}}
> ** {{StatsValuesFactory.java}}
> ** {{Math.sqrt(((count * sumOfSquares) - (sum * sum)) / (count * (count -
> 1.0D)))}}
> Since I"m not really a math guy, I consulting with a bunch of smart math/stat
> nerds I know online to help me sanity check if these equations (some how)
> reduced to eachother (In which case the discrepancies I was seeing in my
> results might have just been due to the order of intermediate operation
> execution & floating point rounding differences).
> They confirmed that the two bits of code are _not_ equivalent to each other,
> and explained that the code JSON Faceting is using is equivalent to the
> "Uncorrected sample stddev" formula, while StatsComponent's code is
> equivalent to the the "Corrected sample stddev" formula...
> https://en.wikipedia.org/wiki/Standard_deviation#Uncorrected_sample_standard_deviation
> When I told them that stuff like this is why no one likes mathematicians and
> pressed them to explain which one was the "most canonical" (or "most
> generally applicable" or "best") definition of stddev, I was told that:
> # This is something statisticians frequently disagree on
> # Practically speaking the diff between the calculations doesn't tend to
> differ significantly when count is "very large"
> # _"Corrected sample stddev" is more appropriate when comparing two
> distributions_
> Given that:
> * the primary usage of computing the stddev of a field/function against a
> Solr result set (or against a sub-set of results defined by a facet
> constraint) is probably to compare that distribution to a different Solr
> result set (or to compare N sub-sets of results defined by N facet
> constraints)
> * the size of the sets of documents (values) can be relatively small when
> computing stats over facet constraint sub-sets
> ...it seems like {{StddevAgg.java}} should be updated to use the "Corrected
> sample stddev" equation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on issue #1316: LUCENE-8929 parallel early termination in TopFieldCollector using minmin score
msokolov commented on issue #1316: LUCENE-8929 parallel early termination in TopFieldCollector using minmin score URL: https://github.com/apache/lucene-solr/pull/1316#issuecomment-601199305 Thanks for the comments @jimczi - I'll get back to this soon, and have a look at the other PR you linked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14345) Error messages are not properly propagated with non-default response parsers
[
https://issues.apache.org/jira/browse/SOLR-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062608#comment-17062608
]
Munendra S N commented on SOLR-14345:
-
I was thinking to handle this in Parser but then, extracting {{error}}
shouldn't be part of parser and would create lot of duplicate code
[~hossman] [~dsmiley] [~mkhl] [~ichattopadhyaya] Any suggestion here, that
could handle it generically including NoOpResponseParser?
> Error messages are not properly propagated with non-default response parsers
>
>
> Key: SOLR-14345
> URL: https://issues.apache.org/jira/browse/SOLR-14345
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14345.patch, SOLR-14345.patch
>
>
> Default {{ResponsParseer}} is {{BinaryResponseParser}}. when non-default
> response parser is specified in the request then, the error message is
> propagated to user. This happens in solrCloud mode.
> I came across this problem when working on adding some test which uses
> {{SolrTestCaseHS}} but similar problem exists with SolrJ client
> Also, same problem exists in both HttpSolrClient and Http2SolrClient
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-12259) Robustly upgrade indexes
[ https://issues.apache.org/jira/browse/SOLR-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-12259: -- Description: This needs more careful planning, see the SIP here: https://cwiki.apache.org/confluence/display/SOLR/SIP-2+Support+safe+index+transformations+without+reindexing The general problem statement is that the current upgrade path is trappy and cumbersome. It would be a great help "in the field" to make the upgrade process less painful. Additionally one of the most common things users want to do is enable docValues, but currently they often have to re-index. Issues: 1> if I upgrade from 5x to 6x and then 7x, theres no guarantee that when I go to 7x all the segments have been rewritten in 6x format. Say I have a segment at max size that has no deletions. It'll never be rewritten until it has deleted docs. And perhaps 50% deleted docs currently. 2> IndexUpgraderTool explicitly does a forcemerge to 1 segment, which is bad. 3> in a large distributed system, running IndexUpgraderTool on all the nodes is cumbersome even if <2> is acceptable. 4> Users who realize specifying docValues on a field would be A Good Thing have to re-index. We have UninvertDocValuesMergePolicyFactory. Wouldn't it be nice to be able to have this done all at once without forceMerging to one segment. Proposal: Somehow avoid the above. Currently LUCENE-7976 is a start in that direction. It will make TMP respect max segments size so can avoid forceMerges that result in one segment. What it does _not_ do is rewrite segments with zero (or a small percentage) deleted documents. So it doesn't seem like a huge stretch to be able to specify to TMP the option to rewrite segments that have no deleted documents. Perhaps a new parameter to optimize? This would likely require another change to TMP or whatever. So upgrading to a new solr would look like 1> install the new Solr 2> execute "http://node:port/solr/collection_or_core/update?optimize=true&upgradeAllSegments=true"; What's not clear to me is whether we'd require UninvertDocValuesMergePolicyFactory to be specified and wrap TMP or not. Anyway, let's discuss. I'll create yet another LUCENE JIRA for TMP do rewrite all segments that I'll link. I'll also link several other JIRAs in here, they're coalescing. was: The general problem statement is that the current upgrade path is trappy and cumbersome. It would be a great help "in the field" to make the upgrade process less painful. Additionally one of the most common things users want to do is enable docValues, but currently they often have to re-index. Issues: 1> if I upgrade from 5x to 6x and then 7x, theres no guarantee that when I go to 7x all the segments have been rewritten in 6x format. Say I have a segment at max size that has no deletions. It'll never be rewritten until it has deleted docs. And perhaps 50% deleted docs currently. 2> IndexUpgraderTool explicitly does a forcemerge to 1 segment, which is bad. 3> in a large distributed system, running IndexUpgraderTool on all the nodes is cumbersome even if <2> is acceptable. 4> Users who realize specifying docValues on a field would be A Good Thing have to re-index. We have UninvertDocValuesMergePolicyFactory. Wouldn't it be nice to be able to have this done all at once without forceMerging to one segment. Proposal: Somehow avoid the above. Currently LUCENE-7976 is a start in that direction. It will make TMP respect max segments size so can avoid forceMerges that result in one segment. What it does _not_ do is rewrite segments with zero (or a small percentage) deleted documents. So it doesn't seem like a huge stretch to be able to specify to TMP the option to rewrite segments that have no deleted documents. Perhaps a new parameter to optimize? This would likely require another change to TMP or whatever. So upgrading to a new solr would look like 1> install the new Solr 2> execute "http://node:port/solr/collection_or_core/update?optimize=true&upgradeAllSegments=true"; What's not clear to me is whether we'd require UninvertDocValuesMergePolicyFactory to be specified and wrap TMP or not. Anyway, let's discuss. I'll create yet another LUCENE JIRA for TMP do rewrite all segments that I'll link. I'll also link several other JIRAs in here, they're coalescing. > Robustly upgrade indexes > > > Key: SOLR-12259 > URL: https://issues.apache.org/jira/browse/SOLR-12259 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12259.patch > > > This needs more careful planning, see the SIP here: > https://cwiki.apache.org/confluence/display/SOLR/SIP-2+Support+safe+index+transformations+without+reindexing > > The general problem statem
[jira] [Resolved] (SOLR-12259) Robustly upgrade indexes
[ https://issues.apache.org/jira/browse/SOLR-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved SOLR-12259. --- Resolution: Later We really need to work from the SIP see: [https://cwiki.apache.org/confluence/display/SOLR/SIP-2+Support+safe+index+transformations+without+reindexing] This will be a significant effort, let's do it right and design it first. > Robustly upgrade indexes > > > Key: SOLR-12259 > URL: https://issues.apache.org/jira/browse/SOLR-12259 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12259.patch > > > This needs more careful planning, see the SIP here: > https://cwiki.apache.org/confluence/display/SOLR/SIP-2+Support+safe+index+transformations+without+reindexing > > The general problem statement is that the current upgrade path is trappy and > cumbersome. It would be a great help "in the field" to make the upgrade > process less painful. > Additionally one of the most common things users want to do is enable > docValues, but currently they often have to re-index. > Issues: > 1> if I upgrade from 5x to 6x and then 7x, theres no guarantee that when I go > to 7x all the segments have been rewritten in 6x format. Say I have a segment > at max size that has no deletions. It'll never be rewritten until it has > deleted docs. And perhaps 50% deleted docs currently. > 2> IndexUpgraderTool explicitly does a forcemerge to 1 segment, which is bad. > 3> in a large distributed system, running IndexUpgraderTool on all the nodes > is cumbersome even if <2> is acceptable. > 4> Users who realize specifying docValues on a field would be A Good Thing > have to re-index. We have UninvertDocValuesMergePolicyFactory. Wouldn't it be > nice to be able to have this done all at once without forceMerging to one > segment. > Proposal: > Somehow avoid the above. Currently LUCENE-7976 is a start in that direction. > It will make TMP respect max segments size so can avoid forceMerges that > result in one segment. What it does _not_ do is rewrite segments with zero > (or a small percentage) deleted documents. > So it doesn't seem like a huge stretch to be able to specify to TMP the > option to rewrite segments that have no deleted documents. Perhaps a new > parameter to optimize? > This would likely require another change to TMP or whatever. > So upgrading to a new solr would look like > 1> install the new Solr > 2> execute > "http://node:port/solr/collection_or_core/update?optimize=true&upgradeAllSegments=true"; > What's not clear to me is whether we'd require > UninvertDocValuesMergePolicyFactory to be specified and wrap TMP or not. > Anyway, let's discuss. I'll create yet another LUCENE JIRA for TMP do rewrite > all segments that I'll link. > I'll also link several other JIRAs in here, they're coalescing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14345) Error messages are not properly propagated with non-default response parsers
[
https://issues.apache.org/jira/browse/SOLR-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062608#comment-17062608
]
Munendra S N edited comment on SOLR-14345 at 3/19/20, 2:12 PM:
---
I was thinking to handle this in Parser but then, extracting {{error}}
shouldn't be part of parser and would create lot of duplicate code.
Other approach is to create duplicate of InputStream and process it using
BinaryResponseParser and required parser. This kind of overkill and also
expensive
[~hossman] [~dsmiley] [~mkhl] [~ichattopadhyaya] Any suggestion here, that
could handle it generically including NoOpResponseParser?
was (Author: munendrasn):
I was thinking to handle this in Parser but then, extracting {{error}}
shouldn't be part of parser and would create lot of duplicate code
[~hossman] [~dsmiley] [~mkhl] [~ichattopadhyaya] Any suggestion here, that
could handle it generically including NoOpResponseParser?
> Error messages are not properly propagated with non-default response parsers
>
>
> Key: SOLR-14345
> URL: https://issues.apache.org/jira/browse/SOLR-14345
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14345.patch, SOLR-14345.patch
>
>
> Default {{ResponsParseer}} is {{BinaryResponseParser}}. when non-default
> response parser is specified in the request then, the error message is
> propagated to user. This happens in solrCloud mode.
> I came across this problem when working on adding some test which uses
> {{SolrTestCaseHS}} but similar problem exists with SolrJ client
> Also, same problem exists in both HttpSolrClient and Http2SolrClient
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy opened a new pull request #1364: SOLR-14335: Lock Solr's memory to prevent swapping
janhoy opened a new pull request #1364: SOLR-14335: Lock Solr's memory to prevent swapping URL: https://github.com/apache/lucene-solr/pull/1364 See https://issues.apache.org/jira/browse/SOLR-14335 Goal of this PR is to start Solr with a custom bootstrap class, which delegates to Jetty's main. In the custom bootstrap class, we will then implement memroy locking through JNA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14335) Lock Solr's memory to prevent swapping
[
https://issues.apache.org/jira/browse/SOLR-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062704#comment-17062704
]
Jan Høydahl commented on SOLR-14335:
Ok, I started a POC in [GitHub Pull Request
#1364|https://github.com/apache/lucene-solr/pull/1364]
The PR introduces a SolrBootstrap class and then replaces {{-jar start.jar}}
with {{-cp ".:solr.jar:start.jar"}} in bin/solr. All it does so far is to
demonstrate that we can start Solr with out own bootstrap class which then
delegates to Jetty's Main class. The SolrBootstrap class currently just prints
a line to stdout.
Next steps (help welcome):
* hook in some JNA code to lock memory
* ivy support (needs to place the solr.jar in server folder)
* replace all -jar start.jar in both bin/solr and bin/solr.cmd
* add a sys param and env.var to control memory locking on/off (or should we
always attempt to lock?)
> Lock Solr's memory to prevent swapping
> --
>
> Key: SOLR-14335
> URL: https://issues.apache.org/jira/browse/SOLR-14335
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Followup from SOLR-10306. While sometimes you are able to disable or reduce
> swap on the host, other times that is not so easy. Having a native option to
> lock Solr's memory would be beneficial.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1364: SOLR-14335: Lock Solr's memory to prevent swapping
dweiss commented on a change in pull request #1364: SOLR-14335: Lock Solr's
memory to prevent swapping
URL: https://github.com/apache/lucene-solr/pull/1364#discussion_r395134989
##
File path: solr/server/build.gradle
##
@@ -111,7 +112,12 @@ task assemblePackaging(type: Sync) {
into "lib/ext"
})
- from { project.configurations.startJar.singleFile } {
+ from("${buildDir}/libs", {
Review comment:
This doesn't seem to be right. you rename twice into the same solr.jar. And
I don't think it's elegant that the server depends on the launcher
(jetty-start). It's a smell.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14335) Lock Solr's memory to prevent swapping
[
https://issues.apache.org/jira/browse/SOLR-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062704#comment-17062704
]
Jan Høydahl edited comment on SOLR-14335 at 3/19/20, 3:58 PM:
--
Ok, I started a POC in [GitHub Pull Request
#1364|https://github.com/apache/lucene-solr/pull/1364]
The PR introduces a SolrBootstrap class and then replaces {{-jar start.jar}}
with {{-cp ".:solr.jar:start.jar"}} in bin/solr. All it does so far is to
demonstrate that we can start Solr with our own bootstrap class which then
delegates to Jetty's Main class. The SolrBootstrap class currently just prints
a line to stdout.
Next steps (help welcome):
* hook in some JNA code to lock memory
* ivy support (needs to place the solr.jar in server folder)
* replace all -jar start.jar in both bin/solr and bin/solr.cmd
* add a sys param and env.var to control memory locking on/off (or should we
always attempt to lock?)
* perhaps move some logic from start scripts into this class?, now that it
becomes a POJO entry point for starting Solr?
was (Author: janhoy):
Ok, I started a POC in [GitHub Pull Request
#1364|https://github.com/apache/lucene-solr/pull/1364]
The PR introduces a SolrBootstrap class and then replaces {{-jar start.jar}}
with {{-cp ".:solr.jar:start.jar"}} in bin/solr. All it does so far is to
demonstrate that we can start Solr with out own bootstrap class which then
delegates to Jetty's Main class. The SolrBootstrap class currently just prints
a line to stdout.
Next steps (help welcome):
* hook in some JNA code to lock memory
* ivy support (needs to place the solr.jar in server folder)
* replace all -jar start.jar in both bin/solr and bin/solr.cmd
* add a sys param and env.var to control memory locking on/off (or should we
always attempt to lock?)
> Lock Solr's memory to prevent swapping
> --
>
> Key: SOLR-14335
> URL: https://issues.apache.org/jira/browse/SOLR-14335
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Followup from SOLR-10306. While sometimes you are able to disable or reduce
> swap on the host, other times that is not so easy. Having a native option to
> lock Solr's memory would be beneficial.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1364: SOLR-14335: Lock Solr's memory to prevent swapping
janhoy commented on a change in pull request #1364: SOLR-14335: Lock Solr's
memory to prevent swapping
URL: https://github.com/apache/lucene-solr/pull/1364#discussion_r395138580
##
File path: solr/server/build.gradle
##
@@ -111,7 +112,12 @@ task assemblePackaging(type: Sync) {
into "lib/ext"
})
- from { project.configurations.startJar.singleFile } {
+ from("${buildDir}/libs", {
Review comment:
It's just a hack so far to get it building. I thought the bootstrap class
belongs in server module, but there is probably a better way to copy the lib in
place. Or we could have a completely new solr/bootstrap module with its own
build file just to produce solr.jar?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14335) Lock Solr's memory to prevent swapping
[ https://issues.apache.org/jira/browse/SOLR-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062711#comment-17062711 ] Dawid Weiss commented on SOLR-14335: >From gradle's point of view it'd make more sense to have a separate launcher >project which would depend on jetty-start. Then you can build a solr.jar with >proper manifest (classpath) and launch it as -jar solr.jar. And put everything >together in the packaging project... But I don't want to be the person to do >it in ant, to be honest. > Lock Solr's memory to prevent swapping > -- > > Key: SOLR-14335 > URL: https://issues.apache.org/jira/browse/SOLR-14335 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Jan Høydahl >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Followup from SOLR-10306. While sometimes you are able to disable or reduce > swap on the host, other times that is not so easy. Having a native option to > lock Solr's memory would be beneficial. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1364: SOLR-14335: Lock Solr's memory to prevent swapping
dweiss commented on a change in pull request #1364: SOLR-14335: Lock Solr's
memory to prevent swapping
URL: https://github.com/apache/lucene-solr/pull/1364#discussion_r395140443
##
File path: solr/server/build.gradle
##
@@ -111,7 +112,12 @@ task assemblePackaging(type: Sync) {
into "lib/ext"
})
- from { project.configurations.startJar.singleFile } {
+ from("${buildDir}/libs", {
Review comment:
I think a new module would make more sense (put a comment in jira).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14347) Autoscaling placement wrong when concurrent replica placements are calculated
[
https://issues.apache.org/jira/browse/SOLR-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated SOLR-14347:
Summary: Autoscaling placement wrong when concurrent replica placements are
calculated (was: Autoscaling placement wrong due to incorrect LockLevel for
collection modifications)
> Autoscaling placement wrong when concurrent replica placements are calculated
> -
>
> Key: SOLR-14347
> URL: https://issues.apache.org/jira/browse/SOLR-14347
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Affects Versions: 8.5
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Attachments: SOLR-14347.patch
>
>
> Steps to reproduce:
> * create a cluster of a few nodes (tested with 7 nodes)
> * define per-collection policies that distribute replicas exclusively on
> different nodes per policy
> * concurrently create a few collections, each using a different policy
> * resulting replica placement will be seriously wrong, causing many policy
> violations
> Running the same scenario but instead creating collections sequentially
> results in no violations.
> I suspect this is caused by incorrect locking level for all collection
> operations (as defined in {{CollectionParams.CollectionAction}}) that create
> new replica placements - i.e. CREATE, ADDREPLICA, MOVEREPLICA, DELETENODE,
> REPLACENODE, SPLITSHARD, RESTORE, REINDEXCOLLECTION. All of these operations
> use the policy engine to create new replica placements, and as a result they
> change the cluster state. However, currently these operations are locked (in
> {{OverseerCollectionMessageHandler.lockTask}} ) using
> {{LockLevel.COLLECTION}}. In practice this means that the lock is held only
> for the particular collection that is being modified.
> A straightforward fix for this issue is to change the locking level to
> CLUSTER (and I confirm this fixes the scenario described above). However,
> this effectively serializes all collection operations listed above, which
> will result in general slow-down of all collection operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] sigram opened a new pull request #1365: SOLR-14347: Autoscaling placement wrong when concurrent replica placements are calculated
sigram opened a new pull request #1365: SOLR-14347: Autoscaling placement wrong when concurrent replica placements are calculated URL: https://github.com/apache/lucene-solr/pull/1365 # Description Please provide a short description of the changes you're making with this pull request. # Solution Please provide a short description of the approach taken to implement your solution. # Tests Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [ ] I have developed this patch against the `master` branch. - [ ] I have run `ant precommit` and the appropriate test suite. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14340) ZkStateReader.readConfigName is doing too much work
[ https://issues.apache.org/jira/browse/SOLR-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062755#comment-17062755 ] David Smiley commented on SOLR-14340: - I think this question gets to the semantics of what CLUSTERSTATUS's job is. I think it should reflect back what is there – what's in DocCollection, aliases, what the collection's config set name is. That's it. I don't think it should be a deeper health check and thus shouldn't be reporting errors. I looked at SOLR-10720 and I think the root cause is actually _this issue_ – ZkStateReader is validating the config when it should simply return its name. If there is no config name, we can even return null and document this and inspect the callers. Once SOLR-14341 happens, the config name will be in the DocCollection and thus won't even be null. > ZkStateReader.readConfigName is doing too much work > --- > > Key: SOLR-14340 > URL: https://issues.apache.org/jira/browse/SOLR-14340 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > > ZkStateReader.readConfigName reads the configSet name given a collection name > parameter. Simple. It's doing too much work to do this though. First it's > calling zkClient.exists() which is redundant given that we then call getData > will will detect if it doesn't exist. Then we validate that the config path > exists. But I don't think that should be verified on read, only on write. > This method is a hotspot for nodes with lots of cores, proven out with > profiling. > Ideally the configSet name ought to be moved to state.json which is where all > the other collection metadata is and is thus already read by most spots that > want to read the configSet name. That would speed things up further and > generally simplify things as well. That can happen on a follow-on issue, I > think. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14325) Core status could be improved to not require an IndexSearcher
[ https://issues.apache.org/jira/browse/SOLR-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Goodman updated SOLR-14325: --- Attachment: SOLR-14325.patch > Core status could be improved to not require an IndexSearcher > - > > Key: SOLR-14325 > URL: https://issues.apache.org/jira/browse/SOLR-14325 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Priority: Major > Attachments: SOLR-14325.patch, SOLR-14325.patch > > > When the core status is told to request "indexInfo", it currently grabs the > SolrIndexSearcher but only to grab the Directory. SolrCore.getIndexSize also > only requires the Directory. By insisting on a SolrIndexSearcher, we > potentially block for awhile if the core is in recovery since there is no > SolrIndexSearcher. > [https://lists.apache.org/thread.html/r076218c964e9bd6ed0a53133be9170c3cf36cc874c1b4652120db417%40%3Cdev.lucene.apache.org%3E] > It'd be nice to have a solution that conditionally used the Directory of the > SolrIndexSearcher only if it's present so that we don't waste time creating > one either. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14325) Core status could be improved to not require an IndexSearcher
[
https://issues.apache.org/jira/browse/SOLR-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062758#comment-17062758
]
Richard Goodman commented on SOLR-14325:
Hey David,
I hope to bring good news to you today, I tried this on a cluster to the same
scale as our live production clusters _(luckily enough we got the headroom to
do this at the moment)_. And after applying your suggestion of using the
{{core.getIndexReaderFactory}} there is significant improvements in results.
The tests I did were;
* Restarting an instance to check recovery and core admin details
* Shutting down an instance, writing 100-200k documents to the collection,
increasing the numDocs on the remaining replicas for that given
collection+shard , started back up the instance to check recovery.
* Did a restore of a backup _(as this also previously failed with the original
patch)_
The following 3 tests worked well, and there were 0 errors in the logs _(where
as previously we were getting them)_, as an extra bonus. I also then ran the
solr prometheus exporter collecting jetty and core level metrics _(as
previously this would also fail for us)_, and they are collecting metrics with
0 error as well.
I have the following 3 data dumps just to also show that {{indexInfo}} is now
populating, and gets updated as well which I thought was pretty awesome:
{code:title=core-recovering-out-of-sync-num-docs.json}
"a_collection_shard14_replica_n73":{
"name":"a_collection_shard14_replica_n73",
"instanceDir":"/data/solr/data/a_collection_shard14_replica_n73",
"dataDir":"/data/solr/data/a_collection_shard14_replica_n73/data/",
"config":"solrconfig.xml",
"schema":"schema.xml",
"startTime":"2020-03-19T13:02:11.849Z",
"uptime":82787,
"lastPublished":"recovering",
"configVersion":0,
"cloud":{
"collection":"a_collection_",
"shard":"shard14",
"replica":"core_node74"},
"index":{
"numDocs":1634772,
"maxDoc":1645736,
"deletedDocs":10964,
"indexHeapUsageBytes":-1,
"version":6723987,
"segmentCount":26,
"current":true,
"hasDeletions":true,
"directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/data/solr/data/a_collection_shard14_replica_n73/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@8cf74fe;
maxCacheMB=48.0 maxMergeSizeMB=4.0)",
"segmentsFile":"segments_24cf",
"segmentsFileSizeInBytes":1937,
"userData":{
"commitCommandVer":"1661596698074415104",
"commitTimeMSec":"1584622095179"},
"lastModified":"2020-03-19T12:48:15.179Z",
"sizeInBytes":3969833777,
"size":"3.7 GB"}},
{code}
{code:title=core-recovering-getting-in-sync-with-num-docs}
"a_collection_shard14_replica_n73":{
"name":"a_collection_shard14_replica_n73",
"instanceDir":"/data/solr/data/a_collection_shard14_replica_n73",
"dataDir":"/data/solr/data/a_collection_shard14_replica_n73/data/",
"config":"solrconfig.xml",
"schema":"schema.xml",
"startTime":"2020-03-19T13:02:11.849Z",
"uptime":84929,
"lastPublished":"recovering",
"configVersion":0,
"cloud":{
"collection":"a_collection_",
"shard":"shard14",
"replica":"core_node74"},
"index":{
"numDocs":1714179,
"maxDoc":1725282,
"deletedDocs":11103,
"indexHeapUsageBytes":-1,
"version":6724780,
"segmentCount":25,
"current":true,
"hasDeletions":true,
"directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/data/solr/data/a_collection_shard14_replica_n73/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@8cf74fe;
maxCacheMB=48.0 maxMergeSizeMB=4.0)",
"segmentsFile":"segments_24cg",
"segmentsFileSizeInBytes":1868,
"userData":{
"commitCommandVer":"1661597662835638272",
"commitTimeMSec":"1584623015247"},
"lastModified":"2020-03-19T13:03:35.247Z",
"sizeInBytes":4858114166,
"size":"4.52 GB"}},
{code}
{code:title=core-now-active-with-in-sync-numDocs.json}
"a_collection_shard14_replica_n73":{
"name":"a_collection_shard14_replica_n73",
"instanceDir":"/data/solr/data/a_collection_shard14_replica_n73",
"dataDir":"/data/solr/data/a_collection_shard14_replica_n73/data/",
"config":"solrconfig.xml",
"schema":"schema.xml",
"startTime":"2020-03-19T13:02:11.849Z",
"uptime":93454,
"lastPublished":"active",
"configVersion":0,
"cloud":{
"collection":"a_collection",
"shard":"shard14",
"replica":"core_node74"},
"index":{
"numDocs":1714179,
"maxDoc":1725282,
"deletedDocs":11103,
"indexHeapUsageBytes":-
[jira] [Comment Edited] (SOLR-14325) Core status could be improved to not require an IndexSearcher
[
https://issues.apache.org/jira/browse/SOLR-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062758#comment-17062758
]
Richard Goodman edited comment on SOLR-14325 at 3/19/20, 4:57 PM:
--
Hey David,
I hope to bring good news to you today, I tried this on a cluster to the same
scale as our live production clusters _(luckily enough we got the headroom to
do this at the moment)_. And after applying your suggestion of using the
{{core.getIndexReaderFactory}} there is significant improvements in results.
The tests I did were;
* Restarting an instance to check recovery and core admin details
* Shutting down an instance, writing 100-200k documents to the collection,
increasing the numDocs on the remaining replicas for that given
collection+shard , started back up the instance to check recovery.
* Did a restore of a backup _(as this also previously failed with the original
patch)_
The following 3 tests worked well, and there were 0 errors in the logs _(where
as previously we were getting them)_, as an extra bonus. I also then ran the
solr prometheus exporter collecting jetty and core level metrics _(as
previously this would also fail for us)_, and they are collecting metrics with
0 error as well.
I have the following 3 data dumps just to also show that {{indexInfo}} is now
populating, and gets updated as well which I thought was pretty awesome:
{code:title=core-recovering-out-of-sync-num-docs.json}
"a_collection_shard14_replica_n73":{
"name":"a_collection_shard14_replica_n73",
"instanceDir":"/data/solr/data/a_collection_shard14_replica_n73",
"dataDir":"/data/solr/data/a_collection_shard14_replica_n73/data/",
"config":"solrconfig.xml",
"schema":"schema.xml",
"startTime":"2020-03-19T13:02:11.849Z",
"uptime":82787,
"lastPublished":"recovering",
"configVersion":0,
"cloud":{
"collection":"a_collection_",
"shard":"shard14",
"replica":"core_node74"},
"index":{
"numDocs":1634772,
"maxDoc":1645736,
"deletedDocs":10964,
"indexHeapUsageBytes":-1,
"version":6723987,
"segmentCount":26,
"current":true,
"hasDeletions":true,
"directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/data/solr/data/a_collection_shard14_replica_n73/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@8cf74fe;
maxCacheMB=48.0 maxMergeSizeMB=4.0)",
"segmentsFile":"segments_24cf",
"segmentsFileSizeInBytes":1937,
"userData":{
"commitCommandVer":"1661596698074415104",
"commitTimeMSec":"1584622095179"},
"lastModified":"2020-03-19T12:48:15.179Z",
"sizeInBytes":3969833777,
"size":"3.7 GB"}},
{code}
{code:title=core-recovering-getting-in-sync-with-num-docs.json}
"a_collection_shard14_replica_n73":{
"name":"a_collection_shard14_replica_n73",
"instanceDir":"/data/solr/data/a_collection_shard14_replica_n73",
"dataDir":"/data/solr/data/a_collection_shard14_replica_n73/data/",
"config":"solrconfig.xml",
"schema":"schema.xml",
"startTime":"2020-03-19T13:02:11.849Z",
"uptime":84929,
"lastPublished":"recovering",
"configVersion":0,
"cloud":{
"collection":"a_collection_",
"shard":"shard14",
"replica":"core_node74"},
"index":{
"numDocs":1714179,
"maxDoc":1725282,
"deletedDocs":11103,
"indexHeapUsageBytes":-1,
"version":6724780,
"segmentCount":25,
"current":true,
"hasDeletions":true,
"directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/data/solr/data/a_collection_shard14_replica_n73/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@8cf74fe;
maxCacheMB=48.0 maxMergeSizeMB=4.0)",
"segmentsFile":"segments_24cg",
"segmentsFileSizeInBytes":1868,
"userData":{
"commitCommandVer":"1661597662835638272",
"commitTimeMSec":"1584623015247"},
"lastModified":"2020-03-19T13:03:35.247Z",
"sizeInBytes":4858114166,
"size":"4.52 GB"}},
{code}
{code:title=core-now-active-with-in-sync-numDocs.json}
"a_collection_shard14_replica_n73":{
"name":"a_collection_shard14_replica_n73",
"instanceDir":"/data/solr/data/a_collection_shard14_replica_n73",
"dataDir":"/data/solr/data/a_collection_shard14_replica_n73/data/",
"config":"solrconfig.xml",
"schema":"schema.xml",
"startTime":"2020-03-19T13:02:11.849Z",
"uptime":93454,
"lastPublished":"active",
"configVersion":0,
"cloud":{
"collection":"a_collection",
"shard":"shard14",
"replica":"core_node74"},
"index":{
"numDocs":1714179,
"maxDoc":1725282,
[jira] [Created] (SOLR-14350) Reproducing failure in JsonRequestApiTest
Erick Erickson created SOLR-14350: - Summary: Reproducing failure in JsonRequestApiTest Key: SOLR-14350 URL: https://issues.apache.org/jira/browse/SOLR-14350 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Reporter: Erick Erickson ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991 -Dtests.file.encoding=ISO-8859-1 On current master. Fails with gradle too: ./gradlew :solr:solrj:test --tests "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest" -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1 > java.lang.AssertionError: expected: java.lang.Integer<3> but was: > java.lang.Long<3> > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0) > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:144) > at > org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14350) Reproducing failure in JsonRequestApiTest
[ https://issues.apache.org/jira/browse/SOLR-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Munendra S N reassigned SOLR-14350: --- Assignee: Munendra S N > Reproducing failure in JsonRequestApiTest > - > > Key: SOLR-14350 > URL: https://issues.apache.org/jira/browse/SOLR-14350 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Munendra S N >Priority: Major > > ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991 > -Dtests.file.encoding=ISO-8859-1 > > On current master. Fails with gradle too: > ./gradlew :solr:solrj:test --tests > "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest" > -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1 > > 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991]) > [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1 > > java.lang.AssertionError: expected: java.lang.Integer<3> but was: > java.lang.Long<3> > > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0) > > at org.junit.Assert.fail(Assert.java:88) > > at org.junit.Assert.failNotEquals(Assert.java:834) > > at org.junit.Assert.assertEquals(Assert.java:118) > > at org.junit.Assert.assertEquals(Assert.java:144) > > at > org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14350) Reproducing failure in JsonRequestApiTest
[ https://issues.apache.org/jira/browse/SOLR-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062772#comment-17062772 ] Munendra S N commented on SOLR-14350: - [~erickerickson] Thanks for reporting this. This is caused due to SOLR-14012 (which i committed today). I'm not sure why pre-commit solr build didn't catch this. I will fix this right away > Reproducing failure in JsonRequestApiTest > - > > Key: SOLR-14350 > URL: https://issues.apache.org/jira/browse/SOLR-14350 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Priority: Major > > ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991 > -Dtests.file.encoding=ISO-8859-1 > > On current master. Fails with gradle too: > ./gradlew :solr:solrj:test --tests > "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest" > -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1 > > 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991]) > [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1 > > java.lang.AssertionError: expected: java.lang.Integer<3> but was: > java.lang.Long<3> > > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0) > > at org.junit.Assert.fail(Assert.java:88) > > at org.junit.Assert.failNotEquals(Assert.java:834) > > at org.junit.Assert.assertEquals(Assert.java:118) > > at org.junit.Assert.assertEquals(Assert.java:144) > > at > org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14350) Reproducing failure in JsonRequestApiTest
[ https://issues.apache.org/jira/browse/SOLR-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062776#comment-17062776 ] ASF subversion and git services commented on SOLR-14350: Commit e36733d01d0b98d40dfbf9b80e8823bfc33d2c30 in lucene-solr's branch refs/heads/master from Munendra S N [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e36733d ] SOLR-14350: fix test failure due to SOLR-14012 > Reproducing failure in JsonRequestApiTest > - > > Key: SOLR-14350 > URL: https://issues.apache.org/jira/browse/SOLR-14350 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Munendra S N >Priority: Major > > ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991 > -Dtests.file.encoding=ISO-8859-1 > > On current master. Fails with gradle too: > ./gradlew :solr:solrj:test --tests > "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest" > -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1 > > 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991]) > [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1 > > java.lang.AssertionError: expected: java.lang.Integer<3> but was: > java.lang.Long<3> > > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0) > > at org.junit.Assert.fail(Assert.java:88) > > at org.junit.Assert.failNotEquals(Assert.java:834) > > at org.junit.Assert.assertEquals(Assert.java:118) > > at org.junit.Assert.assertEquals(Assert.java:144) > > at > org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14012) Different data type for unique aggregation and countDistinct
[ https://issues.apache.org/jira/browse/SOLR-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062777#comment-17062777 ] ASF subversion and git services commented on SOLR-14012: Commit e36733d01d0b98d40dfbf9b80e8823bfc33d2c30 in lucene-solr's branch refs/heads/master from Munendra S N [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e36733d ] SOLR-14350: fix test failure due to SOLR-14012 > Different data type for unique aggregation and countDistinct > > > Key: SOLR-14012 > URL: https://issues.apache.org/jira/browse/SOLR-14012 > Project: Solr > Issue Type: Sub-task >Reporter: Munendra S N >Assignee: Munendra S N >Priority: Major > Fix For: master (9.0) > > Attachments: SOLR-14012.patch, SOLR-14012.patch, SOLR-14012.patch > > > countDistinct value is long but unique aggregation's value is either long or > int depending on shard count -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14350) Reproducing failure in JsonRequestApiTest
[ https://issues.apache.org/jira/browse/SOLR-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062778#comment-17062778 ] Munendra S N commented on SOLR-14350: - I have fixed the failing test case, will resolves once master build succeeds > Reproducing failure in JsonRequestApiTest > - > > Key: SOLR-14350 > URL: https://issues.apache.org/jira/browse/SOLR-14350 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Munendra S N >Priority: Major > > ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991 > -Dtests.file.encoding=ISO-8859-1 > > On current master. Fails with gradle too: > ./gradlew :solr:solrj:test --tests > "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest" > -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1 > > 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991]) > [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1 > > java.lang.AssertionError: expected: java.lang.Integer<3> but was: > java.lang.Long<3> > > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0) > > at org.junit.Assert.fail(Assert.java:88) > > at org.junit.Assert.failNotEquals(Assert.java:834) > > at org.junit.Assert.assertEquals(Assert.java:118) > > at org.junit.Assert.assertEquals(Assert.java:144) > > at > org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14350) Reproducing failure in JsonRequestApiTest
[
https://issues.apache.org/jira/browse/SOLR-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062783#comment-17062783
]
Munendra S N commented on SOLR-14350:
-
Based on this,
https://builds.apache.org/job/PreCommit-SOLR-Build/706/testReport/
It doesn't look like {{org.apache.solr.client.ref_guide_examples}} are being
run as part of build
> Reproducing failure in JsonRequestApiTest
> -
>
> Key: SOLR-14350
> URL: https://issues.apache.org/jira/browse/SOLR-14350
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Munendra S N
>Priority: Major
>
> ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991
> -Dtests.file.encoding=ISO-8859-1
>
> On current master. Fails with gradle too:
> ./gradlew :solr:solrj:test --tests
> "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest"
> -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1
>
> 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991])
> [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1
> > java.lang.AssertionError: expected: java.lang.Integer<3> but was:
> java.lang.Long<3>
> > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0)
> > at org.junit.Assert.fail(Assert.java:88)
> > at org.junit.Assert.failNotEquals(Assert.java:834)
> > at org.junit.Assert.assertEquals(Assert.java:118)
> > at org.junit.Assert.assertEquals(Assert.java:144)
> > at
> org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470)
> > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
> >
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14340) ZkStateReader.readConfigName is doing too much work
[ https://issues.apache.org/jira/browse/SOLR-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062800#comment-17062800 ] Erick Erickson commented on SOLR-14340: --- "I think it should reflect back what is there – what's in DocCollection, aliases, what the collection's config set name is. That's it." Well, we don't even do that currently. If the config isn't there, we omit all the information associated with the collection with the current code. That's something of a nit though given SOLR-14341, just sayin'. What we really need is out of scope; a way to query Solr for as many weird things as we can think of. > ZkStateReader.readConfigName is doing too much work > --- > > Key: SOLR-14340 > URL: https://issues.apache.org/jira/browse/SOLR-14340 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > > ZkStateReader.readConfigName reads the configSet name given a collection name > parameter. Simple. It's doing too much work to do this though. First it's > calling zkClient.exists() which is redundant given that we then call getData > will will detect if it doesn't exist. Then we validate that the config path > exists. But I don't think that should be verified on read, only on write. > This method is a hotspot for nodes with lots of cores, proven out with > profiling. > Ideally the configSet name ought to be moved to state.json which is where all > the other collection metadata is and is thus already read by most spots that > want to read the configSet name. That would speed things up further and > generally simplify things as well. That can happen on a follow-on issue, I > think. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley opened a new pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley opened a new pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud. URL: https://github.com/apache/lucene-solr/pull/1366 https://issues.apache.org/jira/browse/SOLR-14342 (see the issue). I chose to rewrite much of the test to give me greater confidence that it works as designed. If I, for example, comment out much of CoreSorter.init then the test fails. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14350) Reproducing failure in JsonRequestApiTest
[ https://issues.apache.org/jira/browse/SOLR-14350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062810#comment-17062810 ] Erick Erickson commented on SOLR-14350: --- Worked for me, thanks for getting to it so quickly. > Reproducing failure in JsonRequestApiTest > - > > Key: SOLR-14350 > URL: https://issues.apache.org/jira/browse/SOLR-14350 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Munendra S N >Priority: Major > > ant test -Dtestcase=JsonRequestApiTest -Dtests.seed=2E1298D05407991 > -Dtests.file.encoding=ISO-8859-1 > > On current master. Fails with gradle too: > ./gradlew :solr:solrj:test --tests > "org.apache.solr.client.ref_guide_examples.JsonRequestApiTest" > -Ptests.seed=2E1298D05407991 -Ptests.file.encoding=ISO-8859-1 > > 2> 5655 INFO (TEST-JsonRequestApiTest.testStatFacet1-seed#[2E1298D05407991]) > [ ] o.a.s.SolrTestCaseJ4 ###Ending testStatFacet1 > > java.lang.AssertionError: expected: java.lang.Integer<3> but was: > java.lang.Long<3> > > at __randomizedtesting.SeedInfo.seed([2E1298D05407991:185168F18FADF3BB]:0) > > at org.junit.Assert.fail(Assert.java:88) > > at org.junit.Assert.failNotEquals(Assert.java:834) > > at org.junit.Assert.assertEquals(Assert.java:118) > > at org.junit.Assert.assertEquals(Assert.java:144) > > at > org.apache.solr.client.ref_guide_examples.JsonRequestApiTest.testStatFacet1(JsonRequestApiTest.java:470) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14325) Core status could be improved to not require an IndexSearcher
[
https://issues.apache.org/jira/browse/SOLR-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062852#comment-17062852
]
David Smiley commented on SOLR-14325:
-
It's strange to me that using IndexReaderFactory suddenly made things better;
I'm suspicious. The only implementation trivially calls
{{DirectoryReader.open(indexDir);}} which is what you were doing before in the
first patch, and led to the exception sometimes about a missing file.
I see that you added SolrCore.getDirectory() but the callers all need to be
careful to call directoryFactory.release(Directory) on it, so it's kind of a
leaky API and I don't like it as-is. I also don't like getters that actually
create brand new things. As an alternative, maybe consider a "withDirectory"
and a lambda, similar to the withSearcher method. In CoreAdminOperation, the
line where you compute the size, you are calling getDirectory and thus actually
getting a brand new Directory (plus failed to release it!). This is wasteful;
instead call dirReader.directory() since there is _already_ a directory
available here.
Still; code feedback aside, I'm suspicious that there won't be more mysterious
IOExceptions about a file vanishing. You needn't share the JSONs; I believe
you.
Were you able to see if \{{getNewestSearcher(false)}} returns quickly or not
during replication?
> Core status could be improved to not require an IndexSearcher
> -
>
> Key: SOLR-14325
> URL: https://issues.apache.org/jira/browse/SOLR-14325
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
> Attachments: SOLR-14325.patch, SOLR-14325.patch
>
>
> When the core status is told to request "indexInfo", it currently grabs the
> SolrIndexSearcher but only to grab the Directory. SolrCore.getIndexSize also
> only requires the Directory. By insisting on a SolrIndexSearcher, we
> potentially block for awhile if the core is in recovery since there is no
> SolrIndexSearcher.
> [https://lists.apache.org/thread.html/r076218c964e9bd6ed0a53133be9170c3cf36cc874c1b4652120db417%40%3Cdev.lucene.apache.org%3E]
> It'd be nice to have a solution that conditionally used the Directory of the
> SolrIndexSearcher only if it's present so that we don't waste time creating
> one either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062857#comment-17062857
]
Lucene/Solr QA commented on SOLR-14344:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} solrj in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} solrj in the patch failed. {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 8s{color} | {color:green} Release audit (RAT) rat-sources
passed {color} |
| {color:red}-1{color} | {color:red} Release audit (RAT) {color} | {color:red}
0m 3s{color} | {color:red} solrj in the patch failed. {color} |
| {color:red}-1{color} | {color:red} Check forbidden APIs {color} | {color:red}
1m 8s{color} | {color:red} Check forbidden APIs check-forbidden-apis failed
{color} |
| {color:red}-1{color} | {color:red} Validate source patterns {color} |
{color:red} 1m 8s{color} | {color:red} Check forbidden APIs
check-forbidden-apis failed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 46m
33s{color} | {color:green} core in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 13s{color}
| {color:red} solrj in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-14344 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12997120/SOLR-14344.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / e36733d01d0 |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| compile |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/artifact/out/patch-compile-solr_solrj.txt
|
| javac |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/artifact/out/patch-compile-solr_solrj.txt
|
| Release audit (RAT) |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/artifact/out/patch-compile-solr_solrj.txt
|
| Check forbidden APIs |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/artifact/out/patch-check-forbidden-apis-solr.txt
|
| Validate source patterns |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/artifact/out/patch-check-forbidden-apis-solr.txt
|
| unit |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/artifact/out/patch-unit-solr_solrj.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/testReport/ |
| modules | C: solr/core solr/solrj U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/721/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N updated SOLR-14344:
Attachment: SOLR-14344.patch
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch, SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062870#comment-17062870
]
Munendra S N commented on SOLR-14344:
-
[^SOLR-14344.patch]
Fixed import issue which caused build failure
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch, SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12353) SolrDispatchFilter expensive non-conditional debug line degrades performance
[
https://issues.apache.org/jira/browse/SOLR-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062889#comment-17062889
]
ASF subversion and git services commented on SOLR-12353:
Commit 5fd55d77e9f5e3da620171e4348a9450c2ee3482 in lucene-solr's branch
refs/heads/master from Erick Erickson
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5fd55d7 ]
SOLR-12353: SolrDispatchFilter expensive non-conditional debug line degrades
performance
> SolrDispatchFilter expensive non-conditional debug line degrades performance
>
>
> Key: SOLR-12353
> URL: https://issues.apache.org/jira/browse/SOLR-12353
> Project: Solr
> Issue Type: Bug
> Components: Admin UI, Authentication, logging
>Affects Versions: 6.6.3
>Reporter: Pascal Proulx
>Assignee: Erick Erickson
>Priority: Major
>
> Hello,
> We use Solr 6.6.3. Recently on one network when switching on authentication
> (security.json) began experiencing significant delays (5-10 seconds) to
> fulfill each request to /solr index.
> I debugged the issue and it was essentially triggered by line 456 of
> SolrDispatchFilter.java:
> {code:java}
> log.debug("Request to authenticate: {}, domain: {}, port: {}", request,
> request.getLocalName(), request.getLocalPort());
> {code}
> The issue is that on machines and networks with poor configuration or DNS
> issues in particular, request.getLocalName() can trigger expensive reverse
> DNS queries for the ethernet interfaces, and will only return within
> reasonable timeframe if manually written into /etc/hosts.
> More to the point, request.getLocalName() should be considered an expensive
> operation in general, and in SolrDispatchFilter it runs unconditionally even
> if debug is disabled.
> I would suggest to either replace request.getLocalName/Port here, or at the
> least, wrap the debug operation so it doesn't affect any production systems:
> {code:java}
> if (log.isDebugEnabled()) {
> log.debug("Request to authenticate: {}, domain: {}, port: {}", request,
> request.getLocalName(), request.getLocalPort());
> }
> {code}
> The authenticateRequest method in question is private so we could not
> override it and making another HttpServletRequestWrapper to circumvent the
> servlet API was doubtful.
> Thank you
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12353) SolrDispatchFilter expensive non-conditional debug line degrades performance
[
https://issues.apache.org/jira/browse/SOLR-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062892#comment-17062892
]
ASF subversion and git services commented on SOLR-12353:
Commit 338784214525a3c92a5c9593d2db948c7b40d3b0 in lucene-solr's branch
refs/heads/branch_8x from Erick Erickson
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3387842 ]
SOLR-12353: SolrDispatchFilter expensive non-conditional debug line degrades
performance
(cherry picked from commit 5fd55d77e9f5e3da620171e4348a9450c2ee3482)
> SolrDispatchFilter expensive non-conditional debug line degrades performance
>
>
> Key: SOLR-12353
> URL: https://issues.apache.org/jira/browse/SOLR-12353
> Project: Solr
> Issue Type: Bug
> Components: Admin UI, Authentication, logging
>Affects Versions: 6.6.3
>Reporter: Pascal Proulx
>Assignee: Erick Erickson
>Priority: Major
>
> Hello,
> We use Solr 6.6.3. Recently on one network when switching on authentication
> (security.json) began experiencing significant delays (5-10 seconds) to
> fulfill each request to /solr index.
> I debugged the issue and it was essentially triggered by line 456 of
> SolrDispatchFilter.java:
> {code:java}
> log.debug("Request to authenticate: {}, domain: {}, port: {}", request,
> request.getLocalName(), request.getLocalPort());
> {code}
> The issue is that on machines and networks with poor configuration or DNS
> issues in particular, request.getLocalName() can trigger expensive reverse
> DNS queries for the ethernet interfaces, and will only return within
> reasonable timeframe if manually written into /etc/hosts.
> More to the point, request.getLocalName() should be considered an expensive
> operation in general, and in SolrDispatchFilter it runs unconditionally even
> if debug is disabled.
> I would suggest to either replace request.getLocalName/Port here, or at the
> least, wrap the debug operation so it doesn't affect any production systems:
> {code:java}
> if (log.isDebugEnabled()) {
> log.debug("Request to authenticate: {}, domain: {}, port: {}", request,
> request.getLocalName(), request.getLocalPort());
> }
> {code}
> The authenticateRequest method in question is private so we could not
> override it and making another HttpServletRequestWrapper to circumvent the
> servlet API was doubtful.
> Thank you
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12353) SolrDispatchFilter expensive non-conditional debug line degrades performance
[
https://issues.apache.org/jira/browse/SOLR-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson resolved SOLR-12353.
---
Fix Version/s: 8.6
Resolution: Fixed
> SolrDispatchFilter expensive non-conditional debug line degrades performance
>
>
> Key: SOLR-12353
> URL: https://issues.apache.org/jira/browse/SOLR-12353
> Project: Solr
> Issue Type: Bug
> Components: Admin UI, Authentication, logging
>Affects Versions: 6.6.3
>Reporter: Pascal Proulx
>Assignee: Erick Erickson
>Priority: Major
> Fix For: 8.6
>
>
> Hello,
> We use Solr 6.6.3. Recently on one network when switching on authentication
> (security.json) began experiencing significant delays (5-10 seconds) to
> fulfill each request to /solr index.
> I debugged the issue and it was essentially triggered by line 456 of
> SolrDispatchFilter.java:
> {code:java}
> log.debug("Request to authenticate: {}, domain: {}, port: {}", request,
> request.getLocalName(), request.getLocalPort());
> {code}
> The issue is that on machines and networks with poor configuration or DNS
> issues in particular, request.getLocalName() can trigger expensive reverse
> DNS queries for the ethernet interfaces, and will only return within
> reasonable timeframe if manually written into /etc/hosts.
> More to the point, request.getLocalName() should be considered an expensive
> operation in general, and in SolrDispatchFilter it runs unconditionally even
> if debug is disabled.
> I would suggest to either replace request.getLocalName/Port here, or at the
> least, wrap the debug operation so it doesn't affect any production systems:
> {code:java}
> if (log.isDebugEnabled()) {
> log.debug("Request to authenticate: {}, domain: {}, port: {}", request,
> request.getLocalName(), request.getLocalPort());
> }
> {code}
> The authenticateRequest method in question is private so we could not
> override it and making another HttpServletRequestWrapper to circumvent the
> servlet API was doubtful.
> Thank you
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395283880
##
File path: lucene/core/src/java/org/apache/lucene/search/Weight.java
##
@@ -201,20 +201,54 @@ public long cost() {
@Override
public int score(LeafCollector collector, Bits acceptDocs, int min, int
max) throws IOException {
collector.setScorer(scorer);
+ DocIdSetIterator scorerIterator = twoPhase == null? iterator:
twoPhase.approximation();
+ DocIdSetIterator combinedIterator = collector.iterator() == null ?
scorerIterator: combineScorerAndCollectorIterators(scorerIterator, collector);
if (scorer.docID() == -1 && min == 0 && max ==
DocIdSetIterator.NO_MORE_DOCS) {
-scoreAll(collector, iterator, twoPhase, acceptDocs);
+scoreAll(collector, combinedIterator, twoPhase, acceptDocs);
return DocIdSetIterator.NO_MORE_DOCS;
} else {
int doc = scorer.docID();
-if (doc < min) {
- if (twoPhase == null) {
-doc = iterator.advance(min);
- } else {
-doc = twoPhase.approximation().advance(min);
+if (doc < min) scorerIterator.advance(min);
Review comment:
Addressed in
https://github.com/apache/lucene-solr/pull/1351/commits/d732d7eb9de67a597f67e91c9774104aa055e293
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395284073
##
File path: lucene/core/src/java/org/apache/lucene/search/Weight.java
##
@@ -223,44 +257,23 @@ public int score(LeafCollector collector, Bits
acceptDocs, int min, int max) thr
* See https://issues.apache.org/jira/browse/LUCENE-5487";>LUCENE-5487 */
static int scoreRange(LeafCollector collector, DocIdSetIterator iterator,
TwoPhaseIterator twoPhase,
Bits acceptDocs, int currentDoc, int end) throws IOException {
- if (twoPhase == null) {
-while (currentDoc < end) {
- if (acceptDocs == null || acceptDocs.get(currentDoc)) {
-collector.collect(currentDoc);
- }
- currentDoc = iterator.nextDoc();
-}
-return currentDoc;
- } else {
-final DocIdSetIterator approximation = twoPhase.approximation();
-while (currentDoc < end) {
- if ((acceptDocs == null || acceptDocs.get(currentDoc)) &&
twoPhase.matches()) {
-collector.collect(currentDoc);
- }
- currentDoc = approximation.nextDoc();
+ while (currentDoc < end) {
+if ((acceptDocs == null || acceptDocs.get(currentDoc)) && (twoPhase ==
null || twoPhase.matches())) {
+ collector.collect(currentDoc);
}
-return currentDoc;
+currentDoc = iterator.nextDoc();
}
+ return currentDoc;
Review comment:
Addressed in
https://github.com/apache/lucene-solr/pull/1351/commits/d732d7eb9de67a597f67e91c9774104aa055e293
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395284149
##
File path: lucene/core/src/java/org/apache/lucene/search/Weight.java
##
@@ -201,20 +201,54 @@ public long cost() {
@Override
public int score(LeafCollector collector, Bits acceptDocs, int min, int
max) throws IOException {
collector.setScorer(scorer);
+ DocIdSetIterator scorerIterator = twoPhase == null? iterator:
twoPhase.approximation();
+ DocIdSetIterator combinedIterator = collector.iterator() == null ?
scorerIterator: combineScorerAndCollectorIterators(scorerIterator, collector);
if (scorer.docID() == -1 && min == 0 && max ==
DocIdSetIterator.NO_MORE_DOCS) {
-scoreAll(collector, iterator, twoPhase, acceptDocs);
+scoreAll(collector, combinedIterator, twoPhase, acceptDocs);
return DocIdSetIterator.NO_MORE_DOCS;
} else {
int doc = scorer.docID();
-if (doc < min) {
- if (twoPhase == null) {
-doc = iterator.advance(min);
- } else {
-doc = twoPhase.approximation().advance(min);
+if (doc < min) scorerIterator.advance(min);
+return scoreRange(collector, combinedIterator, twoPhase, acceptDocs,
doc, max);
+ }
+}
+
+// conjunction iterator between scorer's iterator and collector's iterator
Review comment:
https://github.com/apache/lucene-solr/pull/1351/commits/d732d7eb9de67a597f67e91c9774104aa055e293
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
@jimczi I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I will keep thinking how to organize it better
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Ser
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I will keep thinking how to organize it better
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
--
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
@jimczi I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I will keep thinking how to organize it better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Se
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
@jimczi I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I think it is better to have a previous implementation with a dedicate
`updateIterator` function called from `TopFieldCollector`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, ple
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
@jimczi I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
Do you know the reason why `hitsThresholdChecker.isThresholdReached` checks
for greater than numHits and not greater or equal numHits?
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I think it is better to have a previous implementation with a dedicate
`updateIterator` function called from `TopFieldCollector`.
This is an automated message from the Apache Git Service.
T
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
@jimczi I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
Do you know the reason why `hitsThresholdChecker.isThresholdReached` checks
for greater than numHits and not greater or equal numHits?
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I think it is better to have a previous implementation with a dedicated
`updateIterator` function called from `TopFieldCollector`.
This is an automated message from the Apache Git Service.
[jira] [Commented] (SOLR-14336) Warn users running old version of Solr
[ https://issues.apache.org/jira/browse/SOLR-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062915#comment-17062915 ] Cassandra Targett commented on SOLR-14336: -- I'm confused how this would work - not saying there wouldn't be a benefit, just not sure how you get an old version like 4.x or 5.x to know it's really old without doing a release of that old version? We could bake something in going forward, my question is about the versions you mentioned as currently needing this functionality. > Warn users running old version of Solr > -- > > Key: SOLR-14336 > URL: https://issues.apache.org/jira/browse/SOLR-14336 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Jan Høydahl >Priority: Major > > There are obviously many very old Solr installs out there. People are still > reporting issues for Solr 4.x and 5.x. This is a proposal that Solr will > print a warning in the logs and display a warning in the Admin UI Dashboard > when running a (probably) outdated version. > I do not aim to "call home" to Apache to check versions, but instead parse > release date from 'solr-impl-version' string and warn if more than 1 year > old. That should be very conservative as there will almost certainly be a > handful new releases, with potential security fixes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1351: LUCENE-9280: Collectors to skip noncompetitive documents
mayya-sharipova commented on a change in pull request #1351: LUCENE-9280:
Collectors to skip noncompetitive documents
URL: https://github.com/apache/lucene-solr/pull/1351#discussion_r395291803
##
File path:
lucene/core/src/java/org/apache/lucene/search/LongDocValuesPointComparator.java
##
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.document.LongPoint;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.NumericDocValues;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.util.DocIdSetBuilder;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import static
org.apache.lucene.search.FieldComparator.IteratorSupplierComparator;
+
+public class LongDocValuesPointComparator extends
IteratorSupplierComparator {
+private final String field;
+private final int numHits;
+private final boolean reverse;
+private final long missingValue;
+private final long[] values;
+private long bottom;
+private long topValue;
+boolean hasTopValue = false; // indicates that topValue for searchAfter is
set
+protected NumericDocValues docValues;
+private DocIdSetIterator iterator;
+private PointValues pointValues;
+private HitsThresholdChecker hitsThresholdChecker = null;
+private int maxDoc;
+private int maxDocVisited;
+private int updateCounter = 0;
+
+public LongDocValuesPointComparator(String field, int numHits, boolean
reverse, Long missingValue) {
+this.field = field;
+this.numHits = numHits;
+this.reverse = reverse;
+this.missingValue = missingValue != null ? missingValue : 0L;
+this.values = new long[numHits];
+}
+
+@Override
+void setHitsThresholdChecker(HitsThresholdChecker hitsThresholdChecker) {
+this.hitsThresholdChecker = hitsThresholdChecker;
+}
+
+private long getValueForDoc(int doc) throws IOException {
+if (docValues.advanceExact(doc)) {
+return docValues.longValue();
+} else {
+return missingValue;
+}
+}
+
+@Override
+public int compare(int slot1, int slot2) {
+return Long.compare(values[slot1], values[slot2]);
+}
+
+@Override
+public void setTopValue(Long value) {
+topValue = value;
+hasTopValue = true;
+}
+
+@Override
+public Long value(int slot) {
+return Long.valueOf(values[slot]);
+}
+
+@Override
+public LeafFieldComparator getLeafComparator(LeafReaderContext context)
throws IOException {
+docValues = DocValues.getNumeric(context.reader(), field);
+iterator = docValues;
+pointValues = context.reader().getPointValues(field);
+maxDoc = context.reader().maxDoc();
+maxDocVisited = 0;
+return this;
+}
+
+@Override
+public void setBottom(int slot) throws IOException {
+this.bottom = values[slot];
+// can't use hitsThresholdChecker.isThresholdReached() as it uses >
numHits,
+// while we want to update iterator as soon as threshold reaches
numHits
+if (hitsThresholdChecker != null &&
(hitsThresholdChecker.getHitsThreshold() >= numHits)) {
Review comment:
@jimczi I am not very happy about this change because of 2 reasons:
1) We can't use `hitsThresholdChecker.isThresholdReached` as it checks for
greater than numHits, but we need to check starting with equal, as if there are
no competitive docs later `setBottom` will not be called.
Do you know the reason why `hitsThresholdChecker.isThresholdReached` checks
for greater than numHits and not greater or equal numHits?
2) totalHitsRelation may not end up to be set to
TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO, as we set it only when we have
later competitive hits.
I think it is better to have a previous implementation with a dedicated
`updateIterator` function called from `TopFieldCollector`. WDYT?
This is an automated message from the Apache Git Ser
[GitHub] [lucene-solr] s1monw commented on issue #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit
s1monw commented on issue #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit URL: https://github.com/apache/lucene-solr/pull/1361#issuecomment-601407553 > 2) fixing exception handling in DW.updateDocuments to handle this exception gracefully (non-aborting). the problem in this method was that we need to flush the DWPT to disk even if we hit an non-aborting exception. Otherwise the next doc would hit the same exception or we would violate the assumption that we never receive a flush pending DWPT. One thing that I keep thinking about is what happens if we index documents close to the limit and then the next batch of documents would exceed that limit. That would mean that we reject docs event though we could index them into a different DWPT. Should we at least try to index it again in a fresh DWPT unless the DWPT that rejected it was empty? I mean that would prevent the issue and should be straight forward to implement. I can work on this in a followup. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14336) Warn users running old version of Solr
[ https://issues.apache.org/jira/browse/SOLR-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062967#comment-17062967 ] Jan Høydahl commented on SOLR-14336: Not for old versions - that train has left the station. But people installing v8.6.0 will be warned one year later in 2021 if they are still on 8.6.0. > Warn users running old version of Solr > -- > > Key: SOLR-14336 > URL: https://issues.apache.org/jira/browse/SOLR-14336 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Jan Høydahl >Priority: Major > > There are obviously many very old Solr installs out there. People are still > reporting issues for Solr 4.x and 5.x. This is a proposal that Solr will > print a warning in the logs and display a warning in the Admin UI Dashboard > when running a (probably) outdated version. > I do not aim to "call home" to Apache to check versions, but instead parse > release date from 'solr-impl-version' string and warn if more than 1 year > old. That should be very conservative as there will almost certainly be a > handful new releases, with potential security fixes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14336) Warn users running old version of Solr
[
https://issues.apache.org/jira/browse/SOLR-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062988#comment-17062988
]
Ishan Chattopadhyaya commented on SOLR-14336:
-
{quote}bq.But people installing v8.6.0 will be warned one year later in 2021 if
they are still on 8.6.0.
{quote}
But what if all committers and PMC members suddenly die and there is no 8.7
version in one year's time?
> Warn users running old version of Solr
> --
>
> Key: SOLR-14336
> URL: https://issues.apache.org/jira/browse/SOLR-14336
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Priority: Major
>
> There are obviously many very old Solr installs out there. People are still
> reporting issues for Solr 4.x and 5.x. This is a proposal that Solr will
> print a warning in the logs and display a warning in the Admin UI Dashboard
> when running a (probably) outdated version.
> I do not aim to "call home" to Apache to check versions, but instead parse
> release date from 'solr-impl-version' string and warn if more than 1 year
> old. That should be very conservative as there will almost certainly be a
> handful new releases, with potential security fixes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9221) Lucene Logo Contest
[
https://issues.apache.org/jira/browse/LUCENE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062993#comment-17062993
]
Dustin Haver commented on LUCENE-9221:
--
This is a submission from Elastic for the logo redesign of Lucene.
A brief description about this logo submission: This concept simplifies and
modernizes the current Lucene logo. The 'L' becomes a feather/wing that
represents speed and quickness while keeping with the feather idea from Apache.
This logo consists of a primary logo and a abbreviated logo mark "L" that can
be used as a favicon, stickers, etc...
!LuceneLogo.png!
> Lucene Logo Contest
> ---
>
> Key: LUCENE-9221
> URL: https://issues.apache.org/jira/browse/LUCENE-9221
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ryan Ernst
>Priority: Trivial
> Attachments: LuceneLogo.png
>
>
> The Lucene logo has served the project well for almost 20 years. However, it
> does sometimes show its age and misses modern nice-to-haves like invertable
> or grayscale variants.
>
> The PMC would like to have a contest to replace the current logo. This issue
> will serve as the submission mechanism for that contest. When the submission
> deadline closes, a community poll will be used to guide the PMC in the
> decision of which logo to choose. Keeping the current logo will be a possible
> outcome of this decision, if a majority likes the current logo more than any
> other proposal.
>
> The logo should adhere to the guidelines set forth by Apache for project
> logos ([https://www.apache.org/foundation/marks/pmcs#graphics]), specifically
> that the full project name, "Apache Lucene", must appear in the logo
> (although the word "Apache" may be in a smaller font than "Lucene").
>
> The contest will last approximately one month. The submission deadline is
> *Monday, March 16, 2020*. Submissions should be attached in a single zip or
> tar archive, with the filename of the form \{{[user]-[proposal
> number].[extension]}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9221) Lucene Logo Contest
[
https://issues.apache.org/jira/browse/LUCENE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dustin Haver updated LUCENE-9221:
-
Attachment: LuceneLogo.png
> Lucene Logo Contest
> ---
>
> Key: LUCENE-9221
> URL: https://issues.apache.org/jira/browse/LUCENE-9221
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ryan Ernst
>Priority: Trivial
> Attachments: LuceneLogo.png
>
>
> The Lucene logo has served the project well for almost 20 years. However, it
> does sometimes show its age and misses modern nice-to-haves like invertable
> or grayscale variants.
>
> The PMC would like to have a contest to replace the current logo. This issue
> will serve as the submission mechanism for that contest. When the submission
> deadline closes, a community poll will be used to guide the PMC in the
> decision of which logo to choose. Keeping the current logo will be a possible
> outcome of this decision, if a majority likes the current logo more than any
> other proposal.
>
> The logo should adhere to the guidelines set forth by Apache for project
> logos ([https://www.apache.org/foundation/marks/pmcs#graphics]), specifically
> that the full project name, "Apache Lucene", must appear in the logo
> (although the word "Apache" may be in a smaller font than "Lucene").
>
> The contest will last approximately one month. The submission deadline is
> *Monday, March 16, 2020*. Submissions should be attached in a single zip or
> tar archive, with the filename of the form \{{[user]-[proposal
> number].[extension]}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9221) Lucene Logo Contest
[
https://issues.apache.org/jira/browse/LUCENE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062998#comment-17062998
]
Ryan Ernst commented on LUCENE-9221:
Thanks for the submission [~DHAVER0313]!
The deadline I had put in this issue came and went; I guess everyone had other
things on their mind in our current virus ridden world...
I propose we give more time given the circumstances so we can hopefully get
some more submissions. Let's target submissions are in by *Monday, April 6,
2020*.
> Lucene Logo Contest
> ---
>
> Key: LUCENE-9221
> URL: https://issues.apache.org/jira/browse/LUCENE-9221
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ryan Ernst
>Priority: Trivial
> Attachments: LuceneLogo.png
>
>
> The Lucene logo has served the project well for almost 20 years. However, it
> does sometimes show its age and misses modern nice-to-haves like invertable
> or grayscale variants.
>
> The PMC would like to have a contest to replace the current logo. This issue
> will serve as the submission mechanism for that contest. When the submission
> deadline closes, a community poll will be used to guide the PMC in the
> decision of which logo to choose. Keeping the current logo will be a possible
> outcome of this decision, if a majority likes the current logo more than any
> other proposal.
>
> The logo should adhere to the guidelines set forth by Apache for project
> logos ([https://www.apache.org/foundation/marks/pmcs#graphics]), specifically
> that the full project name, "Apache Lucene", must appear in the logo
> (although the word "Apache" may be in a smaller font than "Lucene").
>
> The contest will last approximately one month. The submission deadline is
> *Monday, March 16, 2020*. Submissions should be attached in a single zip or
> tar archive, with the filename of the form \{{[user]-[proposal
> number].[extension]}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9221) Lucene Logo Contest
[
https://issues.apache.org/jira/browse/LUCENE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Ernst updated LUCENE-9221:
---
Description:
The Lucene logo has served the project well for almost 20 years. However, it
does sometimes show its age and misses modern nice-to-haves like invertable or
grayscale variants.
The PMC would like to have a contest to replace the current logo. This issue
will serve as the submission mechanism for that contest. When the submission
deadline closes, a community poll will be used to guide the PMC in the decision
of which logo to choose. Keeping the current logo will be a possible outcome of
this decision, if a majority likes the current logo more than any other
proposal.
The logo should adhere to the guidelines set forth by Apache for project logos
([https://www.apache.org/foundation/marks/pmcs#graphics]), specifically that
the full project name, "Apache Lucene", must appear in the logo (although the
word "Apache" may be in a smaller font than "Lucene").
The contest will last approximately one month. The submission deadline is
-*Monday, March 16, 2020*- *Monday, April 6, 2020*. Submissions should be
attached in a single zip or tar archive, with the filename of the form
{{[user]-[proposal number].[extension]}}.
was:
The Lucene logo has served the project well for almost 20 years. However, it
does sometimes show its age and misses modern nice-to-haves like invertable or
grayscale variants.
The PMC would like to have a contest to replace the current logo. This issue
will serve as the submission mechanism for that contest. When the submission
deadline closes, a community poll will be used to guide the PMC in the decision
of which logo to choose. Keeping the current logo will be a possible outcome of
this decision, if a majority likes the current logo more than any other
proposal.
The logo should adhere to the guidelines set forth by Apache for project logos
([https://www.apache.org/foundation/marks/pmcs#graphics]), specifically that
the full project name, "Apache Lucene", must appear in the logo (although the
word "Apache" may be in a smaller font than "Lucene").
The contest will last approximately one month. The submission deadline is
*Monday, March 16, 2020*. Submissions should be attached in a single zip or tar
archive, with the filename of the form \{{[user]-[proposal
number].[extension]}}.
> Lucene Logo Contest
> ---
>
> Key: LUCENE-9221
> URL: https://issues.apache.org/jira/browse/LUCENE-9221
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ryan Ernst
>Priority: Trivial
> Attachments: LuceneLogo.png
>
>
> The Lucene logo has served the project well for almost 20 years. However, it
> does sometimes show its age and misses modern nice-to-haves like invertable
> or grayscale variants.
>
> The PMC would like to have a contest to replace the current logo. This issue
> will serve as the submission mechanism for that contest. When the submission
> deadline closes, a community poll will be used to guide the PMC in the
> decision of which logo to choose. Keeping the current logo will be a possible
> outcome of this decision, if a majority likes the current logo more than any
> other proposal.
>
> The logo should adhere to the guidelines set forth by Apache for project
> logos ([https://www.apache.org/foundation/marks/pmcs#graphics]), specifically
> that the full project name, "Apache Lucene", must appear in the logo
> (although the word "Apache" may be in a smaller font than "Lucene").
>
> The contest will last approximately one month. The submission deadline is
> -*Monday, March 16, 2020*- *Monday, April 6, 2020*. Submissions should be
> attached in a single zip or tar archive, with the filename of the form
> {{[user]-[proposal number].[extension]}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14344) Remove Deprecated RemoteSolrException and RemoteExcecutionException
[
https://issues.apache.org/jira/browse/SOLR-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063005#comment-17063005
]
Lucene/Solr QA commented on SOLR-14344:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 3m 16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 3m 12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 3m 12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m
30s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
14s{color} | {color:green} solrj in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-14344 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12997151/SOLR-14344.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene2-us-west.apache.org 4.4.0-170-generic #199-Ubuntu SMP
Thu Nov 14 01:45:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 5fd55d7 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/722/testReport/ |
| modules | C: solr/core solr/solrj U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/722/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Remove Deprecated RemoteSolrException and RemoteExcecutionException
> ---
>
> Key: SOLR-14344
> URL: https://issues.apache.org/jira/browse/SOLR-14344
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Munendra S N
>Assignee: Munendra S N
>Priority: Major
> Attachments: SOLR-14344.patch, SOLR-14344.patch
>
>
> {{HttpSolrClient.RemoteSolrException}}
> {{HttpSolrClient.RemotExecutionException}} were deprecated in 8.0, we should
> consider removing them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395418877
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
Review comment:
The zkclient may get closed or recreated. Always safer to get the current one
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395419029
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
+
+ public ZkRead(CoreContainer coreContainer) {
+this.coreContainer = coreContainer;
+ }
+
+ @Command
+ public void get(SolrQueryRequest req, SolrQueryResponse rsp) {
+String path = req.getPathTemplateValues().get("*");
+if (path == null || path.isEmpty()) path = "/";
+byte[] d = null;
+try {
+ List l =
coreContainer.getZkController().getZkClient().getChildren(path, null, false);
+ if (l != null && !l.isEmpty()) {
+String prefix = path.endsWith("/") ? path : path + "/";
+
+rsp.add(path, (MapWriter) ew -> {
+ for (String s : l) {
+try {
+ Stat stat =
coreContainer.getZkController().getZkClient().exists(prefix + s, null, false);
+ ew.put(s, (MapWriter) ew1 -> {
+ew1.put("version", stat.getVersion());
+ew1.put("aversion", stat.getAversion());
+ew1.put("children", stat.getNumChildren());
+ew1.put("ctime", stat.getCtime());
+ew1.put("cversion", stat.getCversion());
+ew1.put("czxid", stat.getCzxid());
+ew1.put("ephemeralOwner", stat.getEphemeralOwner());
+ew1.put("mtime", stat.getMtime());
+ew1.put("mzxid", stat.getMzxid());
+ew1.put("pzxid", stat.getPzxid());
+ew1.put("dataLength", stat.getDataLength());
+ });
+} catch (Exception e) {
+ ew.put("s", Collections.singletonMap("error", e.getMessage()));
Review comment:
good catch. it was supposed to be `s`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395419555
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
+
+ public ZkRead(CoreContainer coreContainer) {
+this.coreContainer = coreContainer;
+ }
+
+ @Command
+ public void get(SolrQueryRequest req, SolrQueryResponse rsp) {
+String path = req.getPathTemplateValues().get("*");
+if (path == null || path.isEmpty()) path = "/";
+byte[] d = null;
+try {
+ List l =
coreContainer.getZkController().getZkClient().getChildren(path, null, false);
+ if (l != null && !l.isEmpty()) {
+String prefix = path.endsWith("/") ? path : path + "/";
+
+rsp.add(path, (MapWriter) ew -> {
+ for (String s : l) {
+try {
+ Stat stat =
coreContainer.getZkController().getZkClient().exists(prefix + s, null, false);
+ ew.put(s, (MapWriter) ew1 -> {
+ew1.put("version", stat.getVersion());
+ew1.put("aversion", stat.getAversion());
+ew1.put("children", stat.getNumChildren());
+ew1.put("ctime", stat.getCtime());
+ew1.put("cversion", stat.getCversion());
+ew1.put("czxid", stat.getCzxid());
+ew1.put("ephemeralOwner", stat.getEphemeralOwner());
+ew1.put("mtime", stat.getMtime());
+ew1.put("mzxid", stat.getMzxid());
+ew1.put("pzxid", stat.getPzxid());
+ew1.put("dataLength", stat.getDataLength());
+ });
+} catch (Exception e) {
Review comment:
does it even matter?
Why interrupt the thread when this thread is not doing anything at all. This
is the `qtp` thread. Interrupting it makes no sense
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395419935
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
+
+ public ZkRead(CoreContainer coreContainer) {
+this.coreContainer = coreContainer;
+ }
+
+ @Command
+ public void get(SolrQueryRequest req, SolrQueryResponse rsp) {
+String path = req.getPathTemplateValues().get("*");
+if (path == null || path.isEmpty()) path = "/";
+byte[] d = null;
+try {
+ List l =
coreContainer.getZkController().getZkClient().getChildren(path, null, false);
+ if (l != null && !l.isEmpty()) {
+String prefix = path.endsWith("/") ? path : path + "/";
+
+rsp.add(path, (MapWriter) ew -> {
+ for (String s : l) {
+try {
+ Stat stat =
coreContainer.getZkController().getZkClient().exists(prefix + s, null, false);
+ ew.put(s, (MapWriter) ew1 -> {
+ew1.put("version", stat.getVersion());
+ew1.put("aversion", stat.getAversion());
+ew1.put("children", stat.getNumChildren());
+ew1.put("ctime", stat.getCtime());
+ew1.put("cversion", stat.getCversion());
+ew1.put("czxid", stat.getCzxid());
+ew1.put("ephemeralOwner", stat.getEphemeralOwner());
+ew1.put("mtime", stat.getMtime());
+ew1.put("mzxid", stat.getMzxid());
+ew1.put("pzxid", stat.getPzxid());
+ew1.put("dataLength", stat.getDataLength());
+ });
+} catch (Exception e) {
+ ew.put("s", Collections.singletonMap("error", e.getMessage()));
+}
+ }
+});
+
+ } else {
+d = coreContainer.getZkController().getZkClient().getData(path, null,
null, false);
+if (d == null || d.length == 0) {
+ rsp.add(path, null);
+ return;
+}
+
+Map map = new HashMap<>(1);
+map.put(WT, "raw");
+map.put(OMIT_HEADER, "true");
+req.setParams(SolrParams.wrapDefaults(new MapSolrParams(map),
req.getParams()));
Review comment:
This is important if we are ever writing out raw content that comes from ZK.
We should not use any of the `ResponseWriter` in Solr
This is an automated messag
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395419984
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
+
+ public ZkRead(CoreContainer coreContainer) {
+this.coreContainer = coreContainer;
+ }
+
+ @Command
+ public void get(SolrQueryRequest req, SolrQueryResponse rsp) {
+String path = req.getPathTemplateValues().get("*");
+if (path == null || path.isEmpty()) path = "/";
+byte[] d = null;
+try {
+ List l =
coreContainer.getZkController().getZkClient().getChildren(path, null, false);
+ if (l != null && !l.isEmpty()) {
+String prefix = path.endsWith("/") ? path : path + "/";
+
+rsp.add(path, (MapWriter) ew -> {
+ for (String s : l) {
+try {
+ Stat stat =
coreContainer.getZkController().getZkClient().exists(prefix + s, null, false);
+ ew.put(s, (MapWriter) ew1 -> {
+ew1.put("version", stat.getVersion());
+ew1.put("aversion", stat.getAversion());
+ew1.put("children", stat.getNumChildren());
+ew1.put("ctime", stat.getCtime());
+ew1.put("cversion", stat.getCversion());
+ew1.put("czxid", stat.getCzxid());
+ew1.put("ephemeralOwner", stat.getEphemeralOwner());
+ew1.put("mtime", stat.getMtime());
+ew1.put("mzxid", stat.getMzxid());
+ew1.put("pzxid", stat.getPzxid());
+ew1.put("dataLength", stat.getDataLength());
+ });
+} catch (Exception e) {
+ ew.put("s", Collections.singletonMap("error", e.getMessage()));
+}
+ }
+});
+
+ } else {
+d = coreContainer.getZkController().getZkClient().getData(path, null,
null, false);
+if (d == null || d.length == 0) {
+ rsp.add(path, null);
+ return;
+}
+
+Map map = new HashMap<>(1);
+map.put(WT, "raw");
+map.put(OMIT_HEADER, "true");
+req.setParams(SolrParams.wrapDefaults(new MapSolrParams(map),
req.getParams()));
+
+
+rsp.add(CONTENT, new ContentStreamBase.ByteArrayStream(d, null,
+d[0] == '{' ? CommonParams.JSON_MIME :
BinaryResponseParser.BINARY_CONTENT_TYPE));
+
+ }
+
+} catch (Exception e) {
Review comment:
it does n
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395420255
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
+
+ public ZkRead(CoreContainer coreContainer) {
+this.coreContainer = coreContainer;
+ }
+
+ @Command
+ public void get(SolrQueryRequest req, SolrQueryResponse rsp) {
+String path = req.getPathTemplateValues().get("*");
+if (path == null || path.isEmpty()) path = "/";
+byte[] d = null;
+try {
+ List l =
coreContainer.getZkController().getZkClient().getChildren(path, null, false);
+ if (l != null && !l.isEmpty()) {
+String prefix = path.endsWith("/") ? path : path + "/";
+
+rsp.add(path, (MapWriter) ew -> {
+ for (String s : l) {
+try {
+ Stat stat =
coreContainer.getZkController().getZkClient().exists(prefix + s, null, false);
+ ew.put(s, (MapWriter) ew1 -> {
+ew1.put("version", stat.getVersion());
+ew1.put("aversion", stat.getAversion());
+ew1.put("children", stat.getNumChildren());
+ew1.put("ctime", stat.getCtime());
+ew1.put("cversion", stat.getCversion());
+ew1.put("czxid", stat.getCzxid());
+ew1.put("ephemeralOwner", stat.getEphemeralOwner());
+ew1.put("mtime", stat.getMtime());
+ew1.put("mzxid", stat.getMzxid());
+ew1.put("pzxid", stat.getPzxid());
+ew1.put("dataLength", stat.getDataLength());
+ });
+} catch (Exception e) {
+ ew.put("s", Collections.singletonMap("error", e.getMessage()));
+}
+ }
+});
+
+ } else {
Review comment:
fixing it. you can explicitly request for the node data using a param
`leaf=true`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...
[GitHub] [lucene-solr] madrob commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
madrob commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395420007
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
+
+// compute nodes, some live, some down
+final int maxNodesOfAType = perShardCounts.stream() // not too important
how many we have, but lets have plenty
+.mapToInt(c -> c.totalReplicasInLiveNodes + c.totalReplicasInDownNodes
+ c.myReplicas).max().getAsInt();
+List liveNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.0." + i + "_8983").collect(Collectors.toList());
+Collections.shuffle(liveNodes, random());
+String thisNode = liveNodes.get(0);
+List otherLiveNodes = liveNodes.subList(1, liveNodes.size());
+List downNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.1." + i + "_8983").collect(Collectors.toList());
+
+// divide into two collections
+int numCol1 = random().nextInt(perShardCounts.size());
+Map> collToCounts = new HashMap<>();
+collToCounts.put("col1", perShardCounts.subList(0, numCol1));
+collToCounts.put("col2", perShardCounts.subList(numCol1,
perShardCounts.size()));
+
+Map collToState = new HashMap<>();
+Map> myCountsToDescs = new
HashMap<>();
+for (Map.Entry> entry :
collToCounts.entrySet()) {
+ String collection = entry.getKey();
+ List collCounts = entry.getValue();
+ Map sliceMap = new HashMap<>(collCounts.size());
+ for (CountsForEachShard shardCounts : collCounts) {
+String slice = "s" + shardCounts.hashCode();
+List replicas = new ArrayList<>();
+for (int myRepNum = 0; myRepNum < shardCounts.myReplicas; myRepNum++) {
+ addNewReplica(replicas, collection, slice,
Collections.singletonList(thisNode));
+ // save this mapping for later
+ myCountsToDescs.put(shardCounts,
replicas.stream().map(this::newCoreDescriptor).collect(Collectors.toList()));
[GitHub] [lucene-solr] madrob commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
madrob commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395419608
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
+
+// compute nodes, some live, some down
+final int maxNodesOfAType = perShardCounts.stream() // not too important
how many we have, but lets have plenty
+.mapToInt(c -> c.totalReplicasInLiveNodes + c.totalReplicasInDownNodes
+ c.myReplicas).max().getAsInt();
+List liveNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.0." + i + "_8983").collect(Collectors.toList());
+Collections.shuffle(liveNodes, random());
+String thisNode = liveNodes.get(0);
+List otherLiveNodes = liveNodes.subList(1, liveNodes.size());
+List downNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.1." + i + "_8983").collect(Collectors.toList());
+
+// divide into two collections
+int numCol1 = random().nextInt(perShardCounts.size());
+Map> collToCounts = new HashMap<>();
+collToCounts.put("col1", perShardCounts.subList(0, numCol1));
+collToCounts.put("col2", perShardCounts.subList(numCol1,
perShardCounts.size()));
+
+Map collToState = new HashMap<>();
+Map> myCountsToDescs = new
HashMap<>();
+for (Map.Entry> entry :
collToCounts.entrySet()) {
+ String collection = entry.getKey();
+ List collCounts = entry.getValue();
+ Map sliceMap = new HashMap<>(collCounts.size());
+ for (CountsForEachShard shardCounts : collCounts) {
+String slice = "s" + shardCounts.hashCode();
+List replicas = new ArrayList<>();
+for (int myRepNum = 0; myRepNum < shardCounts.myReplicas; myRepNum++) {
+ addNewReplica(replicas, collection, slice,
Collections.singletonList(thisNode));
+ // save this mapping for later
+ myCountsToDescs.put(shardCounts,
replicas.stream().map(this::newCoreDescriptor).collect(Collectors.toList()));
[GitHub] [lucene-solr] madrob commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
madrob commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395419361
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
+
+// compute nodes, some live, some down
+final int maxNodesOfAType = perShardCounts.stream() // not too important
how many we have, but lets have plenty
+.mapToInt(c -> c.totalReplicasInLiveNodes + c.totalReplicasInDownNodes
+ c.myReplicas).max().getAsInt();
+List liveNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.0." + i + "_8983").collect(Collectors.toList());
+Collections.shuffle(liveNodes, random());
+String thisNode = liveNodes.get(0);
+List otherLiveNodes = liveNodes.subList(1, liveNodes.size());
+List downNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.1." + i + "_8983").collect(Collectors.toList());
+
+// divide into two collections
+int numCol1 = random().nextInt(perShardCounts.size());
Review comment:
this could be zero, is that ok?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
madrob commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395417505
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
Review comment:
pass `random()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
madrob commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395417214
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
Review comment:
What's the value of combining what used to be two test methods into one?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
noblepaul commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r395421010
##
File path:
solr/core/src/test/org/apache/solr/handler/admin/ZookeeperStatusHandlerTest.java
##
@@ -74,6 +78,39 @@ public void tearDown() throws Exception {
super.tearDown();
}
+ @Test
+ public void testZkread() throws Exception {
+URL baseUrl = cluster.getJettySolrRunner(0).getBaseUrl();
+String basezk = baseUrl.toString().replace("/solr", "/api") +
"/cluster/zk";
+
+try( HttpSolrClient client = new
HttpSolrClient.Builder(baseUrl.toString()).build()) {
+ Object o = Utils.executeGET(client.getHttpClient(),
+ basezk + "/security.json",
+ Utils.JSONCONSUMER );
+ assertNotNull(o);
+ o = Utils.executeGET(client.getHttpClient(),
+ basezk + "/configs",
+ Utils.JSONCONSUMER );
+ assertEquals("0", String.valueOf(getObjectByPath(o,true,
split(":/configs:_default:dataLength",':';
+ assertEquals("0", String.valueOf(getObjectByPath(o,true,
split(":/configs:conf:dataLength",':';
+ byte[] bytes = new byte[1024*5];
+ for (int i = 0; i < bytes.length; i++) {
+bytes[i] = (byte) random().nextInt(128);
Review comment:
this was the easiest way to create a `byte[]` . could use anything else.
But this code was simpler
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395421862
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
+
+// compute nodes, some live, some down
+final int maxNodesOfAType = perShardCounts.stream() // not too important
how many we have, but lets have plenty
+.mapToInt(c -> c.totalReplicasInLiveNodes + c.totalReplicasInDownNodes
+ c.myReplicas).max().getAsInt();
+List liveNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.0." + i + "_8983").collect(Collectors.toList());
+Collections.shuffle(liveNodes, random());
+String thisNode = liveNodes.get(0);
+List otherLiveNodes = liveNodes.subList(1, liveNodes.size());
+List downNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.1." + i + "_8983").collect(Collectors.toList());
+
+// divide into two collections
+int numCol1 = random().nextInt(perShardCounts.size());
Review comment:
I know; yup. Then we get one collection and that's fine. The actual
collection distribution shouldn't matter but I wanted to add some
multi-collection in there to mix things up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395421972
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
+
+// compute nodes, some live, some down
+final int maxNodesOfAType = perShardCounts.stream() // not too important
how many we have, but lets have plenty
+.mapToInt(c -> c.totalReplicasInLiveNodes + c.totalReplicasInDownNodes
+ c.myReplicas).max().getAsInt();
+List liveNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.0." + i + "_8983").collect(Collectors.toList());
+Collections.shuffle(liveNodes, random());
+String thisNode = liveNodes.get(0);
+List otherLiveNodes = liveNodes.subList(1, liveNodes.size());
+List downNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.1." + i + "_8983").collect(Collectors.toList());
+
+// divide into two collections
+int numCol1 = random().nextInt(perShardCounts.size());
+Map> collToCounts = new HashMap<>();
+collToCounts.put("col1", perShardCounts.subList(0, numCol1));
+collToCounts.put("col2", perShardCounts.subList(numCol1,
perShardCounts.size()));
+
+Map collToState = new HashMap<>();
+Map> myCountsToDescs = new
HashMap<>();
+for (Map.Entry> entry :
collToCounts.entrySet()) {
+ String collection = entry.getKey();
+ List collCounts = entry.getValue();
+ Map sliceMap = new HashMap<>(collCounts.size());
+ for (CountsForEachShard shardCounts : collCounts) {
+String slice = "s" + shardCounts.hashCode();
+List replicas = new ArrayList<>();
+for (int myRepNum = 0; myRepNum < shardCounts.myReplicas; myRepNum++) {
+ addNewReplica(replicas, collection, slice,
Collections.singletonList(thisNode));
+ // save this mapping for later
+ myCountsToDescs.put(shardCounts,
replicas.stream().map(this::newCoreDescriptor).collect(Collectors.toList()));
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395422074
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
+
+// compute nodes, some live, some down
+final int maxNodesOfAType = perShardCounts.stream() // not too important
how many we have, but lets have plenty
+.mapToInt(c -> c.totalReplicasInLiveNodes + c.totalReplicasInDownNodes
+ c.myReplicas).max().getAsInt();
+List liveNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.0." + i + "_8983").collect(Collectors.toList());
+Collections.shuffle(liveNodes, random());
+String thisNode = liveNodes.get(0);
+List otherLiveNodes = liveNodes.subList(1, liveNodes.size());
+List downNodes = IntStream.range(0, maxNodesOfAType).mapToObj(i ->
"192.168.1." + i + "_8983").collect(Collectors.toList());
+
+// divide into two collections
+int numCol1 = random().nextInt(perShardCounts.size());
+Map> collToCounts = new HashMap<>();
+collToCounts.put("col1", perShardCounts.subList(0, numCol1));
+collToCounts.put("col2", perShardCounts.subList(numCol1,
perShardCounts.size()));
+
+Map collToState = new HashMap<>();
+Map> myCountsToDescs = new
HashMap<>();
+for (Map.Entry> entry :
collToCounts.entrySet()) {
+ String collection = entry.getKey();
+ List collCounts = entry.getValue();
+ Map sliceMap = new HashMap<>(collCounts.size());
+ for (CountsForEachShard shardCounts : collCounts) {
+String slice = "s" + shardCounts.hashCode();
+List replicas = new ArrayList<>();
+for (int myRepNum = 0; myRepNum < shardCounts.myReplicas; myRepNum++) {
+ addNewReplica(replicas, collection, slice,
Collections.singletonList(thisNode));
+ // save this mapping for later
+ myCountsToDescs.put(shardCounts,
replicas.stream().map(this::newCoreDescriptor).collect(Collectors.toList()));
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395422456
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
Review comment:
At least there are separate methods and so a test failure would indicate
which from the call stack.
I wanted both the existing "comparator" test and the new integration test to
use the same input. I suppose I could move the input into fields of the class
and then have two test methods that operate on them. LMK and I'll change it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley commented on a change in pull request #1366: SOLR-14342: Optimize core
loading order in SolrCloud.
URL: https://github.com/apache/lucene-solr/pull/1366#discussion_r395422857
##
File path: solr/core/src/test/org/apache/solr/core/CoreSorterTest.java
##
@@ -71,168 +69,141 @@ public void testComparator() {
);
+testComparator(expected, l);
+
+integrationTest(expected, l);
+ }
+
+ private void testComparator(List expectedCounts,
List inputCounts) {
for (int i = 0; i < 10; i++) {
- List copy = new ArrayList<>(l);
+ List copy = new ArrayList<>(inputCounts);
Collections.shuffle(copy, random());
Collections.sort(copy, CoreSorter.countsComparator);
for (int j = 0; j < copy.size(); j++) {
-assertEquals(expected.get(j), copy.get(j));
+assertEquals(expectedCounts.get(j), copy.get(j));
}
}
}
- public void testSort() throws Exception {
-CoreContainer mockCC = getMockContainer();
-MockCoreSorter coreSorter = (MockCoreSorter) new
MockCoreSorter().init(mockCC);
-List copy = new ArrayList<>(coreSorter.getLocalCores());
-Collections.sort(copy, coreSorter::compare);
-List l = copy.stream()
-.map(CoreDescriptor::getCloudDescriptor)
-.map(it -> coreSorter.shardsVsReplicaCounts.get(getShardName(it)))
-.collect(toList());
-for (int i = 1; i < l.size(); i++) {
- CountsForEachShard curr = l.get(i);
- CountsForEachShard prev = l.get(i-1);
- assertTrue(CoreSorter.countsComparator.compare(prev, curr) < 1);
-}
-
-for (CountsForEachShard c : l) {
- System.out.println(c);
-}
- }
-
- private CoreContainer getMockContainer() {
+ private void integrationTest(List expectedCounts,
List _inputCounts) {
assumeWorkingMockito();
-
-CoreContainer mockCC = mock(CoreContainer.class);
-ZkController mockZKC = mock(ZkController.class);
-ClusterState mockClusterState = mock(ClusterState.class);
-when(mockCC.isZooKeeperAware()).thenReturn(true);
-when(mockCC.getZkController()).thenReturn(mockZKC);
-when(mockClusterState.getLiveNodes()).thenReturn(liveNodes);
-when(mockZKC.getClusterState()).thenReturn(mockClusterState);
-return mockCC;
- }
- static class ReplicaInfo {
-final int coll, slice, replica;
-final String replicaName;
-CloudDescriptor cd;
-
-ReplicaInfo(int coll, int slice, int replica) {
- this.coll = coll;
- this.slice = slice;
- this.replica = replica;
- replicaName = "coll_" + coll + "_" + slice + "_" + replica;
- Properties p = new Properties();
- p.setProperty(CoreDescriptor.CORE_SHARD, "shard_" + slice);
- p.setProperty(CoreDescriptor.CORE_COLLECTION, "coll_" + slice);
- p.setProperty(CoreDescriptor.CORE_NODE_NAME, replicaName);
- cd = new CloudDescriptor(null, replicaName, p);
-}
-
-@Override
-public boolean equals(Object obj) {
- if (obj instanceof ReplicaInfo) {
-ReplicaInfo replicaInfo = (ReplicaInfo) obj;
-return replicaInfo.replicaName.equals(replicaName);
+List perShardCounts = new ArrayList<>(_inputCounts);
+Collections.shuffle(perShardCounts);
Review comment:
Doh! And this is what forbiddenAPIs caught.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on issue #1366: SOLR-14342: Optimize core loading order in SolrCloud.
dsmiley commented on issue #1366: SOLR-14342: Optimize core loading order in SolrCloud. URL: https://github.com/apache/lucene-solr/pull/1366#issuecomment-601508721 The previous CoreSorter had a getCloudDescriptors method that called CoreContainer.getCores(). But there are no cores registered yet! It was a pitty because it was only calling this method to get the core descriptors... but of course the core descriptors should be the input to the CoreSorter and so it's silly to try to find them some other way. I think this demonstrated that the API to use CoreSorter was more complicated than it needed to be; CoreSorter should have exactly one method that CoreContainer needs to call. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit
dnhatn commented on a change in pull request #1361: LUCENE-8118: Throw
exception if DWPT grows beyond it's maximum ram limit
URL: https://github.com/apache/lucene-solr/pull/1361#discussion_r395423835
##
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriter.java
##
@@ -3941,4 +3943,108 @@ public void testRandomOperationsWithSoftDeletes()
throws Exception {
}
}
}
+
+ public void testAbortDocWhenHardLimitExceeds() throws IOException {
+Directory dir = newDirectory();
+IndexWriterConfig indexWriterConfig = new IndexWriterConfig();
+IndexWriter w = new IndexWriter(dir, indexWriterConfig);
+indexWriterConfig.setRAMPerThreadHardLimitMB(1);
+w.addDocument(new Document()); // no problem
+w.flush();
+assertEquals(1, w.getFlushCount());
+final long seed = random().nextLong();
+int numDocs = 1;
+try (LineFileDocs docs = new LineFileDocs(new Random(seed))) {
+ while (w.getFlushCount() == 1) {
+w.addDocument(docs.nextDoc());
+numDocs++;
+ }
+}
+try (LineFileDocs docs = new LineFileDocs(new Random(seed))) {
+ if (random().nextBoolean()) {
+w.addDocument(new Document());
+numDocs++;
+ }
+ w.addDocuments(() -> new Iterator<>() {
+@Override
+public boolean hasNext() {
+ return true;
+}
+
+@Override
+public Iterable next() {
+ try {
+return docs.nextDoc();
+ } catch (IOException e) {
+throw new UncheckedIOException(e);
+ }
+}
+ });
+} catch (IllegalArgumentException e) {
+ assertEquals("RAM used by a single DocumentsWriterPerThread can't
exceed: 1MB", e.getMessage());
+ assertNull(w.getTragicException());
+}
+w.addDocument(new Document());
+numDocs++;
+try (LineFileDocs docs = new LineFileDocs(new Random(seed))) {
+ final int currentNumDocs = numDocs;
+ for (int i = 0; i < currentNumDocs; i++) {
+w.addDocument(docs.nextDoc());
+numDocs++;
+ }
+}
+w.addDocument(new Document());
+numDocs++;
+w.forceMergeDeletes(true);
+w.commit();
+
+try (IndexReader reader = DirectoryReader.open(dir)) {
+ assertEquals(numDocs, reader.numDocs());
+}
+
+
+w.close();
+dir.close();
+ }
+
+ @Nightly
+ public void testAddDocumentsMassive() throws Exception {
+Directory dir = newFSDirectory(createTempDir("addDocumentsMassive"));
+IndexWriter w = new IndexWriter(dir, new IndexWriterConfig());
+StringBuilder sb = new StringBuilder();
+// same doc with lots of duplicated terms to put pressure on
FreqProxTermsWriter instead of terms dict.
+for (int i = 0; i < 1024; i++) {
+ sb.append(i);
+ sb.append(' ');
+}
+final AtomicInteger count = new AtomicInteger(0);
+Document doc = new Document();
+try {
+ doc.add(new TextField("text", sb.toString(), Field.Store.NO));
+ w.addDocument(doc);
+ w.addDocuments((Iterable) () -> new Iterator<>() {
+
+@Override
+public boolean hasNext() {
+ return count.get() < 100_000_000;
+}
+
+@Override
+public Document next() {
+ if (!hasNext()) {
+throw new AssertionError();
+ }
+ count.incrementAndGet();
+ return doc;
+}
+ });
+} catch (IllegalArgumentException ex) {
+ assertEquals("RAM used by a single DocumentsWriterPerThread can exceed:
1945MB", ex.getMessage());
Review comment:
can -> can't
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dnhatn commented on a change in pull request #1361: LUCENE-8118: Throw exception if DWPT grows beyond it's maximum ram limit
dnhatn commented on a change in pull request #1361: LUCENE-8118: Throw
exception if DWPT grows beyond it's maximum ram limit
URL: https://github.com/apache/lucene-solr/pull/1361#discussion_r395425026
##
File path: lucene/core/src/test/org/apache/lucene/index/TestIndexWriter.java
##
@@ -3941,4 +3943,108 @@ public void testRandomOperationsWithSoftDeletes()
throws Exception {
}
}
}
+
+ public void testAbortDocWhenHardLimitExceeds() throws IOException {
+Directory dir = newDirectory();
+IndexWriterConfig indexWriterConfig = new IndexWriterConfig();
+IndexWriter w = new IndexWriter(dir, indexWriterConfig);
+indexWriterConfig.setRAMPerThreadHardLimitMB(1);
+w.addDocument(new Document()); // no problem
+w.flush();
+assertEquals(1, w.getFlushCount());
+final long seed = random().nextLong();
+int numDocs = 1;
+try (LineFileDocs docs = new LineFileDocs(new Random(seed))) {
+ while (w.getFlushCount() == 1) {
+w.addDocument(docs.nextDoc());
+numDocs++;
+ }
+}
+try (LineFileDocs docs = new LineFileDocs(new Random(seed))) {
+ if (random().nextBoolean()) {
+w.addDocument(new Document());
+numDocs++;
+ }
+ w.addDocuments(() -> new Iterator<>() {
+@Override
+public boolean hasNext() {
+ return true;
+}
+
+@Override
+public Iterable next() {
+ try {
+return docs.nextDoc();
+ } catch (IOException e) {
+throw new UncheckedIOException(e);
+ }
+}
+ });
+} catch (IllegalArgumentException e) {
+ assertEquals("RAM used by a single DocumentsWriterPerThread can't
exceed: 1MB", e.getMessage());
+ assertNull(w.getTragicException());
+}
+w.addDocument(new Document());
+numDocs++;
+try (LineFileDocs docs = new LineFileDocs(new Random(seed))) {
+ final int currentNumDocs = numDocs;
+ for (int i = 0; i < currentNumDocs; i++) {
+w.addDocument(docs.nextDoc());
+numDocs++;
+ }
+}
+w.addDocument(new Document());
+numDocs++;
+w.forceMergeDeletes(true);
+w.commit();
+
+try (IndexReader reader = DirectoryReader.open(dir)) {
+ assertEquals(numDocs, reader.numDocs());
+}
+
+
+w.close();
+dir.close();
+ }
+
+ @Nightly
+ public void testAddDocumentsMassive() throws Exception {
+Directory dir = newFSDirectory(createTempDir("addDocumentsMassive"));
+IndexWriter w = new IndexWriter(dir, new IndexWriterConfig());
+StringBuilder sb = new StringBuilder();
+// same doc with lots of duplicated terms to put pressure on
FreqProxTermsWriter instead of terms dict.
+for (int i = 0; i < 1024; i++) {
+ sb.append(i);
+ sb.append(' ');
+}
+final AtomicInteger count = new AtomicInteger(0);
+Document doc = new Document();
+try {
+ doc.add(new TextField("text", sb.toString(), Field.Store.NO));
+ w.addDocument(doc);
+ w.addDocuments((Iterable) () -> new Iterator<>() {
+
+@Override
+public boolean hasNext() {
+ return count.get() < 100_000_000;
+}
+
+@Override
+public Document next() {
+ if (!hasNext()) {
+throw new AssertionError();
Review comment:
I think we should throw an error in `hasNext` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-8274) Add per-request MDC logging based on user-provided value.
[ https://issues.apache.org/jira/browse/SOLR-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063082#comment-17063082 ] David Smiley commented on SOLR-8274: OpenTracing which is in Solr 8.2 so perhaps this is enough [~gerlowskija]? On the other hand, OpenTracing is maybe a heavy-handed solution when sometimes we "just" want to propagate a request ID – sort of a ultra light version of what OpenTracing accomplishes. [~caomanhdat] can the existing support for OpenTracing do simple ID passing without a server? > Add per-request MDC logging based on user-provided value. > - > > Key: SOLR-8274 > URL: https://issues.apache.org/jira/browse/SOLR-8274 > Project: Solr > Issue Type: Improvement > Components: logging >Reporter: Jason Gerlowski >Priority: Minor > Attachments: SOLR-8274.patch > > > *Problem 1* Currently, there's no way (AFAIK) to find all log messages > associated with a particular request. > *Problem 2* There's also no easy way for multi-tenant Solr setups to find all > log messages associated with a particular customer/tenant. > Both of these problems would be more manageable if Solr could be configured > to record an MDC tag based on a header, or some other user provided value. > This would allow admins to group together logs about a single request. If > the same header value is repeated multiple times this functionality could > also be used to group together arbitrary requests, such as those that come > from a particular user, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14351) Harden MDCLoggingContext.clear depth tracking
David Smiley created SOLR-14351: --- Summary: Harden MDCLoggingContext.clear depth tracking Key: SOLR-14351 URL: https://issues.apache.org/jira/browse/SOLR-14351 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: David Smiley Assignee: David Smiley MDCLoggingContext tracks recursive calls and only clears when the recursion level is back down to 0. If a caller forgets to register and ends up calling clear any ways, then this can mess things up. Additionally I found at least one place this is occurring, which led me to investigate this matter. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14346) Solr fails to check zombie server in LbHttpSolrClient
[ https://issues.apache.org/jira/browse/SOLR-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoxiajun updated SOLR-14346: - Priority: Critical (was: Major) > Solr fails to check zombie server in LbHttpSolrClient > - > > Key: SOLR-14346 > URL: https://issues.apache.org/jira/browse/SOLR-14346 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Affects Versions: 8.3.1 > Environment: solr: 8.3.1 (not cloud mode) > java: 1.8.0_191 > os: centos 7 >Reporter: maoxiajun >Priority: Critical > > solr version: 8.3.1 > steps to reappear: > step1: configured LbHttpSolrClient whith ['[http://host1:8983/solr',] > '[http://host2:8983/solr'|http://host2:8983/solr',]] > step2: shutdown host2, and keep query going > step3: start host2, wait for 1 minute or more, but we'll find host2 not > receiving any query request -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
