[jira] [Assigned] (SOLR-11245) Cloud native Dockerfile
[ https://issues.apache.org/jira/browse/SOLR-11245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Høydahl reassigned SOLR-11245: -- Assignee: Jan Høydahl > Cloud native Dockerfile > --- > > Key: SOLR-11245 > URL: https://issues.apache.org/jira/browse/SOLR-11245 > Project: Solr > Issue Type: Bug > Components: Build >Affects Versions: 6.6 >Reporter: jay vyas >Assignee: Jan Høydahl >Priority: Major > Labels: docker > Time Spent: 10m > Remaining Estimate: 0h > > SOLR Should have its own Dockerfile, ideally one that is cloud native (i.e. > doesn't expect anything special from the operating system in terms of user > IDs, etc), for deployment, that we can curate and submit changes to as part > of the official ASF process, rather then externally. The idea here is that > testing SOLR regression, as a microservice, is something we should be doing > as part of our continuous integration, rather then something done externally. > We have a team here that would be more then happy to do the work to port > whatever existing SOLR dockerfiles are out there into something that is ASF > maintainable, and cloud native, and easily testable, as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1387: SOLR-14210: Include replica health in healtcheck handler
janhoy commented on a change in pull request #1387: SOLR-14210: Include replica
health in healtcheck handler
URL: https://github.com/apache/lucene-solr/pull/1387#discussion_r401424526
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/HealthCheckHandler.java
##
@@ -88,15 +95,42 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
return;
}
-// Set status to true if this node is in live_nodes
-if
(clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
- rsp.add(STATUS, OK);
-} else {
+// Fail if not in live_nodes
+if
(!clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
rsp.add(STATUS, FAILURE);
rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Host Unavailable:
Not in live nodes as per zk"));
+ return;
}
-rsp.setHttpCaching(false);
+// Optionally require that all cores on this node are active if param
'failWhenRecovering=true'
+if (req.getParams().getBool(PARAM_REQUIRE_HEALTHY_CORES, false)) {
+ List unhealthyCores = findUnhealthyCores(clusterState,
cores.getNodeConfig().getNodeName());
+ if (unhealthyCores.size() > 0) {
+ rsp.add(STATUS, FAILURE);
+ rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE,
+ "Replica(s) " + unhealthyCores + " are currently
initializing or recovering"));
+ return;
+ }
+ rsp.add("MESSAGE", "All cores are healthy");
+}
+
+// All lights green, report healthy
+rsp.add(STATUS, OK);
+ }
+
+ /**
+ * Find replicas DOWN or RECOVERING
+ * @param clusterState clusterstate from ZK
+ * @param nodeName this node name
+ * @return list of core names that are either DOWN ore RECOVERING on
'nodeName'
+ */
+ static List findUnhealthyCores(ClusterState clusterState, String
nodeName) {
+return clusterState.getCollectionsMap().values().stream()
Review comment:
> maybe we want to return false if there are any replicas from inactive
slices on the node
Inactive shards are not searched, so we should not care about those. A shard
split will not clean up the old shard, but instead mark it inactive, until user
manually deletes those shards, or the Autoscaling framework rules go reap them.
That is why I chose to check active shards only. We should be ok if the active
shard(s) only are up and active, then k8s can go restart the next node.
If a shard split is currently running (could be long running), on a node
being restarted, the split would be aborted but when the node comes up again I
believe the overseer might try again??
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1387: SOLR-14210: Include replica health in healtcheck handler
janhoy commented on a change in pull request #1387: SOLR-14210: Include replica
health in healtcheck handler
URL: https://github.com/apache/lucene-solr/pull/1387#discussion_r401425180
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/HealthCheckHandler.java
##
@@ -88,15 +95,42 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
return;
}
-// Set status to true if this node is in live_nodes
-if
(clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
- rsp.add(STATUS, OK);
-} else {
+// Fail if not in live_nodes
+if
(!clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
rsp.add(STATUS, FAILURE);
rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Host Unavailable:
Not in live nodes as per zk"));
+ return;
}
-rsp.setHttpCaching(false);
+// Optionally require that all cores on this node are active if param
'failWhenRecovering=true'
+if (req.getParams().getBool(PARAM_REQUIRE_HEALTHY_CORES, false)) {
+ List unhealthyCores = findUnhealthyCores(clusterState,
cores.getNodeConfig().getNodeName());
+ if (unhealthyCores.size() > 0) {
+ rsp.add(STATUS, FAILURE);
+ rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE,
+ "Replica(s) " + unhealthyCores + " are currently
initializing or recovering"));
+ return;
+ }
+ rsp.add("MESSAGE", "All cores are healthy");
+}
+
+// All lights green, report healthy
+rsp.add(STATUS, OK);
+ }
+
+ /**
+ * Find replicas DOWN or RECOVERING
+ * @param clusterState clusterstate from ZK
+ * @param nodeName this node name
+ * @return list of core names that are either DOWN ore RECOVERING on
'nodeName'
+ */
+ static List findUnhealthyCores(ClusterState clusterState, String
nodeName) {
+return clusterState.getCollectionsMap().values().stream()
Review comment:
> Also if the clusterState thinks that cores live on this node, but the core
directories do not exist, then I think that this handler should respond not
healthy.
Yes, we can add an extra check that for each replica in clusterstate for
node, we check that it exists locally?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1387: SOLR-14210: Include replica health in healtcheck handler
janhoy commented on a change in pull request #1387: SOLR-14210: Include replica
health in healtcheck handler
URL: https://github.com/apache/lucene-solr/pull/1387#discussion_r401431310
##
File path:
solr/core/src/test/org/apache/solr/handler/admin/HealthCheckHandlerTest.java
##
@@ -177,4 +183,13 @@ public void testHealthCheckV2Api() throws Exception {
}
}
+ @Test
+ public void testFindUnhealthyCores() throws Exception {
+ZkStateReader reader =
ClusterStateMockUtil.buildClusterState("csrr2rDcsr2rR", 1, 1, "node1", "node2");
Review comment:
Will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on issue #1387: SOLR-14210: Include replica health in healtcheck handler
janhoy commented on issue #1387: SOLR-14210: Include replica health in healtcheck handler URL: https://github.com/apache/lucene-solr/pull/1387#issuecomment-607101900 I addressed the review comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
Stamatis Zampetakis created LUCENE-9299: --- Summary: ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1 Key: LUCENE-9299 URL: https://issues.apache.org/jira/browse/LUCENE-9299 Project: Lucene - Core Issue Type: Bug Affects Versions: master (9.0) Reporter: Stamatis Zampetakis The visitor that was introduced in LUCENE-8811 (obtained by IndexSearcher#getNumClausesCheckVisitor) that checks that the number of clauses in query does not exceed a certain limit (defined by org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated LUCENE-9299: Attachment: LUCENE-9299.patch Lucene Fields: Patch Available Status: Open (was: Open) > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated LUCENE-9299: Status: Patch Available (was: Open) > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9266) ant nightly-smoke fails due to presence of build.gradle
[ https://issues.apache.org/jira/browse/LUCENE-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072523#comment-17072523 ] Dawid Weiss commented on LUCENE-9266: - You're right - I didn't touch source packaging. Splitting the top-level build won't be as easy as with ant. It can be done in many different ways but it's definitely a larger issue and I would exclude it from this one... I'm really hesitant to say it given the emotions it always entails, but the absolute best way would be to split Lucene as an entirely independent project (with its own gradle build) and then have Solr declare a git submodule "lucene" and its own gradle build that would include Lucene's. This would allow Solr devs to work on head-of-development Lucene and at the same time allow both project builds to be smaller and more focused > ant nightly-smoke fails due to presence of build.gradle > --- > > Key: LUCENE-9266 > URL: https://issues.apache.org/jira/browse/LUCENE-9266 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Mike Drob >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > Seen on Jenkins - > [https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1617/console] > > Reproduced locally. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8811) Add maximum clause count check to IndexSearcher rather than BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072543#comment-17072543 ] Stamatis Zampetakis commented on LUCENE-8811: - Given that the change was reverted on 8.x shouldn't this issue be reopened? I guess the fixVersion should also be modified. > Add maximum clause count check to IndexSearcher rather than BooleanQuery > > > Key: LUCENE-8811 > URL: https://issues.apache.org/jira/browse/LUCENE-8811 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Assignee: Alan Woodward >Priority: Minor > Fix For: 8.2 > > Attachments: LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch, > LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch > > > Currently we only check whether boolean queries have too many clauses. > However there are other ways that queries may have too many clauses, for > instance if you have boolean queries that have themselves inner boolean > queries. > Could we use the new Query visitor API to move this check from BooleanQuery > to IndexSearcher in order to make this check more consistent across queries? > See for instance LUCENE-8810 where a rewrite rule caused the maximum clause > count to be hit even though the total number of leaf queries remained the > same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8811) Add maximum clause count check to IndexSearcher rather than BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072565#comment-17072565 ] Alan Woodward commented on LUCENE-8811: --- I've changed the fixVersion to be 9.0. The change is still in master, so no need to reopen. > Add maximum clause count check to IndexSearcher rather than BooleanQuery > > > Key: LUCENE-8811 > URL: https://issues.apache.org/jira/browse/LUCENE-8811 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Assignee: Alan Woodward >Priority: Minor > Fix For: master (9.0) > > Attachments: LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch, > LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch > > > Currently we only check whether boolean queries have too many clauses. > However there are other ways that queries may have too many clauses, for > instance if you have boolean queries that have themselves inner boolean > queries. > Could we use the new Query visitor API to move this check from BooleanQuery > to IndexSearcher in order to make this check more consistent across queries? > See for instance LUCENE-8810 where a rewrite rule caused the maximum clause > count to be hit even though the total number of leaf queries remained the > same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8811) Add maximum clause count check to IndexSearcher rather than BooleanQuery
[ https://issues.apache.org/jira/browse/LUCENE-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward updated LUCENE-8811: -- Fix Version/s: (was: 8.2) master (9.0) > Add maximum clause count check to IndexSearcher rather than BooleanQuery > > > Key: LUCENE-8811 > URL: https://issues.apache.org/jira/browse/LUCENE-8811 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Assignee: Alan Woodward >Priority: Minor > Fix For: master (9.0) > > Attachments: LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch, > LUCENE-8811.patch, LUCENE-8811.patch, LUCENE-8811.patch > > > Currently we only check whether boolean queries have too many clauses. > However there are other ways that queries may have too many clauses, for > instance if you have boolean queries that have themselves inner boolean > queries. > Could we use the new Query visitor API to move this check from BooleanQuery > to IndexSearcher in order to make this check more consistent across queries? > See for instance LUCENE-8810 where a rewrite rule caused the maximum clause > count to be hit even though the total number of leaf queries remained the > same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072602#comment-17072602 ] Atri Sharma commented on LUCENE-9299: - Can you open a PR instead? Easier to review. I was split with the same decision when working on 8811. IMO, we should be throwing an error when the max clause count is *breached*, which is what the current check accomplishes. Hence this was intentionally done. Keep in mind that this can be potentially a breaking change -- some queries that worked before might stop working all of a sudden post an upgrade. > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072612#comment-17072612 ] Stamatis Zampetakis commented on LUCENE-9299: - Looking in [HowToContribute|https://cwiki.apache.org/confluence/display/LUCENE/HowToContribute]I thought that the patch approach was the nominal way for contributing to the project. Out of curiosity is there another page more up to date? I will open a PR no problem. I don't completely follow your remark. Currently if maxClauseCount is set to 1024 and a query (as the one provided in the unit test) has 1025 clauses the visitor will not throw TooManyClauses exception. Is this expected? > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9170) wagon-ssh Maven HTTPS issue
[ https://issues.apache.org/jira/browse/LUCENE-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072635#comment-17072635 ] ASF subversion and git services commented on LUCENE-9170: - Commit 46d011645cb43e765528d0eec622f97bdf96ef74 in lucene-solr's branch refs/heads/master from Mike Drob [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=46d0116 ] LUCENE-9170: Use HTTPS when downloading wagon-ssh artifacts Co-authored-by: Ishan Chattopadhyaya > wagon-ssh Maven HTTPS issue > --- > > Key: LUCENE-9170 > URL: https://issues.apache.org/jira/browse/LUCENE-9170 > Project: Lucene - Core > Issue Type: Bug >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Fix For: master (9.0), 8.6 > > Attachments: LUCENE-9170.patch, LUCENE-9170.patch, LUCENE-9170.patch > > > When I do, from lucene/ in branch_8_4: > ant -Dversion=8.4.2 generate-maven-artifacts > I see that wagon-ssh is being resolved from http://repo1.maven.org/maven2 > instead of https equivalent. This is surprising to me, since I can't find the > http URL anywhere. > Here's my log: > https://paste.centos.org/view/be2d3f3f > This is a critical issue since releases won't work without this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9170) wagon-ssh Maven HTTPS issue
[ https://issues.apache.org/jira/browse/LUCENE-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072634#comment-17072634 ] ASF subversion and git services commented on LUCENE-9170: - Commit fa22973af4a1a4a353cece4fccc3c5d33dd5c675 in lucene-solr's branch refs/heads/branch_8x from Mike Drob [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=fa22973 ] LUCENE-9170: Use HTTPS when downloading wagon-ssh artifacts Co-authored-by: Ishan Chattopadhyaya > wagon-ssh Maven HTTPS issue > --- > > Key: LUCENE-9170 > URL: https://issues.apache.org/jira/browse/LUCENE-9170 > Project: Lucene - Core > Issue Type: Bug >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Fix For: master (9.0), 8.6 > > Attachments: LUCENE-9170.patch, LUCENE-9170.patch, LUCENE-9170.patch > > > When I do, from lucene/ in branch_8_4: > ant -Dversion=8.4.2 generate-maven-artifacts > I see that wagon-ssh is being resolved from http://repo1.maven.org/maven2 > instead of https equivalent. This is surprising to me, since I can't find the > http URL anywhere. > Here's my log: > https://paste.centos.org/view/be2d3f3f > This is a critical issue since releases won't work without this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14265) Move collections admin API to v2 completely
[ https://issues.apache.org/jira/browse/SOLR-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072648#comment-17072648 ] Andrzej Bialecki commented on SOLR-14265: - We should also move the old apispec JSON descriptor files to the new annotation-based API, it's so much simpler to create and easier to maintain! > Move collections admin API to v2 completely > > > Key: SOLR-14265 > URL: https://issues.apache.org/jira/browse/SOLR-14265 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Anshum Gupta >Assignee: Anshum Gupta >Priority: Major > > V2 admin API has been available in Solr for a very long time, making it > difficult for both users and developers to remember and understand which > format to use when. We should move to v2 API completely for all Solr Admin > calls for the following reasons: > # converge code - there are multiple ways of doing the same thing, there's > unwanted back-compat code, and we should get rid of that > # POJO all the way - no more NamedList. I know this would have split > opinions, but I strongly think we should move in this direction. I created > Jira about this specific task in the past and went half way but I think we > should just close this one out now. > # Automatic documentation > # Others > This is just an umbrella Jira for the task. Let's create sub-tasks and split > this up as it would require a bunch of rewriting of the code and it makes a > lot of sense to get this out with 9.0 so we don't have to support v1 forever! > There have been some conversations going on about this and it feels like most > folks are happy to go this route. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9300) Index corruption with doc values updates and addIndexes
Jim Ferenczi created LUCENE-9300: Summary: Index corruption with doc values updates and addIndexes Key: LUCENE-9300 URL: https://issues.apache.org/jira/browse/LUCENE-9300 Project: Lucene - Core Issue Type: Improvement Reporter: Jim Ferenczi Today a doc values update creates a new field infos file that contains the original field infos updated for the new generation as well as the new fields created by the doc values update. However existing fields are cloned through the global fields (shared in the index writer) instead of the local ones (present in the segment). In practice this is not an issue since field numbers are shared between segments created by the same index writer. But this assumption doesn't hold for segments created by different writers and added through IndexWriter#addIndexes(Directory). In this case, the field number of the same field can differ between segments so any doc values update can corrupt the index by assigning the wrong field number to an existing field in the next generation. When this happens, queries and merges can access wrong fields without throwing any error, leading to a silent corruption in the index. Since segments are not guaranteed to have the same field number consistently we should ensure that doc values update preserves the segment's field number when rewriting field infos. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9300) Index corruption with doc values updates and addIndexes
[ https://issues.apache.org/jira/browse/LUCENE-9300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Ferenczi updated LUCENE-9300: - Issue Type: Bug (was: Improvement) > Index corruption with doc values updates and addIndexes > --- > > Key: LUCENE-9300 > URL: https://issues.apache.org/jira/browse/LUCENE-9300 > Project: Lucene - Core > Issue Type: Bug >Reporter: Jim Ferenczi >Priority: Major > > Today a doc values update creates a new field infos file that contains the > original field infos updated for the new generation as well as the new fields > created by the doc values update. > However existing fields are cloned through the global fields (shared in the > index writer) instead of the local ones (present in the segment). In practice > this is not an issue since field numbers are shared between segments created > by the same index writer. But this assumption doesn't hold for segments > created by different writers and added through > IndexWriter#addIndexes(Directory). In this case, the field number of the same > field can differ between segments so any doc values update can corrupt the > index by assigning the wrong field number to an existing field in the next > generation. > When this happens, queries and merges can access wrong fields without > throwing any error, leading to a silent corruption in the index. > > Since segments are not guaranteed to have the same field number consistently > we should ensure that doc values update preserves the segment's field number > when rewriting field infos. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jimczi opened a new pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jimczi opened a new pull request #1394: LUCENE-9300: Fix field infos update on doc values update URL: https://github.com/apache/lucene-solr/pull/1394 Today a doc values update creates a new field infos file that contains the original field infos updated for the new generation as well as the new fields created by the doc values update. However existing fields are cloned through the global fields (shared in the index writer) instead of the local ones (present in the segment). In practice this is not an issue since field numbers are shared between segments created by the same index writer. But this assumption doesn't hold for segments created by different writers and added through IndexWriter#addIndexes(Directory). In this case, the field number of the same field can differ between segments so any doc values update can corrupt the index by assigning the wrong field number to an existing field in the next generation. When this happens, queries and merges can access wrong fields without throwing any error, leading to a silent corruption in the index. This change ensures that we preserve local field numbers when creating a new field infos generation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
atris commented on a change in pull request #1294: LUCENE-9074: Slice
Allocation Control Plane For Concurrent Searches
URL: https://github.com/apache/lucene-solr/pull/1294#discussion_r401542281
##
File path: lucene/core/src/test/org/apache/lucene/search/TestIndexSearcher.java
##
@@ -70,13 +75,31 @@ public void setUp() throws Exception {
}
reader = iw.getReader();
iw.close();
+
+Random random = random();
+RandomIndexWriter iw2 = new RandomIndexWriter(random(), dir2,
newIndexWriterConfig().setMergePolicy(newLogMergePolicy()));
+for (int i = 0; i < 100; i++) {
+ Document doc = new Document();
+ doc.add(newStringField("field", Integer.toString(i), Field.Store.NO));
+ doc.add(newStringField("field2", Boolean.toString(i % 2 == 0),
Field.Store.NO));
+ doc.add(new SortedDocValuesField("field2", new
BytesRef(Boolean.toString(i % 2 == 0;
+ iw2.addDocument(doc);
+
+ if (random.nextBoolean()) {
+iw2.commit();
+ }
+}
+reader2 = iw2.getReader();
+iw2.close();
Review comment:
reader2 has random commits baked into it to ensure random slice
distributions each time the test is run.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
atris commented on a change in pull request #1294: LUCENE-9074: Slice
Allocation Control Plane For Concurrent Searches
URL: https://github.com/apache/lucene-solr/pull/1294#discussion_r401544477
##
File path: lucene/core/src/java/org/apache/lucene/search/SliceExecutor.java
##
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.util.Collection;
+import java.util.concurrent.Executor;
+import java.util.concurrent.RejectedExecutionException;
+
+/**
+ * Executor which is responsible
+ * for execution of slices based on the current status
+ * of the system and current system load
+ */
+class SliceExecutor {
+ private final Executor executor;
+
+ public SliceExecutor(Executor executor) {
+this.executor = executor;
+ }
+
+ public void invokeAll(Collection tasks) {
+
+if (tasks == null) {
+ throw new IllegalArgumentException("Tasks is null");
+}
+
+if (executor == null) {
+ throw new IllegalArgumentException("Executor is null");
+}
+
+int i = 0;
+
+for (Runnable task : tasks) {
+ boolean shouldExecuteOnCallerThread = false;
+
+ // Execute last task on caller thread
+ if (i == tasks.size() - 1) {
+shouldExecuteOnCallerThread = true;
+ }
+
+ processTask(task, shouldExecuteOnCallerThread);
+ ++i;
+};
+ }
+
+ // Helper method to execute a single task
+ protected void processTask(final Runnable task,
+ final boolean shouldExecuteOnCallerThread) {
+if (task == null) {
+ throw new IllegalArgumentException("Input is null");
+}
+
+if (!shouldExecuteOnCallerThread) {
+ try {
+executor.execute(task);
+
+return;
+ } catch (RejectedExecutionException e) {
+// Execute on caller thread
+ }
+}
+
+runTaskOnCallerThread(task);
+ }
+
+ // Private helper method to run a task on the caller thread
+ private void runTaskOnCallerThread(Runnable task) {
+task.run();
+ }
Review comment:
Fair point -- I dont know what I was thinking :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
jpountz commented on a change in pull request #1294: LUCENE-9074: Slice
Allocation Control Plane For Concurrent Searches
URL: https://github.com/apache/lucene-solr/pull/1294#discussion_r401555369
##
File path: lucene/core/src/test/org/apache/lucene/search/TestIndexSearcher.java
##
@@ -70,13 +75,31 @@ public void setUp() throws Exception {
}
reader = iw.getReader();
iw.close();
+
+Random random = random();
+RandomIndexWriter iw2 = new RandomIndexWriter(random(), dir2,
newIndexWriterConfig().setMergePolicy(newLogMergePolicy()));
+for (int i = 0; i < 100; i++) {
+ Document doc = new Document();
+ doc.add(newStringField("field", Integer.toString(i), Field.Store.NO));
+ doc.add(newStringField("field2", Boolean.toString(i % 2 == 0),
Field.Store.NO));
+ doc.add(new SortedDocValuesField("field2", new
BytesRef(Boolean.toString(i % 2 == 0;
+ iw2.addDocument(doc);
+
+ if (random.nextBoolean()) {
+iw2.commit();
+ }
+}
+reader2 = iw2.getReader();
+iw2.close();
Review comment:
Would it be a problem to add these random commits to the existing reader?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] juanka588 commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
juanka588 commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401559955
##
File path:
lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
##
@@ -1483,6 +1484,81 @@ public void testAddIndexes() throws Exception {
IOUtils.close(dir1, dir2);
}
+ public void testUpdatesAterAddIndexes() throws Exception {
Review comment:
after?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
atris commented on a change in pull request #1294: LUCENE-9074: Slice
Allocation Control Plane For Concurrent Searches
URL: https://github.com/apache/lucene-solr/pull/1294#discussion_r401565776
##
File path: lucene/core/src/test/org/apache/lucene/search/TestIndexSearcher.java
##
@@ -70,13 +75,31 @@ public void setUp() throws Exception {
}
reader = iw.getReader();
iw.close();
+
+Random random = random();
+RandomIndexWriter iw2 = new RandomIndexWriter(random(), dir2,
newIndexWriterConfig().setMergePolicy(newLogMergePolicy()));
+for (int i = 0; i < 100; i++) {
+ Document doc = new Document();
+ doc.add(newStringField("field", Integer.toString(i), Field.Store.NO));
+ doc.add(newStringField("field2", Boolean.toString(i % 2 == 0),
Field.Store.NO));
+ doc.add(new SortedDocValuesField("field2", new
BytesRef(Boolean.toString(i % 2 == 0;
+ iw2.addDocument(doc);
+
+ if (random.nextBoolean()) {
+iw2.commit();
+ }
+}
+reader2 = iw2.getReader();
+iw2.close();
Review comment:
I did not want to touch the existing reader since it is used by many tests
-- dont see any risk though, given the specifics of the tests that use the
existing reader, should be safe. Updated, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on issue #895: LUCENE-8985: SynonymGraphFilter cannot handle input stream with tokens filtered.
janhoy commented on issue #895: LUCENE-8985: SynonymGraphFilter cannot handle input stream with tokens filtered. URL: https://github.com/apache/lucene-solr/pull/895#issuecomment-607219248 @chenkovsky I'd like to help move this forward, and I think the next step is to add 5-10 more unit tests that prove that behavior is not broken in unpredictable ways. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on issue #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
atris commented on issue #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches URL: https://github.com/apache/lucene-solr/pull/1294#issuecomment-60729 @jpountz Updated. Please see and let me know your comments and thoughts This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072722#comment-17072722 ] Atri Sharma commented on LUCENE-9299: - Ok, I am misreading your patch then -- will give it a run later today. Thanks for picking it up. > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned LUCENE-9299: --- Assignee: Atri Sharma > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Assignee: Atri Sharma >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source
dweiss commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source URL: https://github.com/apache/lucene-solr/pull/1390#issuecomment-607224015 [changes.patch.txt](https://github.com/apache/lucene-solr/files/4415071/changes.patch.txt) Hi Mike. I've restructured the downloader class a bit so that gradle wrapper version and checksum are aligned with the target file - makes it a single place to change things (instead of messing with the scripts). I also simplified error messaging a bit, hope it's ok. More interestingly I discovered a hideous problem with CLASSPATH environment variable: this is used by java in single-source mode and was locking the wrapper file (on Windows), effectively preventing any changes (in case of overwriting due to a wrong checksum, for example). Corrected. Looks good to me to commit this in (removing the gradle-wrapper.jar file). Anything else can be a follow-up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[
https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072729#comment-17072729
]
Lucene/Solr QA commented on LUCENE-9299:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 0m 38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
20s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-9299 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998420/LUCENE-9299.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene2-us-west.apache.org 4.4.0-170-generic #199-Ubuntu SMP
Thu Nov 14 01:45:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 46d0116 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-LUCENE-Build/260/testReport/ |
| modules | C: lucene lucene/core U: lucene |
| Console output |
https://builds.apache.org/job/PreCommit-LUCENE-Build/260/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses
> in a query exceeds the limit by 1
> --
>
> Key: LUCENE-9299
> URL: https://issues.apache.org/jira/browse/LUCENE-9299
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: master (9.0)
>Reporter: Stamatis Zampetakis
>Assignee: Atri Sharma
>Priority: Trivial
> Attachments: LUCENE-9299.patch
>
>
> The visitor that was introduced in LUCENE-8811 (obtained by
> IndexSearcher#getNumClausesCheckVisitor) that checks that the number of
> clauses in query does not exceed a certain limit (defined by
> org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case
> where the clauses/terms in the query exceed the limit by one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] gerlowskija merged pull request #1379: SOLR-14363: Separate /get requests into their own type designation
gerlowskija merged pull request #1379: SOLR-14363: Separate /get requests into their own type designation URL: https://github.com/apache/lucene-solr/pull/1379 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
jpountz commented on a change in pull request #1294: LUCENE-9074: Slice
Allocation Control Plane For Concurrent Searches
URL: https://github.com/apache/lucene-solr/pull/1294#discussion_r401583032
##
File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
##
@@ -208,9 +210,21 @@ public IndexSearcher(IndexReader r, Executor executor) {
* @lucene.experimental
*/
public IndexSearcher(IndexReaderContext context, Executor executor) {
+this(context, executor, getSliceExecutionControlPlane(executor));
+ }
+
+ // Package private for testing
+ IndexSearcher(IndexReaderContext context, Executor executor, SliceExecutor
sliceExecutor) {
assert context.isTopLevel: "IndexSearcher's ReaderContext must be topLevel
for reader" + context.reader();
+if (executor == null) {
+ assert sliceExecutor == null;
+}
Review comment:
this is covered by the below assert already?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14363) Add separate type for "/get" requests to SolrLogParseTool
[ https://issues.apache.org/jira/browse/SOLR-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072731#comment-17072731 ] ASF subversion and git services commented on SOLR-14363: Commit 1f5705ff5cbaf77fe75a07159a37f243d246a90c in lucene-solr's branch refs/heads/master from Jason Gerlowski [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=1f5705f ] SOLR-14363: Separate /get requests into their own type designation (#1379) > Add separate type for "/get" requests to SolrLogParseTool > - > > Key: SOLR-14363 > URL: https://issues.apache.org/jira/browse/SOLR-14363 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: scripts and tools >Affects Versions: master (9.0) >Reporter: Jason Gerlowski >Assignee: Jason Gerlowski >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > The SolrLogParseTool parses most log records containing the string "QTime=" > as being a query. While this is useful, the designation would be a little > more helpful if operations with a different performance profile were broken > out into their own "type_s" value. That gives users the ability to view them > together or separately, as they prefer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
jpountz commented on a change in pull request #1294: LUCENE-9074: Slice
Allocation Control Plane For Concurrent Searches
URL: https://github.com/apache/lucene-solr/pull/1294#discussion_r401584443
##
File path: lucene/core/src/test/org/apache/lucene/search/TestIndexSearcher.java
##
@@ -347,4 +354,89 @@ public void execute(final Runnable runnable) {
throw new RejectedExecutionException();
}
}
+
+ public void testQueueSizeBasedSliceExecutor() throws Exception {
+ThreadPoolExecutor service = new ThreadPoolExecutor(4, 4, 0L,
TimeUnit.MILLISECONDS,
+new LinkedBlockingQueue(),
+new NamedThreadFactory("TestIndexSearcher"));
+
+runSliceExecutorTest(service, false);
+
+TestUtil.shutdownExecutorService(service);
+ }
+
+ public void testRandomBlockingSliceExecutor() throws Exception {
+ThreadPoolExecutor service = new ThreadPoolExecutor(4, 4, 0L,
TimeUnit.MILLISECONDS,
+new LinkedBlockingQueue(),
+new NamedThreadFactory("TestIndexSearcher"));
+
+runSliceExecutorTest(service, true);
+
+TestUtil.shutdownExecutorService(service);
+ }
+
+ private void runSliceExecutorTest(ThreadPoolExecutor service, boolean
useRandomSliceExecutor) throws Exception {
+SliceExecutor sliceExecutor = useRandomSliceExecutor == true ? new
RandomBlockingSliceExecutor(service) :
+ new
QueueSizeBasedExecutor(service);
+
+IndexSearcher searcher = new IndexSearcher(reader.getContext(), service,
sliceExecutor);
+
+Query queries[] = new Query[] {
+new MatchAllDocsQuery(),
+new TermQuery(new Term("field", "1"))
+};
+Sort sorts[] = new Sort[] {
+null,
+new Sort(new SortField("field2", SortField.Type.STRING))
+};
+ScoreDoc afters[] = new ScoreDoc[] {
+null,
+new FieldDoc(0, 0f, new Object[] { new BytesRef("boo!") })
+};
+
+for (ScoreDoc after : afters) {
+ for (Query query : queries) {
+for (Sort sort : sorts) {
+ searcher.search(query, Integer.MAX_VALUE);
+ searcher.searchAfter(after, query, Integer.MAX_VALUE);
+ if (sort != null) {
+TopDocs topDocs = searcher.search(query, Integer.MAX_VALUE, sort);
+assert topDocs.totalHits.value > 0;
+
+topDocs = searcher.search(query, Integer.MAX_VALUE, sort, true);
+assert topDocs.totalHits.value > 0;
+
+topDocs = searcher.search(query, Integer.MAX_VALUE, sort, false);
+assert topDocs.totalHits.value > 0;
+
+topDocs = searcher.searchAfter(after, query, Integer.MAX_VALUE,
sort);
+assert topDocs.totalHits.value > 0;
+
+topDocs = searcher.searchAfter(after, query, Integer.MAX_VALUE,
sort, true);
+assert topDocs.totalHits.value > 0;
+
+topDocs = searcher.searchAfter(after, query, Integer.MAX_VALUE,
sort, false);
+assert topDocs.totalHits.value > 0;
Review comment:
we should use assertTrue rather than assert in tests, since the latter only
runs when assertions are enabled, and it can be useful to run tests with
assertions disabled to make sure that the logic remains correct when assertions
are off
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14363) Add separate type for "/get" requests to SolrLogParseTool
[ https://issues.apache.org/jira/browse/SOLR-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072749#comment-17072749 ] ASF subversion and git services commented on SOLR-14363: Commit e463934ab190dd9950616130897f477f7d1e72f8 in lucene-solr's branch refs/heads/branch_8x from Jason Gerlowski [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e463934 ] SOLR-14363: Separate /get requests into their own type designation (#1379) > Add separate type for "/get" requests to SolrLogParseTool > - > > Key: SOLR-14363 > URL: https://issues.apache.org/jira/browse/SOLR-14363 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: scripts and tools >Affects Versions: master (9.0) >Reporter: Jason Gerlowski >Assignee: Jason Gerlowski >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > The SolrLogParseTool parses most log records containing the string "QTime=" > as being a query. While this is useful, the designation would be a little > more helpful if operations with a different performance profile were broken > out into their own "type_s" value. That gives users the ability to view them > together or separately, as they prefer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14363) Add separate type for "/get" requests to SolrLogParseTool
[ https://issues.apache.org/jira/browse/SOLR-14363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gerlowski resolved SOLR-14363. Fix Version/s: 8.6 master (9.0) Resolution: Fixed Merged in for 8.6 and 9.0. > Add separate type for "/get" requests to SolrLogParseTool > - > > Key: SOLR-14363 > URL: https://issues.apache.org/jira/browse/SOLR-14363 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: scripts and tools >Affects Versions: master (9.0) >Reporter: Jason Gerlowski >Assignee: Jason Gerlowski >Priority: Minor > Fix For: master (9.0), 8.6 > > Time Spent: 1h > Remaining Estimate: 0h > > The SolrLogParseTool parses most log records containing the string "QTime=" > as being a query. While this is useful, the designation would be a little > more helpful if operations with a different performance profile were broken > out into their own "type_s" value. That gives users the ability to view them > together or separately, as they prefer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jpountz commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401619000
##
File path: lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java
##
@@ -543,27 +543,30 @@ public synchronized boolean writeFieldUpdates(Directory
dir, FieldInfos.FieldNum
try {
// clone FieldInfos so that we can update their dvGen separately from
-// the reader's infos and write them to a new fieldInfos_gen file
-FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
-// cannot use builder.add(reader.getFieldInfos()) because it does not
-// clone FI.attributes as well FI.dvGen
+// the reader's infos and write them to a new fieldInfos_gen file.
+Map newFields = new HashMap<>();
for (FieldInfo fi : reader.getFieldInfos()) {
- FieldInfo clone = builder.add(fi);
- // copy the stuff FieldInfos.Builder doesn't copy
- for (Entry e : fi.attributes().entrySet()) {
-clone.putAttribute(e.getKey(), e.getValue());
- }
- clone.setDocValuesGen(fi.getDocValuesGen());
+ // cannot use builder.add(fi) because it does not preserve
+ // the local field number. Field numbers can be different from the
global ones
+ // if the segment was created externally (with
IndexWriter#addIndexes(Directory)).
+ FieldInfo clone = new FieldInfo(fi);
+ newFields.put(clone.name, clone);
}
// create new fields with the right DV type
+FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
for (List updates : pendingDVUpdates.values()) {
DocValuesFieldUpdates update = updates.get(0);
- FieldInfo fieldInfo = builder.getOrAdd(update.field);
- fieldInfo.setDocValuesType(update.type);
+ FieldInfo fi = newFields.get(update.field);
+ if (fi == null) {
+// the field is not present in this segment so we can fallback to
the global fields.
+fi = builder.getOrAdd(update.field);
Review comment:
Had a quick discussion about this line with Jim. This isn't good enough as
this might return a field number that already exists in the reader if the field
exists in the writer with a number that it less than the number of fields in
the reader.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072788#comment-17072788 ] Tommaso Teofili commented on LUCENE-9299: - the changes look good to me. > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Assignee: Atri Sharma >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9004) Approximate nearest vector search
[
https://issues.apache.org/jira/browse/LUCENE-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072803#comment-17072803
]
Tommaso Teofili commented on LUCENE-9004:
-
I agree with [~jim.ferenczi] as discussions in this issue are super
interesting.
I like the idea of keeping per entry neighborhood information layer-wise only.
I am wondering if this can be precomputed and maybe stored for faster
computation when needed, e.g. sparsifying single node neighbors should be
possible hence we could even reuse a temporary per node posting list.
> Approximate nearest vector search
> -
>
> Key: LUCENE-9004
> URL: https://issues.apache.org/jira/browse/LUCENE-9004
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Michael Sokolov
>Priority: Major
> Attachments: hnsw_layered_graph.png
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> "Semantic" search based on machine-learned vector "embeddings" representing
> terms, queries and documents is becoming a must-have feature for a modern
> search engine. SOLR-12890 is exploring various approaches to this, including
> providing vector-based scoring functions. This is a spinoff issue from that.
> The idea here is to explore approximate nearest-neighbor search. Researchers
> have found an approach based on navigating a graph that partially encodes the
> nearest neighbor relation at multiple scales can provide accuracy > 95% (as
> compared to exact nearest neighbor calculations) at a reasonable cost. This
> issue will explore implementing HNSW (hierarchical navigable small-world)
> graphs for the purpose of approximate nearest vector search (often referred
> to as KNN or k-nearest-neighbor search).
> At a high level the way this algorithm works is this. First assume you have a
> graph that has a partial encoding of the nearest neighbor relation, with some
> short and some long-distance links. If this graph is built in the right way
> (has the hierarchical navigable small world property), then you can
> efficiently traverse it to find nearest neighbors (approximately) in log N
> time where N is the number of nodes in the graph. I believe this idea was
> pioneered in [1]. The great insight in that paper is that if you use the
> graph search algorithm to find the K nearest neighbors of a new document
> while indexing, and then link those neighbors (undirectedly, ie both ways) to
> the new document, then the graph that emerges will have the desired
> properties.
> The implementation I propose for Lucene is as follows. We need two new data
> structures to encode the vectors and the graph. We can encode vectors using a
> light wrapper around {{BinaryDocValues}} (we also want to encode the vector
> dimension and have efficient conversion from bytes to floats). For the graph
> we can use {{SortedNumericDocValues}} where the values we encode are the
> docids of the related documents. Encoding the interdocument relations using
> docids directly will make it relatively fast to traverse the graph since we
> won't need to lookup through an id-field indirection. This choice limits us
> to building a graph-per-segment since it would be impractical to maintain a
> global graph for the whole index in the face of segment merges. However
> graph-per-segment is a very natural at search time - we can traverse each
> segments' graph independently and merge results as we do today for term-based
> search.
> At index time, however, merging graphs is somewhat challenging. While
> indexing we build a graph incrementally, performing searches to construct
> links among neighbors. When merging segments we must construct a new graph
> containing elements of all the merged segments. Ideally we would somehow
> preserve the work done when building the initial graphs, but at least as a
> start I'd propose we construct a new graph from scratch when merging. The
> process is going to be limited, at least initially, to graphs that can fit
> in RAM since we require random access to the entire graph while constructing
> it: In order to add links bidirectionally we must continually update existing
> documents.
> I think we want to express this API to users as a single joint
> {{KnnGraphField}} abstraction that joins together the vectors and the graph
> as a single joint field type. Mostly it just looks like a vector-valued
> field, but has this graph attached to it.
> I'll push a branch with my POC and would love to hear comments. It has many
> nocommits, basic design is not really set, there is no Query implementation
> and no integration iwth IndexSearcher, but it does work by some measure using
> a standalone test class. I've tested with uniform random vectors and on my
> laptop indexed 10K documents in around 10 seconds and searche
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401672435
##
File path: lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java
##
@@ -543,27 +543,30 @@ public synchronized boolean writeFieldUpdates(Directory
dir, FieldInfos.FieldNum
try {
// clone FieldInfos so that we can update their dvGen separately from
-// the reader's infos and write them to a new fieldInfos_gen file
-FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
-// cannot use builder.add(reader.getFieldInfos()) because it does not
-// clone FI.attributes as well FI.dvGen
+// the reader's infos and write them to a new fieldInfos_gen file.
+Map newFields = new HashMap<>();
for (FieldInfo fi : reader.getFieldInfos()) {
- FieldInfo clone = builder.add(fi);
- // copy the stuff FieldInfos.Builder doesn't copy
- for (Entry e : fi.attributes().entrySet()) {
-clone.putAttribute(e.getKey(), e.getValue());
- }
- clone.setDocValuesGen(fi.getDocValuesGen());
+ // cannot use builder.add(fi) because it does not preserve
+ // the local field number. Field numbers can be different from the
global ones
+ // if the segment was created externally (with
IndexWriter#addIndexes(Directory)).
+ FieldInfo clone = new FieldInfo(fi);
+ newFields.put(clone.name, clone);
}
// create new fields with the right DV type
+FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
for (List updates : pendingDVUpdates.values()) {
DocValuesFieldUpdates update = updates.get(0);
- FieldInfo fieldInfo = builder.getOrAdd(update.field);
- fieldInfo.setDocValuesType(update.type);
+ FieldInfo fi = newFields.get(update.field);
+ if (fi == null) {
+// the field is not present in this segment so we can fallback to
the global fields.
+fi = builder.getOrAdd(update.field);
Review comment:
How to introduce the same bug but the other way around :(. Thanks for
catching this Adrien. I pushed
https://github.com/apache/lucene-solr/pull/1394/commits/abd5e08170cc8b7963c0aff3d58afa5766afcb32
to handle this case properly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401672524
##
File path:
lucene/core/src/test/org/apache/lucene/index/TestNumericDocValuesUpdates.java
##
@@ -1483,6 +1484,81 @@ public void testAddIndexes() throws Exception {
IOUtils.close(dir1, dir2);
}
+ public void testUpdatesAterAddIndexes() throws Exception {
Review comment:
thanks,
https://github.com/apache/lucene-solr/pull/1394/commits/abd5e08170cc8b7963c0aff3d58afa5766afcb32
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] HoustonPutman commented on a change in pull request #1387: SOLR-14210: Include replica health in healtcheck handler
HoustonPutman commented on a change in pull request #1387: SOLR-14210: Include
replica health in healtcheck handler
URL: https://github.com/apache/lucene-solr/pull/1387#discussion_r401691874
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/HealthCheckHandler.java
##
@@ -88,15 +95,42 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
return;
}
-// Set status to true if this node is in live_nodes
-if
(clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
- rsp.add(STATUS, OK);
-} else {
+// Fail if not in live_nodes
+if
(!clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
rsp.add(STATUS, FAILURE);
rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Host Unavailable:
Not in live nodes as per zk"));
+ return;
}
-rsp.setHttpCaching(false);
+// Optionally require that all cores on this node are active if param
'failWhenRecovering=true'
+if (req.getParams().getBool(PARAM_REQUIRE_HEALTHY_CORES, false)) {
+ List unhealthyCores = findUnhealthyCores(clusterState,
cores.getNodeConfig().getNodeName());
+ if (unhealthyCores.size() > 0) {
+ rsp.add(STATUS, FAILURE);
+ rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE,
+ "Replica(s) " + unhealthyCores + " are currently
initializing or recovering"));
+ return;
+ }
+ rsp.add("MESSAGE", "All cores are healthy");
+}
+
+// All lights green, report healthy
+rsp.add(STATUS, OK);
+ }
+
+ /**
+ * Find replicas DOWN or RECOVERING
+ * @param clusterState clusterstate from ZK
+ * @param nodeName this node name
+ * @return list of core names that are either DOWN ore RECOVERING on
'nodeName'
+ */
+ static List findUnhealthyCores(ClusterState clusterState, String
nodeName) {
+return clusterState.getCollectionsMap().values().stream()
Review comment:
> > Also if the clusterState thinks that cores live on this node, but the
core directories do not exist, then I think that this handler should respond
not healthy.
>
> Yes, we can add an extra check that for each replica in clusterstate for
node, we check that it exists locally?
I think that the logic as it stands now should work, because the cluster
state will report the replica as "DOWN" (From my experience, but we can also
add tests around this). The comment was meant to validate the current approach
over the other one you mentioned:
> The alternative is to instead iterate cores on current node, and consult
with clusterState their overall state.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris merged pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
atris merged pull request #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches URL: https://github.com/apache/lucene-solr/pull/1294 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9074) Account for Executor's Queue Length When Planning Slices in IndexSearcher
[ https://issues.apache.org/jira/browse/LUCENE-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072852#comment-17072852 ] ASF subversion and git services commented on LUCENE-9074: - Commit 9ed71a6efe2d11fd184de53d4191753b2183b73e in lucene-solr's branch refs/heads/master from Atri Sharma [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9ed71a6 ] LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches (#1294) This commit introduces a mechanism to control allocation of threads to slices planned for a query. The default implementation uses the size of backlog queue of the executor to determine if a slice should be allocated a new thread > Account for Executor's Queue Length When Planning Slices in IndexSearcher > - > > Key: LUCENE-9074 > URL: https://issues.apache.org/jira/browse/LUCENE-9074 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on issue #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches
atris commented on issue #1294: LUCENE-9074: Slice Allocation Control Plane For Concurrent Searches URL: https://github.com/apache/lucene-solr/pull/1294#issuecomment-607309069 Thank you @jpountz This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] HoustonPutman commented on a change in pull request #1387: SOLR-14210: Include replica health in healtcheck handler
HoustonPutman commented on a change in pull request #1387: SOLR-14210: Include
replica health in healtcheck handler
URL: https://github.com/apache/lucene-solr/pull/1387#discussion_r401708980
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/HealthCheckHandler.java
##
@@ -88,15 +95,42 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
return;
}
-// Set status to true if this node is in live_nodes
-if
(clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
- rsp.add(STATUS, OK);
-} else {
+// Fail if not in live_nodes
+if
(!clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
rsp.add(STATUS, FAILURE);
rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Host Unavailable:
Not in live nodes as per zk"));
+ return;
}
-rsp.setHttpCaching(false);
+// Optionally require that all cores on this node are active if param
'failWhenRecovering=true'
+if (req.getParams().getBool(PARAM_REQUIRE_HEALTHY_CORES, false)) {
+ List unhealthyCores = findUnhealthyCores(clusterState,
cores.getNodeConfig().getNodeName());
+ if (unhealthyCores.size() > 0) {
+ rsp.add(STATUS, FAILURE);
+ rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE,
+ "Replica(s) " + unhealthyCores + " are currently
initializing or recovering"));
+ return;
+ }
+ rsp.add("MESSAGE", "All cores are healthy");
+}
+
+// All lights green, report healthy
+rsp.add(STATUS, OK);
+ }
+
+ /**
+ * Find replicas DOWN or RECOVERING
+ * @param clusterState clusterstate from ZK
+ * @param nodeName this node name
+ * @return list of core names that are either DOWN ore RECOVERING on
'nodeName'
+ */
+ static List findUnhealthyCores(ClusterState clusterState, String
nodeName) {
+return clusterState.getCollectionsMap().values().stream()
Review comment:
> If a shard split is currently running (could be long running), on a node
being restarted, the split would be aborted but when the node comes up again I
believe the overseer might try again??
And I understand your decision for the active shards better now. As long as
the overseer thing is true, then we should be fine. And if in the future if we
need to, we can add another parameter to fail on inactive slices as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9301) Gradle: Jar MANIFEST incomplete
Jan Høydahl created LUCENE-9301:
---
Summary: Gradle: Jar MANIFEST incomplete
Key: LUCENE-9301
URL: https://issues.apache.org/jira/browse/LUCENE-9301
Project: Lucene - Core
Issue Type: Bug
Components: general/build
Reporter: Jan Høydahl
After building with gradle, the MANIFEST.MF file for e.g. solr-core.jar
containst
{noformat}
Manifest-Version: 1.0
{noformat}
While when building with ant, it says
{noformat}
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.10.7
Created-By: 11.0.6+10 (AdoptOpenJDK)
Extension-Name: org.apache.solr
Specification-Title: Apache Solr Search Server: solr-core
Specification-Version: 9.0.0
Specification-Vendor: The Apache Software Foundation
Implementation-Title: org.apache.solr
Implementation-Version: 9.0.0-SNAPSHOT 9b5542ad55da601e0bdfda96bad8c2c
cabbbc397 - janhoy - 2020-04-01 16:24:09
Implementation-Vendor: The Apache Software Foundation
X-Compile-Source-JDK: 11
X-Compile-Target-JDK: 11
{noformat}
In addition, with ant, the META-INF folder also contains LICENSE.txt and
NOTICE.txt files.
There is a macro {{build-manifest}} in common-build.xml that seems to build the
manifest.
The effect of this is e.g. that spec an implementation versions do not show in
Solr Admin UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9301) Gradle: Jar MANIFEST incomplete
[
https://issues.apache.org/jira/browse/LUCENE-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated LUCENE-9301:
Affects Version/s: master (9.0)
> Gradle: Jar MANIFEST incomplete
> ---
>
> Key: LUCENE-9301
> URL: https://issues.apache.org/jira/browse/LUCENE-9301
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: master (9.0)
>Reporter: Jan Høydahl
>Priority: Major
>
> After building with gradle, the MANIFEST.MF file for e.g. solr-core.jar
> containst
> {noformat}
> Manifest-Version: 1.0
> {noformat}
> While when building with ant, it says
> {noformat}
> Manifest-Version: 1.0
> Ant-Version: Apache Ant 1.10.7
> Created-By: 11.0.6+10 (AdoptOpenJDK)
> Extension-Name: org.apache.solr
> Specification-Title: Apache Solr Search Server: solr-core
> Specification-Version: 9.0.0
> Specification-Vendor: The Apache Software Foundation
> Implementation-Title: org.apache.solr
> Implementation-Version: 9.0.0-SNAPSHOT 9b5542ad55da601e0bdfda96bad8c2c
> cabbbc397 - janhoy - 2020-04-01 16:24:09
> Implementation-Vendor: The Apache Software Foundation
> X-Compile-Source-JDK: 11
> X-Compile-Target-JDK: 11
> {noformat}
> In addition, with ant, the META-INF folder also contains LICENSE.txt and
> NOTICE.txt files.
> There is a macro {{build-manifest}} in common-build.xml that seems to build
> the manifest.
> The effect of this is e.g. that spec an implementation versions do not show
> in Solr Admin UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] HoustonPutman commented on issue #1387: SOLR-14210: Include replica health in healtcheck handler
HoustonPutman commented on issue #1387: SOLR-14210: Include replica health in healtcheck handler URL: https://github.com/apache/lucene-solr/pull/1387#issuecomment-607327108 Are you targeting this for 8.6 or 9.0? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jpountz commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401719320
##
File path: lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java
##
@@ -543,27 +543,36 @@ public synchronized boolean writeFieldUpdates(Directory
dir, FieldInfos.FieldNum
try {
// clone FieldInfos so that we can update their dvGen separately from
-// the reader's infos and write them to a new fieldInfos_gen file
-FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
-// cannot use builder.add(reader.getFieldInfos()) because it does not
-// clone FI.attributes as well FI.dvGen
+// the reader's infos and write them to a new fieldInfos_gen file.
+int maxFieldNumber = -1;
+Map newFields = new HashMap<>();
for (FieldInfo fi : reader.getFieldInfos()) {
- FieldInfo clone = builder.add(fi);
- // copy the stuff FieldInfos.Builder doesn't copy
- for (Entry e : fi.attributes().entrySet()) {
-clone.putAttribute(e.getKey(), e.getValue());
- }
- clone.setDocValuesGen(fi.getDocValuesGen());
+ // cannot use builder.add(fi) because it does not preserve
+ // the local field number. Field numbers can be different from the
global ones
+ // if the segment was created externally (with
IndexWriter#addIndexes(Directory)).
+ FieldInfo clone = cloneFieldInfo(fi, fi.number);
+ newFields.put(clone.name, clone);
+ maxFieldNumber = Math.max(clone.number, maxFieldNumber);
}
// create new fields with the right DV type
+FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
for (List updates : pendingDVUpdates.values()) {
DocValuesFieldUpdates update = updates.get(0);
- FieldInfo fieldInfo = builder.getOrAdd(update.field);
- fieldInfo.setDocValuesType(update.type);
+ FieldInfo fi = newFields.get(update.field);
+ if (fi == null) {
+// the field is not present in this segment so we can fallback to
the global fields.
+fi = builder.getOrAdd(update.field);
+if (fi.number <= maxFieldNumber) {
+ // the global field number is already used in this segment for a
different field so we force a new one locally.
+ fi = cloneFieldInfo(fi, ++maxFieldNumber);
Review comment:
Do we need to set the doc-value type before cloning to make sure the
doc-value type is set on the writer? I also wonder if we should use
`FieldInfos.Builder` since this logic you implemented looks similar to
`FieldInfos.Builder#add`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] HoustonPutman commented on a change in pull request #1387: SOLR-14210: Include replica health in healtcheck handler
HoustonPutman commented on a change in pull request #1387: SOLR-14210: Include
replica health in healtcheck handler
URL: https://github.com/apache/lucene-solr/pull/1387#discussion_r401737200
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/HealthCheckHandler.java
##
@@ -88,15 +96,46 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
return;
}
-// Set status to true if this node is in live_nodes
-if
(clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
- rsp.add(STATUS, OK);
-} else {
+// Fail if not in live_nodes
+if
(!clusterState.getLiveNodes().contains(cores.getZkController().getNodeName())) {
rsp.add(STATUS, FAILURE);
rsp.setException(new
SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Host Unavailable:
Not in live nodes as per zk"));
+ return;
}
-rsp.setHttpCaching(false);
+// Optionally require that all cores on this node are active if param
'failWhenRecovering=true'
Review comment:
`requireHealthyCores` not `failWhenRecovering`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401749799
##
File path: lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java
##
@@ -543,27 +543,36 @@ public synchronized boolean writeFieldUpdates(Directory
dir, FieldInfos.FieldNum
try {
// clone FieldInfos so that we can update their dvGen separately from
-// the reader's infos and write them to a new fieldInfos_gen file
-FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
-// cannot use builder.add(reader.getFieldInfos()) because it does not
-// clone FI.attributes as well FI.dvGen
+// the reader's infos and write them to a new fieldInfos_gen file.
+int maxFieldNumber = -1;
+Map newFields = new HashMap<>();
for (FieldInfo fi : reader.getFieldInfos()) {
- FieldInfo clone = builder.add(fi);
- // copy the stuff FieldInfos.Builder doesn't copy
- for (Entry e : fi.attributes().entrySet()) {
-clone.putAttribute(e.getKey(), e.getValue());
- }
- clone.setDocValuesGen(fi.getDocValuesGen());
+ // cannot use builder.add(fi) because it does not preserve
+ // the local field number. Field numbers can be different from the
global ones
+ // if the segment was created externally (with
IndexWriter#addIndexes(Directory)).
+ FieldInfo clone = cloneFieldInfo(fi, fi.number);
+ newFields.put(clone.name, clone);
+ maxFieldNumber = Math.max(clone.number, maxFieldNumber);
}
// create new fields with the right DV type
+FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
for (List updates : pendingDVUpdates.values()) {
DocValuesFieldUpdates update = updates.get(0);
- FieldInfo fieldInfo = builder.getOrAdd(update.field);
- fieldInfo.setDocValuesType(update.type);
+ FieldInfo fi = newFields.get(update.field);
+ if (fi == null) {
+// the field is not present in this segment so we can fallback to
the global fields.
+fi = builder.getOrAdd(update.field);
+if (fi.number <= maxFieldNumber) {
+ // the global field number is already used in this segment for a
different field so we force a new one locally.
+ fi = cloneFieldInfo(fi, ++maxFieldNumber);
Review comment:
> I also wonder if we should use FieldInfos.Builder since this logic you
implemented looks similar to FieldInfos.Builder#add.
I don't think it works since the logic there is to use the global field
number if the field already exists in another segment. In the logic above we
try to re-assign the global field number locally since it clashes with a local
one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field infos update on doc values update
jimczi commented on a change in pull request #1394: LUCENE-9300: Fix field
infos update on doc values update
URL: https://github.com/apache/lucene-solr/pull/1394#discussion_r401750154
##
File path: lucene/core/src/java/org/apache/lucene/index/ReadersAndUpdates.java
##
@@ -543,27 +543,36 @@ public synchronized boolean writeFieldUpdates(Directory
dir, FieldInfos.FieldNum
try {
// clone FieldInfos so that we can update their dvGen separately from
-// the reader's infos and write them to a new fieldInfos_gen file
-FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
-// cannot use builder.add(reader.getFieldInfos()) because it does not
-// clone FI.attributes as well FI.dvGen
+// the reader's infos and write them to a new fieldInfos_gen file.
+int maxFieldNumber = -1;
+Map newFields = new HashMap<>();
for (FieldInfo fi : reader.getFieldInfos()) {
- FieldInfo clone = builder.add(fi);
- // copy the stuff FieldInfos.Builder doesn't copy
- for (Entry e : fi.attributes().entrySet()) {
-clone.putAttribute(e.getKey(), e.getValue());
- }
- clone.setDocValuesGen(fi.getDocValuesGen());
+ // cannot use builder.add(fi) because it does not preserve
+ // the local field number. Field numbers can be different from the
global ones
+ // if the segment was created externally (with
IndexWriter#addIndexes(Directory)).
+ FieldInfo clone = cloneFieldInfo(fi, fi.number);
+ newFields.put(clone.name, clone);
+ maxFieldNumber = Math.max(clone.number, maxFieldNumber);
}
// create new fields with the right DV type
+FieldInfos.Builder builder = new FieldInfos.Builder(fieldNumbers);
for (List updates : pendingDVUpdates.values()) {
DocValuesFieldUpdates update = updates.get(0);
- FieldInfo fieldInfo = builder.getOrAdd(update.field);
- fieldInfo.setDocValuesType(update.type);
+ FieldInfo fi = newFields.get(update.field);
+ if (fi == null) {
+// the field is not present in this segment so we can fallback to
the global fields.
+fi = builder.getOrAdd(update.field);
+if (fi.number <= maxFieldNumber) {
+ // the global field number is already used in this segment for a
different field so we force a new one locally.
+ fi = cloneFieldInfo(fi, ++maxFieldNumber);
Review comment:
> Do we need to set the doc-value type before cloning to make sure the
doc-value type is set on the writer?
thanks, I pushed
https://github.com/apache/lucene-solr/pull/1394/commits/090ef78171dfe671b0fb1ab238951fdd617bd65c
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source
madrob commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source URL: https://github.com/apache/lucene-solr/pull/1390#issuecomment-607356036 Your patch moved the *nix invocation of WrapperDownloader to before we set JAVA_CMD, when it still needed to be after. I switched that back, but kept your change of setting the CLASSPATH after we download the new gradle-wrapper.jar. I'll push a squashed/rebased version of everything here for one last look before committing. In your Windows patch you use backslashes for GRADLE_WRAPPER_JAR but forward slashes for the JAVA_EXE command - is that fine/expected/necessary? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9575) Initialize an empty solr-home
[ https://issues.apache.org/jira/browse/SOLR-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072953#comment-17072953 ] David Smiley commented on SOLR-9575: If you look at the comments above; I've always been supportive of this and still am. You need to convince [~hossman] of your proposal. If he chooses not to respond then silent consensus. > Initialize an empty solr-home > - > > Key: SOLR-9575 > URL: https://issues.apache.org/jira/browse/SOLR-9575 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Major > Labels: docker > > The user may not want to use Solr's default solr-home dir location -- most > likely to use a separate disk. If you do this, there are two main problems: > * solr.xml & zoo.cfg aren't there > * configsets aren't there > Of course you could copy it manually but that's an extra step, and it's > particularly annoying to add this step to a Docker setup. Docker is all the > rage these days, and for good reason. If I mount a volume at > /opt/solr/server/solr then it basically masks this part of the built-in Solr > image (thus making configsets completely invisible) and points to some place > that will be empty. Solr obviously complains. I could set the solr-home to > some other path that I mount, but Solr would still complain about an empty > solr-home -- no solr.xml > If solr-home is empty, and if it's a dir other than the default solr-home, > then I think the solr-home should be initialized with solr.xml and zoo.cfg > copied from the default solr-home. I think configsets should be referenced > from the default solr-home if there is no configsets dir in solr-home. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9301) Gradle: Jar MANIFEST incomplete
[
https://issues.apache.org/jira/browse/LUCENE-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-9301:
--
Parent: LUCENE-9077
Issue Type: Sub-task (was: Bug)
> Gradle: Jar MANIFEST incomplete
> ---
>
> Key: LUCENE-9301
> URL: https://issues.apache.org/jira/browse/LUCENE-9301
> Project: Lucene - Core
> Issue Type: Sub-task
> Components: general/build
>Affects Versions: master (9.0)
>Reporter: Jan Høydahl
>Priority: Major
>
> After building with gradle, the MANIFEST.MF file for e.g. solr-core.jar
> containst
> {noformat}
> Manifest-Version: 1.0
> {noformat}
> While when building with ant, it says
> {noformat}
> Manifest-Version: 1.0
> Ant-Version: Apache Ant 1.10.7
> Created-By: 11.0.6+10 (AdoptOpenJDK)
> Extension-Name: org.apache.solr
> Specification-Title: Apache Solr Search Server: solr-core
> Specification-Version: 9.0.0
> Specification-Vendor: The Apache Software Foundation
> Implementation-Title: org.apache.solr
> Implementation-Version: 9.0.0-SNAPSHOT 9b5542ad55da601e0bdfda96bad8c2c
> cabbbc397 - janhoy - 2020-04-01 16:24:09
> Implementation-Vendor: The Apache Software Foundation
> X-Compile-Source-JDK: 11
> X-Compile-Target-JDK: 11
> {noformat}
> In addition, with ant, the META-INF folder also contains LICENSE.txt and
> NOTICE.txt files.
> There is a macro {{build-manifest}} in common-build.xml that seems to build
> the manifest.
> The effect of this is e.g. that spec an implementation versions do not show
> in Solr Admin UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9169) Gradle Wrapper JAR files checksum should be validated
[ https://issues.apache.org/jira/browse/LUCENE-9169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-9169. --- Fix Version/s: master (9.0) Assignee: Dawid Weiss Resolution: Fixed This was committed in 24f7a28ac14 > Gradle Wrapper JAR files checksum should be validated > - > > Key: LUCENE-9169 > URL: https://issues.apache.org/jira/browse/LUCENE-9169 > Project: Lucene - Core > Issue Type: Task >Reporter: Paul Merlin >Assignee: Dawid Weiss >Priority: Trivial > Fix For: master (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > > See [https://github.com/marketplace/actions/gradle-wrapper-validation] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9266) ant nightly-smoke fails due to presence of build.gradle
[ https://issues.apache.org/jira/browse/LUCENE-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob reassigned LUCENE-9266: - Assignee: Mike Drob > ant nightly-smoke fails due to presence of build.gradle > --- > > Key: LUCENE-9266 > URL: https://issues.apache.org/jira/browse/LUCENE-9266 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Seen on Jenkins - > [https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1617/console] > > Reproduced locally. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9575) Initialize an empty solr-home
[ https://issues.apache.org/jira/browse/SOLR-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072973#comment-17072973 ] Chris M. Hostetter commented on SOLR-9575: -- i still feel like this is an anti-pattern that will almost certainly lead to confusion for folks who makes mistakes, but if the consensus is that it's better to "make the default easier" then to be able to "fail fast on user error" then i'm willing to stay out of the way. Consider my previous -1 now a -0 > Initialize an empty solr-home > - > > Key: SOLR-9575 > URL: https://issues.apache.org/jira/browse/SOLR-9575 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Major > Labels: docker > > The user may not want to use Solr's default solr-home dir location -- most > likely to use a separate disk. If you do this, there are two main problems: > * solr.xml & zoo.cfg aren't there > * configsets aren't there > Of course you could copy it manually but that's an extra step, and it's > particularly annoying to add this step to a Docker setup. Docker is all the > rage these days, and for good reason. If I mount a volume at > /opt/solr/server/solr then it basically masks this part of the built-in Solr > image (thus making configsets completely invisible) and points to some place > that will be empty. Solr obviously complains. I could set the solr-home to > some other path that I mount, but Solr would still complain about an empty > solr-home -- no solr.xml > If solr-home is empty, and if it's a dir other than the default solr-home, > then I think the solr-home should be initialized with solr.xml and zoo.cfg > copied from the default solr-home. I think configsets should be referenced > from the default solr-home if there is no configsets dir in solr-home. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] HoustonPutman commented on a change in pull request #1392: SOLR-14371 Zk StatusHandler should know about dynamic zk config
HoustonPutman commented on a change in pull request #1392: SOLR-14371 Zk
StatusHandler should know about dynamic zk config
URL: https://github.com/apache/lucene-solr/pull/1392#discussion_r401790833
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/ZookeeperStatusHandler.java
##
@@ -109,25 +133,30 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
numOk++;
}
String state = String.valueOf(stat.get("zk_server_state"));
-if ("follower".equals(state)) {
+if ("follower".equals(state) || "observer".equals(state)) {
followers++;
} else if ("leader".equals(state)) {
leaders++;
reportedFollowers =
Integer.parseInt(String.valueOf(stat.get("zk_followers")));
} else if ("standalone".equals(state)) {
standalone++;
}
+if (zk.role != null) {
+ stat.put("role", zk.role);
+ dynamicReconfig = true;
Review comment:
I think this should live up here where the reconfig check is done.
https://github.com/apache/lucene-solr/pull/1392/files#diff-2f0decb1314fc7f7d3e6e0f497839745R109
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13687) Enable the bin/solr script to accept a solr url to run commands
[
https://issues.apache.org/jira/browse/SOLR-13687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073039#comment-17073039
]
Marcus Eagan commented on SOLR-13687:
-
I think this is a good idea for usability [~noble.paul]
> Enable the bin/solr script to accept a solr url to run commands
>
>
> Key: SOLR-13687
> URL: https://issues.apache.org/jira/browse/SOLR-13687
> Project: Solr
> Issue Type: Bug
>Reporter: Noble Paul
>Priority: Major
>
> The problem we have today with our {{bin/solr}} script is that we have to run
> it from one of the nodes where Solr is running. This is a security issue b/c
> only admins are usaully be allowed to login to a machine where solr is
> running.If you have multiple cluster running in that host we don't know which
> one it's going to use. It is much easier to write a simple script that works
> over a url and the user has no ambiguity as to how it works. You can just
> unpack a solr distribution to your local machine and start using the script
> without bothering to install solr .
> The following commands can easily be executed remotely. These commands can
> accept the base url of any solr node in the cluster and perform the opertaion
> * healthcheck
> * create
> * create_core
> * create_collection
> * delete, version,
> * config
> * autoscaling
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14307) "user caches" don't support "enabled" attribute
[
https://issues.apache.org/jira/browse/SOLR-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073069#comment-17073069
]
ASF subversion and git services commented on SOLR-14307:
Commit f779bc632d107bb8cf28ffa2b2ada4cb58203e8a in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f779bc6 ]
SOLR-14307: User defined "" entries in solrconfig.xml now support
enabled="true|false" just like core searcher caches.
> "user caches" don't support "enabled" attribute
> ---
>
> Key: SOLR-14307
> URL: https://issues.apache.org/jira/browse/SOLR-14307
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-14307.patch
>
>
> while trying to help write some test cases for SOLR-13807 i discovered that
> the code path used for building the {{List}} of _user_ caches
> (ie: {{}} doesn't respect the idea of an "enabled"
> attribute ... that is only checked for in the code path use for building
> singular CacheConfig options from explicit xpaths (ie: {{ />}} etc...)
> We should fix this, if for no other reason then so it's easy for tests to use
> system properties to enable/disable all caches.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source
dweiss commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source URL: https://github.com/apache/lucene-solr/pull/1390#issuecomment-607423166 bq. Your patch moved the *nix invocation of WrapperDownloader to before we set JAVA_CMD, when it still needed to be after. I switched that back, but kept your change of setting the CLASSPATH after we download the new gradle-wrapper.jar I should have checked, sorry. I had a million things to do today. bq. In your Windows patch you use backslashes for GRADLE_WRAPPER_JAR but forward slashes for the JAVA_EXE command - is that fine/expected/necessary? Neither matters on modern Java versions. You can switch them to windows path separators if you wish but you don't have to. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1390: LUCENE-9266 remove gradle wrapper jar from source
dweiss commented on a change in pull request #1390: LUCENE-9266 remove gradle
wrapper jar from source
URL: https://github.com/apache/lucene-solr/pull/1390#discussion_r401829201
##
File path: solr/build.xml
##
@@ -459,7 +459,7 @@
fullpath="${fullnamever}/solr/LUCENE_CHANGES.txt" />
Review comment:
Indent different for some reason?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13004) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/SOLR-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya reassigned SOLR-13004:
---
Assignee: Ishan Chattopadhyaya
> Integer overflow in total count in grouping results
> ---
>
> Key: SOLR-13004
> URL: https://issues.apache.org/jira/browse/SOLR-13004
> Project: Solr
> Issue Type: Bug
> Components: search, SolrCloud
>Affects Versions: 5.5.3
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #1392: SOLR-14371 Zk StatusHandler should know about dynamic zk config
janhoy commented on a change in pull request #1392: SOLR-14371 Zk StatusHandler
should know about dynamic zk config
URL: https://github.com/apache/lucene-solr/pull/1392#discussion_r401835111
##
File path:
solr/core/src/java/org/apache/solr/handler/admin/ZookeeperStatusHandler.java
##
@@ -108,27 +128,30 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
if ("true".equals(String.valueOf(stat.get("ok" {
numOk++;
}
-String state = String.valueOf(stat.get("zk_server_state"));
+String state = zk.role != null ? zk.role :
String.valueOf(stat.get("zk_server_state"));
if ("follower".equals(state)) {
followers++;
} else if ("leader".equals(state)) {
leaders++;
reportedFollowers =
Integer.parseInt(String.valueOf(stat.get("zk_followers")));
} else if ("standalone".equals(state)) {
standalone++;
+} else if ("participant".equals(state)) {
+ // NOCOMMIT: What does participant mean vs leader or follower?
Review comment:
I confused server state with role. Seems like even in dynamic reconfig mode,
the servers report state leader/follower, and can also have a role participant
or observer.
The only thing I'm not 100% sure about is whether `zk_server_state` can have
the value `observer` when not in dynamic reconfig mode, cause observer feature
pre-dates dynamic reconfig I think. I put it in there as an OR check and count
it as follower for now, but it might never get that value.
I do not generate any WARN/ERROR messages in UI based on number of observers
vs participants, as it is allowed to scale down to 1 participant if you like.
Not sure what happens if you have 2 participants, is that still "A Bad Thing™"?
Today we warn if there is an even number of leader/follower, but our code does
not yet know about observers. Perhaps we should count participants and WARN if
they are even in number, or will ZK handle it gracefully so we don't need to
warn?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13004) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/SOLR-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073090#comment-17073090
]
Ishan Chattopadhyaya commented on SOLR-13004:
-
This is something I want to fix. When matches (as collected from various
shards) exceeds integer limit, it overflows.
At the heart of the implementation is a Lucene class for TopGroups object,
which has totalHitCount as an integer. The merge method in this class adds up
the counts from various shards into a total which overflows.
> Integer overflow in total count in grouping results
> ---
>
> Key: SOLR-13004
> URL: https://issues.apache.org/jira/browse/SOLR-13004
> Project: Solr
> Issue Type: Bug
> Components: search, SolrCloud
>Affects Versions: 5.5.3
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13004) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/SOLR-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-13004:
Attachment: SOLR-13004.patch
Status: Open (was: Open)
Here's a WIP patch.
Given the change to TopGroups, should this be a LUCENE issue now? [~jpountz],
[~dsmiley] any thoughts, please?
> Integer overflow in total count in grouping results
> ---
>
> Key: SOLR-13004
> URL: https://issues.apache.org/jira/browse/SOLR-13004
> Project: Solr
> Issue Type: Bug
> Components: search, SolrCloud
>Affects Versions: 5.5.3
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13004) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/SOLR-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073090#comment-17073090
]
Ishan Chattopadhyaya edited comment on SOLR-13004 at 4/1/20, 6:54 PM:
--
This is something I want to fix. When matches (as collected from various
shards) exceeds integer limit, it overflows.
At the heart of the implementation is a Lucene class for TopGroups object,
which has totalHitCount as an integer. The merge method in this class adds up
the counts from various shards into a total which overflows.
Also, totalGroupsCount can also be a number more than integer range and should
be a long.
was (Author: ichattopadhyaya):
This is something I want to fix. When matches (as collected from various
shards) exceeds integer limit, it overflows.
At the heart of the implementation is a Lucene class for TopGroups object,
which has totalHitCount as an integer. The merge method in this class adds up
the counts from various shards into a total which overflows.
> Integer overflow in total count in grouping results
> ---
>
> Key: SOLR-13004
> URL: https://issues.apache.org/jira/browse/SOLR-13004
> Project: Solr
> Issue Type: Bug
> Components: search, SolrCloud
>Affects Versions: 5.5.3
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13004) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/SOLR-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-13004:
Attachment: SOLR-13004.patch
> Integer overflow in total count in grouping results
> ---
>
> Key: SOLR-13004
> URL: https://issues.apache.org/jira/browse/SOLR-13004
> Project: Solr
> Issue Type: Bug
> Components: search, SolrCloud
>Affects Versions: 5.5.3
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source
madrob commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source URL: https://github.com/apache/lucene-solr/pull/1390#issuecomment-607435284 From @gerlowskija on Slack: >Initially it failed for me with: `./gradlew --info tasks Error: Could not find or load main class buildSrc.src.main.java.org.apache.lucene.gradle.WrapperDownloader.java` But that turned out to be because I was using Java8. Might be worth doing some sort of java -version check somewhere, but maybe not worth the trouble. >Sure. But while people are developing on a mix of master/8x, there’s gonna be Java version mixups pretty frequently, so people are gonna start seeing that opaque error message unless we give them a better one. A better error message would be nice here, but I also don't want to figure out how to parse the output. Maybe the easiest solution is to add another class with an empty main method and if that fails we assume the java launcher is too old? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source
dweiss commented on issue #1390: LUCENE-9266 remove gradle wrapper jar from source URL: https://github.com/apache/lucene-solr/pull/1390#issuecomment-607439907 I would really want to limit the number of forked java processes to the absolute minimum in those scripts. Each invocation of java adds a bit of penalty to startup and these accumulate. I would assume people are running with Java 11+ and if it fails for them so be it... If you really want to be nice here then I would rewrite the Downloader to be compatible with Java 8, then *compile* that class with javac and run it as regular class. The first thing you do in the downloader then is you can System.exit(-1) if java version < 11. It can be also left for a separate patch/ improvement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14307) "user caches" don't support "enabled" attribute
[
https://issues.apache.org/jira/browse/SOLR-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073119#comment-17073119
]
ASF subversion and git services commented on SOLR-14307:
Commit 1cf156d7faf2ad545ae5f68a9888ce498d98c44d in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=1cf156d ]
SOLR-14307: User defined "" entries in solrconfig.xml now support
enabled="true|false" just like core searcher caches.
(cherry picked from commit f779bc632d107bb8cf28ffa2b2ada4cb58203e8a)
> "user caches" don't support "enabled" attribute
> ---
>
> Key: SOLR-14307
> URL: https://issues.apache.org/jira/browse/SOLR-14307
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-14307.patch
>
>
> while trying to help write some test cases for SOLR-13807 i discovered that
> the code path used for building the {{List}} of _user_ caches
> (ie: {{}} doesn't respect the idea of an "enabled"
> attribute ... that is only checked for in the code path use for building
> singular CacheConfig options from explicit xpaths (ie: {{ />}} etc...)
> We should fix this, if for no other reason then so it's easy for tests to use
> system properties to enable/disable all caches.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13004) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/SOLR-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073137#comment-17073137
]
David Smiley commented on SOLR-13004:
-
I suggest moving to a Lucene issue since the core of the change is in Lucene.
Its debatable if the Lucene side it s a bug or improvement; I tend to think the
latter. But I can see from a Solr user perspective, this shows as a bug. Not
a big deal any way.
> Integer overflow in total count in grouping results
> ---
>
> Key: SOLR-13004
> URL: https://issues.apache.org/jira/browse/SOLR-13004
> Project: Solr
> Issue Type: Bug
> Components: search, SolrCloud
>Affects Versions: 5.5.3
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Moved] (LUCENE-9302) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/LUCENE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya moved SOLR-13004 to LUCENE-9302:
-
Component/s: (was: SolrCloud)
(was: search)
Key: LUCENE-9302 (was: SOLR-13004)
Lucene Fields: New,Patch Available
Affects Version/s: (was: 5.5.3)
Issue Type: Improvement (was: Bug)
Project: Lucene - Core (was: Solr)
> Integer overflow in total count in grouping results
> ---
>
> Key: LUCENE-9302
> URL: https://issues.apache.org/jira/browse/LUCENE-9302
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14014) Allow Solr to start with Admin UI disabled
[
https://issues.apache.org/jira/browse/SOLR-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073144#comment-17073144
]
David Smiley commented on SOLR-14014:
-
If we only show the admin UI on one node (and jump through hoops to make this
works like Jan's proxy) then doesn't this subvert the very point of disabling
the UI at all? There's still XSS vulnerability potential.
FWIW I'm not really a fan of disabling the admin UI but whatever. I'm looking
back briefly at my earlier comments and I referred to issues that are now
closed. So is the XSS threat still real?
> Allow Solr to start with Admin UI disabled
> --
>
> Key: SOLR-14014
> URL: https://issues.apache.org/jira/browse/SOLR-14014
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI, security
>Affects Versions: master (9.0), 8.3.1
>Reporter: Jason Gerlowski
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently Solr always runs the Admin UI. With the history of XSS issues and
> other security concerns that have been found in the Admin UI, Solr should
> offer a mode where the Admin UI is disabled. Maybe, and this is a topic
> that'll need some serious discussion, this should even be the default when
> Solr starts.
> NOTE: Disabling the Admin UI removes XSS and other attack vectors. But even
> with the Admin UI disabled, Solr will still be inherently unsafe without
> firewall protection on a public network.
> *Proposed design:*
> A java system property called *headless* will be used as an internal flag for
> starting Solr in headless mode. This property will default to true. A java
> property can be used at startup to set this flag to false.
> Here is an example:
> {code:java}
> bin/solr start -Dheadless=false {code}
> A message will be added following startup describing the mode.
> In headless mode the following message will be displayed:
> "solr is running in headless mode. The admin console is unavailable. To to
> turn off headless mode and allow the admin console use the following
> parameter startup parameter:
> -Dheadless=false
>
> In non-headless mode the following message will be displayed:
> "solr is running with headless mode turned off. The admin console is
> available in this mode. Disabling the Admin UI removes XSS and other attack
> vectors"
> If a user attempts to access the admin console while Solr is in headless mode
> it Solr will return 401 unauthorized.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on issue #1368: SOLR-14351: Fix/improve MDCLoggingContext usage
dsmiley commented on issue #1368: SOLR-14351: Fix/improve MDCLoggingContext usage URL: https://github.com/apache/lucene-solr/pull/1368#issuecomment-607459278 There's some stupid precommit import or something I'll fix but otherwise I think this is ready. I slept on this one a bit and reviewed some pieces of code that gave me doubts, and I'm comfortable with my latest commits on the PR. I'll commit Friday. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9302) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/LUCENE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073151#comment-17073151
]
Ishan Chattopadhyaya commented on LUCENE-9302:
--
Thanks David, moved to a Lucene issue now. Would like some early feedback on
the patch here. Mainly curious whether there's some reason why the
totalHitCounts is an integer, not a long? Given that the merge() method's
javadocs say the following, I feel the total hits across all shards can
legitimately overflow the integer range (this is happening in our production
cluster).
{code:java}
/** Merges an array of TopGroups, for example obtained
* from the second-pass collector across multiple
* shards. Each TopGroups must have been sorted by the
* same groupSort and docSort, and the top groups passed
* to all second-pass collectors must be the same.
*
* NOTE: We can't always compute an exact totalGroupCount.
* Documents belonging to a group may occur on more than
* one shard and thus the merged totalGroupCount can be
* higher than the actual totalGroupCount. In this case the
* totalGroupCount represents a upper bound. If the documents
* of one group do only reside in one shard then the
* totalGroupCount is exact.
*
* NOTE: the topDocs in each GroupDocs is actually
* an instance of TopDocsAndShards
*/
public static TopGroups merge(TopGroups[] shardGroups, Sort
groupSort, Sort docSort, int docOffset, int docTopN, ScoreMergeMode
scoreMergeMode) {
{code}
> Integer overflow in total count in grouping results
> ---
>
> Key: LUCENE-9302
> URL: https://issues.apache.org/jira/browse/LUCENE-9302
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14365) CollapsingQParser - Avoiding always allocate int[] and float[] with size equals to number of unique values
[
https://issues.apache.org/jira/browse/SOLR-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073153#comment-17073153
]
David Smiley commented on SOLR-14365:
-
Can you please do a PR instead as it's more conducive to a code review? If it
were a trivial patch, I wouldn't have suggested it.
> CollapsingQParser - Avoiding always allocate int[] and float[] with size
> equals to number of unique values
> --
>
> Key: SOLR-14365
> URL: https://issues.apache.org/jira/browse/SOLR-14365
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.4.1
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-14365.patch
>
>
> Since Collapsing is a PostFilter, documents reach Collapsing must match with
> all filters and queries, so the number of documents Collapsing need to
> collect/compute score is a small fraction of the total number documents in
> the index. So why do we need to always consume the memory (for int[] and
> float[] array) for all unique values of the collapsed field? If the number of
> unique values of the collapsed field found in the documents that match
> queries and filters is 300 then we only need int[] and float[] array with
> size of 300 and not 1.2 million in size. However, we don't know which value
> of the collapsed field will show up in the results so we cannot use a smaller
> array.
> The easy fix for this problem is using as much as we need by using IntIntMap
> and IntFloatMap that hold primitives and are much more space efficient than
> the Java HashMap. These maps can be slower (10x or 20x) than plain int[] and
> float[] if matched documents is large (almost all documents matched queries
> and other filters). But our belief is that does not happen that frequently
> (how frequently do we run collapsing on the entire index?).
> For this issue I propose adding 2 methods for collapsing which is
> * array : which is current implementation
> * hash : which is new approach and will be default method
> later we can add another method {{smart}} which is automatically pick method
> based on comparision between {{number of docs matched queries and filters}}
> and {{number of unique values of the field}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9302) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/LUCENE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073156#comment-17073156
]
Adrien Grand commented on LUCENE-9302:
--
The in-progress patch looks good. I'm not familiar with how things work in Solr
when the client and server are on different versions or when not all nodes of
the cluster are on the same version, does the code need to be able to handle
the case when the hit count is still returned as an integer by another node?
> Integer overflow in total count in grouping results
> ---
>
> Key: LUCENE-9302
> URL: https://issues.apache.org/jira/browse/LUCENE-9302
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9302) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/LUCENE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073162#comment-17073162
]
Ishan Chattopadhyaya commented on LUCENE-9302:
--
{quote}The in-progress patch looks good. I'm not familiar with how things work
in Solr when the client and server are on different versions or when not all
nodes of the cluster are on the same version, does the code need to be able to
handle the case when the hit count is still returned as an integer by another
node?
{quote}
Thanks for your review, Adrien. Right, need to add backcompat handling
(wondering how ugly that could be; I hope not). Also, handle the
totalGroupCount to be a long.
Shall update the patch soon.
> Integer overflow in total count in grouping results
> ---
>
> Key: LUCENE-9302
> URL: https://issues.apache.org/jira/browse/LUCENE-9302
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9302) Integer overflow in total count in grouping results
[
https://issues.apache.org/jira/browse/LUCENE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073172#comment-17073172
]
Adrien Grand commented on LUCENE-9302:
--
Hopefully replacing "(Long) foo" with "((Number) foo).longValue()" would work?
> Integer overflow in total count in grouping results
> ---
>
> Key: LUCENE-9302
> URL: https://issues.apache.org/jira/browse/LUCENE-9302
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ian
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
> Attachments: SOLR-13004.patch, SOLR-13004.patch
>
>
> When doing a Grouping search in solr cloud you can get a negative number for
> the total found.
> This is caused by the accumulated total being held in an integer and not a
> long.
>
> example result:
> {{{ "responseHeader": { "status": 0, "QTime": 9231, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "group.field":
> "decade", "group": "true", "_": "1542773674247" } }, "grouped": { "decade": {
> "matches": -629516788, "groups": [ { "groupValue": "200", "doclist": {
> "numFound": -629516788, "start": 0, "maxScore": 1.9315376, "docs": [ {
> "decade": "200" } ] } } ] } } }}}
>
> {{result without grouping:}}
> {{{ "responseHeader": { "status": 0, "QTime": 1063, "params": { "q":
> "decade:200", "indent": "true", "fl": "decade", "wt": "json", "_":
> "1542773791855" } }, "response": { "numFound": 3665450508, "start": 0,
> "maxScore": 1.9315376, "docs": [ { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" }, { "decade": "200" }, { "decade": "200" }, { "decade":
> "200" }, { "decade": "200" }, { "decade": "200" }, { "decade": "200" }, {
> "decade": "200" } ] } }}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9575) Initialize an empty solr-home
[ https://issues.apache.org/jira/browse/SOLR-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073205#comment-17073205 ] Jan Høydahl commented on SOLR-9575: --- What does solr do today if solr.xml is in zookeeper? Will it accept to start with any SOLR_HOME as long as the folder exists? > Initialize an empty solr-home > - > > Key: SOLR-9575 > URL: https://issues.apache.org/jira/browse/SOLR-9575 > Project: Solr > Issue Type: Improvement >Reporter: David Smiley >Priority: Major > Labels: docker > > The user may not want to use Solr's default solr-home dir location -- most > likely to use a separate disk. If you do this, there are two main problems: > * solr.xml & zoo.cfg aren't there > * configsets aren't there > Of course you could copy it manually but that's an extra step, and it's > particularly annoying to add this step to a Docker setup. Docker is all the > rage these days, and for good reason. If I mount a volume at > /opt/solr/server/solr then it basically masks this part of the built-in Solr > image (thus making configsets completely invisible) and points to some place > that will be empty. Solr obviously complains. I could set the solr-home to > some other path that I mount, but Solr would still complain about an empty > solr-home -- no solr.xml > If solr-home is empty, and if it's a dir other than the default solr-home, > then I think the solr-home should be initialized with solr.xml and zoo.cfg > copied from the default solr-home. I think configsets should be referenced > from the default solr-home if there is no configsets dir in solr-home. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14371) Zk StatusHandler should know about dynamic zk config
[
https://issues.apache.org/jira/browse/SOLR-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073223#comment-17073223
]
Jan Høydahl commented on SOLR-14371:
The PR is now ready for broader review, for those interested.
Since embedded ZK and ZK in tests are hardcoded to a standalone variant, and
big code changes are needed to spin up a Quorum, there are no unit tests for
the new {{getConfig()}} method in {{SolrZkClient}}. I have tested manually with
an external zk cluster that it works though.
Here is a condensed summary of the current PR
* If zk quorum has dynamic reconfiguration, the ZkStatus handler and Admin UI
view will now display info for each of the currently active Zookeepers, even if
they differ from initial ZK_HOST
* That meanns you can configure ZK_HOST with a single LB address in front of
all ZKs (like Solr Operator for k8s) and it will "just work"
* New info in JSON response and UI: {{dynamicReconfig=true/false}}
* New info for each ZK host: {{role=participant/follower}}
> Zk StatusHandler should know about dynamic zk config
>
>
> Key: SOLR-14371
> URL: https://issues.apache.org/jira/browse/SOLR-14371
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With zk 3.5 it supports dynamic reconfig, which is used by the solr-operator
> for Kubernetes. Then Solr is given a zkHost of one url pointing to a LB
> (Service) in front of all zookeepers, and the zkclient will then fetch list
> of all zookeepers from special zknode /zookeeper/config and reconfigure
> itself with connection to all zk nodes listed. So you can then scale up/down
> number of zk nodes dynamically without restarting solr.
> However, the Admin UI displays errors since it believes it is connected to
> only one zk, which is contradictory to what zk itself reports. We need to
> make ZookeeperStatusHandler aware of dynamic reconfig so it asks zkclient
> what current zkHost is instead of relying on Zk_HOST static setting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on issue #1387: SOLR-14210: Include replica health in healtcheck handler
janhoy commented on issue #1387: SOLR-14210: Include replica health in healtcheck handler URL: https://github.com/apache/lucene-solr/pull/1387#issuecomment-607526425 > Are you targeting this for 8.6 or 9.0 I see no reason for not targeting 8.6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14210) Introduce Node-level status handler for replicas
[
https://issues.apache.org/jira/browse/SOLR-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-14210:
---
Affects Version/s: (was: master (9.0))
> Introduce Node-level status handler for replicas
>
>
> Key: SOLR-14210
> URL: https://issues.apache.org/jira/browse/SOLR-14210
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5
>Reporter: Houston Putman
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> h2. Background
> As was brought up in SOLR-13055, in order to run Solr in a more cloud-native
> way, we need some additional features around node-level healthchecks.
> {quote}Like in Kubernetes we need 'liveliness' and 'readiness' probe
> explained in
> [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/n]
> determine if a node is live and ready to serve live traffic.
> {quote}
>
> However there are issues around kubernetes managing it's own rolling
> restarts. With the current healthcheck setup, it's easy to envision a
> scenario in which Solr reports itself as "healthy" when all of its replicas
> are actually recovering. Therefore kubernetes, seeing a healthy pod would
> then go and restart the next Solr node. This can happen until all replicas
> are "recovering" and none are healthy. (maybe the last one restarted will be
> "down", but still there are no "active" replicas)
> h2. Proposal
> I propose we make an additional healthcheck handler that returns whether all
> replicas hosted by that Solr node are healthy and "active". That way we will
> be able to use the [default kubernetes rolling restart
> logic|https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies]
> with Solr.
> To add on to [Jan's point
> here|https://issues.apache.org/jira/browse/SOLR-13055?focusedCommentId=16716559&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16716559],
> this handler should be more friendly for other Content-Types and should use
> bettter HTTP response statuses.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14210) Introduce Node-level status handler for replicas
[
https://issues.apache.org/jira/browse/SOLR-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073235#comment-17073235
]
Jan Høydahl commented on SOLR-14210:
This is ready for broader review.
So if you just call {{/api/node/health}} there is no change. But if you call
{{/api/node/health?requireHealthyCores=true}}, then the new checks will kick in:
* Return 200 OK only if all replicas from clusterstate on this node are ACTIVE
(or RECOVERY_FAILED), and that the core actually exists in CoreContainer
* If OK, an extra {{"message": "All cores are healthy"}} is added to the
response JSON
* If one of the replicas for an *active* shard on the node is DOWN or
RECOVERING, then 503 is returned with error text {{"error": "Replica(s) [foo,
bar] are currently initializing or recovering"}}. We do not care about inactive
shards.
The extra checks will add some extra CPU load for looping through clusterstate
objects, and this is not benchmarked. Only for very large clusters with
thousands of shards would that be a theoretical issue, and only if health
endpoint is hit very frequently. Reviewers are encouraged to [give feedback on
the findUnhealthyCores()
method|https://github.com/apache/lucene-solr/pull/1387/files#diff-9f25e225f70fa66d2b27079a9511c0daR124-R138].
I'm targeting 8.6 with this
> Introduce Node-level status handler for replicas
>
>
> Key: SOLR-14210
> URL: https://issues.apache.org/jira/browse/SOLR-14210
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.5
>Reporter: Houston Putman
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> h2. Background
> As was brought up in SOLR-13055, in order to run Solr in a more cloud-native
> way, we need some additional features around node-level healthchecks.
> {quote}Like in Kubernetes we need 'liveliness' and 'readiness' probe
> explained in
> [https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/n]
> determine if a node is live and ready to serve live traffic.
> {quote}
>
> However there are issues around kubernetes managing it's own rolling
> restarts. With the current healthcheck setup, it's easy to envision a
> scenario in which Solr reports itself as "healthy" when all of its replicas
> are actually recovering. Therefore kubernetes, seeing a healthy pod would
> then go and restart the next Solr node. This can happen until all replicas
> are "recovering" and none are healthy. (maybe the last one restarted will be
> "down", but still there are no "active" replicas)
> h2. Proposal
> I propose we make an additional healthcheck handler that returns whether all
> replicas hosted by that Solr node are healthy and "active". That way we will
> be able to use the [default kubernetes rolling restart
> logic|https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies]
> with Solr.
> To add on to [Jan's point
> here|https://issues.apache.org/jira/browse/SOLR-13055?focusedCommentId=16716559&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16716559],
> this handler should be more friendly for other Content-Types and should use
> bettter HTTP response statuses.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14371) Zk StatusHandler should know about dynamic zk config
[ https://issues.apache.org/jira/browse/SOLR-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Høydahl updated SOLR-14371: --- Attachment: dynamic-reconfig.png > Zk StatusHandler should know about dynamic zk config > > > Key: SOLR-14371 > URL: https://issues.apache.org/jira/browse/SOLR-14371 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Attachments: dynamic-reconfig.png > > Time Spent: 50m > Remaining Estimate: 0h > > With zk 3.5 it supports dynamic reconfig, which is used by the solr-operator > for Kubernetes. Then Solr is given a zkHost of one url pointing to a LB > (Service) in front of all zookeepers, and the zkclient will then fetch list > of all zookeepers from special zknode /zookeeper/config and reconfigure > itself with connection to all zk nodes listed. So you can then scale up/down > number of zk nodes dynamically without restarting solr. > However, the Admin UI displays errors since it believes it is connected to > only one zk, which is contradictory to what zk itself reports. We need to > make ZookeeperStatusHandler aware of dynamic reconfig so it asks zkclient > what current zkHost is instead of relying on Zk_HOST static setting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14371) Zk StatusHandler should know about dynamic zk config
[
https://issues.apache.org/jira/browse/SOLR-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073223#comment-17073223
]
Jan Høydahl edited comment on SOLR-14371 at 4/1/20, 11:07 PM:
--
The PR is now ready for broader review, for those interested.
Since embedded ZK and ZK in tests are hardcoded to a standalone variant, and
big code changes are needed to spin up a Quorum, there are no unit tests for
the new {{getConfig()}} method in {{SolrZkClient}}. I have tested manually with
an external zk cluster that it works though.
Here is a condensed summary of the current PR
* If zk quorum has dynamic reconfiguration, the ZkStatus handler and Admin UI
view will now display info for each of the currently active Zookeepers, even if
they differ from initial ZK_HOST
* That meanns you can configure ZK_HOST with a single LB address in front of
all ZKs (like Solr Operator for k8s) and it will "just work"
* New info in JSON response and UI: {{dynamicReconfig=true/false}}
* New info for each ZK host: {{role=participant/follower}}
!dynamic-reconfig.png|width=600!
was (Author: janhoy):
The PR is now ready for broader review, for those interested.
Since embedded ZK and ZK in tests are hardcoded to a standalone variant, and
big code changes are needed to spin up a Quorum, there are no unit tests for
the new {{getConfig()}} method in {{SolrZkClient}}. I have tested manually with
an external zk cluster that it works though.
Here is a condensed summary of the current PR
* If zk quorum has dynamic reconfiguration, the ZkStatus handler and Admin UI
view will now display info for each of the currently active Zookeepers, even if
they differ from initial ZK_HOST
* That meanns you can configure ZK_HOST with a single LB address in front of
all ZKs (like Solr Operator for k8s) and it will "just work"
* New info in JSON response and UI: {{dynamicReconfig=true/false}}
* New info for each ZK host: {{role=participant/follower}}
> Zk StatusHandler should know about dynamic zk config
>
>
> Key: SOLR-14371
> URL: https://issues.apache.org/jira/browse/SOLR-14371
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Attachments: dynamic-reconfig.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With zk 3.5 it supports dynamic reconfig, which is used by the solr-operator
> for Kubernetes. Then Solr is given a zkHost of one url pointing to a LB
> (Service) in front of all zookeepers, and the zkclient will then fetch list
> of all zookeepers from special zknode /zookeeper/config and reconfigure
> itself with connection to all zk nodes listed. So you can then scale up/down
> number of zk nodes dynamically without restarting solr.
> However, the Admin UI displays errors since it believes it is connected to
> only one zk, which is contradictory to what zk itself reports. We need to
> make ZookeeperStatusHandler aware of dynamic reconfig so it asks zkclient
> what current zkHost is instead of relying on Zk_HOST static setting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14371) Zk StatusHandler should know about dynamic zk config
[
https://issues.apache.org/jira/browse/SOLR-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073223#comment-17073223
]
Jan Høydahl edited comment on SOLR-14371 at 4/1/20, 11:08 PM:
--
The PR is now ready for broader review, for those interested.
Since embedded ZK and ZK in tests are hardcoded to a standalone variant, and
big code changes are needed to spin up a Quorum, there are no unit tests for
the new {{getConfig()}} method in {{SolrZkClient}}. I have tested manually with
an external zk cluster that it works though.
Here is a condensed summary of the current PR
* If zk quorum has dynamic reconfiguration, the ZkStatus handler and Admin UI
view will now display info for each of the currently active Zookeepers, even if
they differ from initial ZK_HOST
* That meanns you can configure ZK_HOST with a single LB address in front of
all ZKs (like Solr Operator for k8s) and it will "just work"
* New info in JSON response and UI: {{dynamicReconfig=true/false}}
* New info for each ZK host: {{role=participant/observer}}
!dynamic-reconfig.png|width=600!
was (Author: janhoy):
The PR is now ready for broader review, for those interested.
Since embedded ZK and ZK in tests are hardcoded to a standalone variant, and
big code changes are needed to spin up a Quorum, there are no unit tests for
the new {{getConfig()}} method in {{SolrZkClient}}. I have tested manually with
an external zk cluster that it works though.
Here is a condensed summary of the current PR
* If zk quorum has dynamic reconfiguration, the ZkStatus handler and Admin UI
view will now display info for each of the currently active Zookeepers, even if
they differ from initial ZK_HOST
* That meanns you can configure ZK_HOST with a single LB address in front of
all ZKs (like Solr Operator for k8s) and it will "just work"
* New info in JSON response and UI: {{dynamicReconfig=true/false}}
* New info for each ZK host: {{role=participant/follower}}
!dynamic-reconfig.png|width=600!
> Zk StatusHandler should know about dynamic zk config
>
>
> Key: SOLR-14371
> URL: https://issues.apache.org/jira/browse/SOLR-14371
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Attachments: dynamic-reconfig.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With zk 3.5 it supports dynamic reconfig, which is used by the solr-operator
> for Kubernetes. Then Solr is given a zkHost of one url pointing to a LB
> (Service) in front of all zookeepers, and the zkclient will then fetch list
> of all zookeepers from special zknode /zookeeper/config and reconfigure
> itself with connection to all zk nodes listed. So you can then scale up/down
> number of zk nodes dynamically without restarting solr.
> However, the Admin UI displays errors since it believes it is connected to
> only one zk, which is contradictory to what zk itself reports. We need to
> make ZookeeperStatusHandler aware of dynamic reconfig so it asks zkclient
> what current zkHost is instead of relying on Zk_HOST static setting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14371) Zk StatusHandler should know about dynamic zk config
[ https://issues.apache.org/jira/browse/SOLR-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073243#comment-17073243 ] Jan Høydahl commented on SOLR-14371: [~houston], I wanted to test dynamic reconfiguration with 3 external ZKs like in the screenshot above. Sadly it does not work, so your assumption in Solr Operator is wrong - Solr will only maintain a connection to one ZK at a time, and while the service LB in front of ZK will make sure that Solr will find a healty ZK, it is not the best way to connect to ZK. Each node should have a live connection to multiple ZKs at the same time. I have not investigated why it does not work. Solr uses a new 3.5.x Zk client library which should have the capability, but probably the way we configure and use it does not take avantage of this. When I killed zoo1 in my setup above (which is the one on localhost:2181), Solr stopped working and was not able to "find" zoo2 or zoo3. > Zk StatusHandler should know about dynamic zk config > > > Key: SOLR-14371 > URL: https://issues.apache.org/jira/browse/SOLR-14371 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Attachments: dynamic-reconfig.png > > Time Spent: 50m > Remaining Estimate: 0h > > With zk 3.5 it supports dynamic reconfig, which is used by the solr-operator > for Kubernetes. Then Solr is given a zkHost of one url pointing to a LB > (Service) in front of all zookeepers, and the zkclient will then fetch list > of all zookeepers from special zknode /zookeeper/config and reconfigure > itself with connection to all zk nodes listed. So you can then scale up/down > number of zk nodes dynamically without restarting solr. > However, the Admin UI displays errors since it believes it is connected to > only one zk, which is contradictory to what zk itself reports. We need to > make ZookeeperStatusHandler aware of dynamic reconfig so it asks zkclient > what current zkHost is instead of relying on Zk_HOST static setting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #1327: SOLR-13942: /api/cluster/zk/* to fetch raw ZK data
tflobbe commented on a change in pull request #1327: SOLR-13942:
/api/cluster/zk/* to fetch raw ZK data
URL: https://github.com/apache/lucene-solr/pull/1327#discussion_r401974256
##
File path: solr/core/src/java/org/apache/solr/handler/admin/ZkRead.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.handler.admin;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.api.Command;
+import org.apache.solr.api.EndPoint;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.impl.BinaryResponseParser;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.MapSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ContentStreamBase;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.CoreContainer;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.zookeeper.data.Stat;
+
+import static org.apache.solr.common.params.CommonParams.OMIT_HEADER;
+import static org.apache.solr.common.params.CommonParams.WT;
+import static org.apache.solr.response.RawResponseWriter.CONTENT;
+import static
org.apache.solr.security.PermissionNameProvider.Name.COLL_READ_PERM;
+
+/**Exposes the content of the Zookeeper
+ * This is an expert feature that exposes the data inside the back end
zookeeper.This API may change or
+ * be removed in future versions.
+ * This is not a public API. The data that is returned is not guaranteed to
remain same
+ * across releases, as the data stored in Zookeeper may change from time to
time.
+ */
+@EndPoint(path = "/cluster/zk/*",
+method = SolrRequest.METHOD.GET,
+permission = COLL_READ_PERM)
+public class ZkRead {
+ private final CoreContainer coreContainer;
+
+ public ZkRead(CoreContainer coreContainer) {
+this.coreContainer = coreContainer;
+ }
+
+ @Command
+ public void get(SolrQueryRequest req, SolrQueryResponse rsp) {
+String path = req.getPathTemplateValues().get("*");
+if (path == null || path.isEmpty()) path = "/";
+byte[] d = null;
+try {
+ List l =
coreContainer.getZkController().getZkClient().getChildren(path, null, false);
+ if (l != null && !l.isEmpty()) {
+String prefix = path.endsWith("/") ? path : path + "/";
+
+rsp.add(path, (MapWriter) ew -> {
+ for (String s : l) {
+try {
+ Stat stat =
coreContainer.getZkController().getZkClient().exists(prefix + s, null, false);
+ ew.put(s, (MapWriter) ew1 -> {
+ew1.put("version", stat.getVersion());
+ew1.put("aversion", stat.getAversion());
+ew1.put("children", stat.getNumChildren());
+ew1.put("ctime", stat.getCtime());
+ew1.put("cversion", stat.getCversion());
+ew1.put("czxid", stat.getCzxid());
+ew1.put("ephemeralOwner", stat.getEphemeralOwner());
+ew1.put("mtime", stat.getMtime());
+ew1.put("mzxid", stat.getMzxid());
+ew1.put("pzxid", stat.getPzxid());
+ew1.put("dataLength", stat.getDataLength());
+ });
+} catch (Exception e) {
Review comment:
Is not yet fixed in the list case
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14371) Zk StatusHandler should know about dynamic zk config
[
https://issues.apache.org/jira/browse/SOLR-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073243#comment-17073243
]
Jan Høydahl edited comment on SOLR-14371 at 4/1/20, 11:55 PM:
--
[~houston], I wanted to test dynamic reconfiguration with 3 external ZKs like
in the screenshot above. Sadly it does not work, so your assumption in Solr
Operator seems to be wrong - Solr will only be able to connect to the
zookeeper(s) listed in ZK_HOST connection string. Each solr node will select
one of them and connect, and on connection loss, the {{ConnectionManager}} will
try one of the other addresses in ZK_HOST. I thought this failover was handled
by Zookeeper's client code, but seems it is not.
So the reason the Solr Operator approach may still work, is that at connection
loss, Solr will retry and connect to one of the other ZK nodes through the same
host:port and succeed. But since Zookeeper assumes a persistent 2-way
connection with its clients, I wonder if that connection will frequently go up
and down or switch between ZK servers like a pogo stick? Can you test that?
When I killed zoo1 in my setup above (which is the one on localhost:2181, no
LB), Solr stopped working and was not able to "find" zoo2 or zoo3. More
research is needed to figure out if the official ZK java client gives us
something here, or if we need to make our own {{ConnectionManager}} put a watch
on {{/zookeeper/config}} and only use ZK_HOST as a bootstrap to get to the
dynamic config?
was (Author: janhoy):
[~houston], I wanted to test dynamic reconfiguration with 3 external ZKs like
in the screenshot above. Sadly it does not work, so your assumption in Solr
Operator is wrong - Solr will only maintain a connection to one ZK at a time,
and while the service LB in front of ZK will make sure that Solr will find a
healty ZK, it is not the best way to connect to ZK. Each node should have a
live connection to multiple ZKs at the same time.
I have not investigated why it does not work. Solr uses a new 3.5.x Zk client
library which should have the capability, but probably the way we configure and
use it does not take avantage of this.
When I killed zoo1 in my setup above (which is the one on localhost:2181), Solr
stopped working and was not able to "find" zoo2 or zoo3.
> Zk StatusHandler should know about dynamic zk config
>
>
> Key: SOLR-14371
> URL: https://issues.apache.org/jira/browse/SOLR-14371
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Attachments: dynamic-reconfig.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With zk 3.5 it supports dynamic reconfig, which is used by the solr-operator
> for Kubernetes. Then Solr is given a zkHost of one url pointing to a LB
> (Service) in front of all zookeepers, and the zkclient will then fetch list
> of all zookeepers from special zknode /zookeeper/config and reconfigure
> itself with connection to all zk nodes listed. So you can then scale up/down
> number of zk nodes dynamically without restarting solr.
> However, the Admin UI displays errors since it believes it is connected to
> only one zk, which is contradictory to what zk itself reports. We need to
> make ZookeeperStatusHandler aware of dynamic reconfig so it asks zkclient
> what current zkHost is instead of relying on Zk_HOST static setting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14014) Allow Solr to start with Admin UI disabled
[
https://issues.apache.org/jira/browse/SOLR-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073284#comment-17073284
]
Marcus Eagan commented on SOLR-14014:
-
[~dsmiley] I think the threat vector a web application with JavaScript
libraries is often unknown until an attacker strikes. From reading the comments
across a number of issues since I've been hacking on Solr, the Admin UI is not
the priority of the Solr committers at large at this time. That reality makes
the UI even more of a liability for some. Ignoring the attack surface of the
web entirely, a feature that is not needed for the essential function of an
application should be pluggable, or _un-plugabble_ in cases like this one where
the non-essential feature is still a nice-to-have for those who want it,
especially first-timers.
> Allow Solr to start with Admin UI disabled
> --
>
> Key: SOLR-14014
> URL: https://issues.apache.org/jira/browse/SOLR-14014
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI, security
>Affects Versions: master (9.0), 8.3.1
>Reporter: Jason Gerlowski
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently Solr always runs the Admin UI. With the history of XSS issues and
> other security concerns that have been found in the Admin UI, Solr should
> offer a mode where the Admin UI is disabled. Maybe, and this is a topic
> that'll need some serious discussion, this should even be the default when
> Solr starts.
> NOTE: Disabling the Admin UI removes XSS and other attack vectors. But even
> with the Admin UI disabled, Solr will still be inherently unsafe without
> firewall protection on a public network.
> *Proposed design:*
> A java system property called *headless* will be used as an internal flag for
> starting Solr in headless mode. This property will default to true. A java
> property can be used at startup to set this flag to false.
> Here is an example:
> {code:java}
> bin/solr start -Dheadless=false {code}
> A message will be added following startup describing the mode.
> In headless mode the following message will be displayed:
> "solr is running in headless mode. The admin console is unavailable. To to
> turn off headless mode and allow the admin console use the following
> parameter startup parameter:
> -Dheadless=false
>
> In non-headless mode the following message will be displayed:
> "solr is running with headless mode turned off. The admin console is
> available in this mode. Disabling the Admin UI removes XSS and other attack
> vectors"
> If a user attempts to access the admin console while Solr is in headless mode
> it Solr will return 401 unauthorized.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14014) Allow Solr to start with Admin UI disabled
[
https://issues.apache.org/jira/browse/SOLR-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073285#comment-17073285
]
Marcus Eagan commented on SOLR-14014:
-
I am a regular user of Solr and I love many of the features in the UI when I
need them. Would like to remove the footprint in production for some uses.
> Allow Solr to start with Admin UI disabled
> --
>
> Key: SOLR-14014
> URL: https://issues.apache.org/jira/browse/SOLR-14014
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI, security
>Affects Versions: master (9.0), 8.3.1
>Reporter: Jason Gerlowski
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently Solr always runs the Admin UI. With the history of XSS issues and
> other security concerns that have been found in the Admin UI, Solr should
> offer a mode where the Admin UI is disabled. Maybe, and this is a topic
> that'll need some serious discussion, this should even be the default when
> Solr starts.
> NOTE: Disabling the Admin UI removes XSS and other attack vectors. But even
> with the Admin UI disabled, Solr will still be inherently unsafe without
> firewall protection on a public network.
> *Proposed design:*
> A java system property called *headless* will be used as an internal flag for
> starting Solr in headless mode. This property will default to true. A java
> property can be used at startup to set this flag to false.
> Here is an example:
> {code:java}
> bin/solr start -Dheadless=false {code}
> A message will be added following startup describing the mode.
> In headless mode the following message will be displayed:
> "solr is running in headless mode. The admin console is unavailable. To to
> turn off headless mode and allow the admin console use the following
> parameter startup parameter:
> -Dheadless=false
>
> In non-headless mode the following message will be displayed:
> "solr is running with headless mode turned off. The admin console is
> available in this mode. Disabling the Admin UI removes XSS and other attack
> vectors"
> If a user attempts to access the admin console while Solr is in headless mode
> it Solr will return 401 unauthorized.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14014) Allow Solr to start with Admin UI disabled
[
https://issues.apache.org/jira/browse/SOLR-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073284#comment-17073284
]
Marcus Eagan edited comment on SOLR-14014 at 4/2/20, 1:18 AM:
--
[~dsmiley] I think the threat vector a web application with JavaScript
libraries is often unknown until an attacker strikes. The {{eval}} issue is one
I don't even look for anymore because it has been so heavily discouraged and
moved from almost every widely-used JS library I can think of today. From
reading the comments across a number of issues since I've been hacking on Solr,
the Admin UI is not the priority of the Solr committers at large at this time.
That reality makes the UI even more of a liability for some. Ignoring the
attack surface of the web entirely, a feature that is not needed for the
essential function of an application should be pluggable, or _un-plugabble_ in
cases like this one where the non-essential feature is still a nice-to-have for
those who want it, especially first-timers.
was (Author: marcussorealheis):
[~dsmiley] I think the threat vector a web application with JavaScript
libraries is often unknown until an attacker strikes. From reading the comments
across a number of issues since I've been hacking on Solr, the Admin UI is not
the priority of the Solr committers at large at this time. That reality makes
the UI even more of a liability for some. Ignoring the attack surface of the
web entirely, a feature that is not needed for the essential function of an
application should be pluggable, or _un-plugabble_ in cases like this one where
the non-essential feature is still a nice-to-have for those who want it,
especially first-timers.
> Allow Solr to start with Admin UI disabled
> --
>
> Key: SOLR-14014
> URL: https://issues.apache.org/jira/browse/SOLR-14014
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI, security
>Affects Versions: master (9.0), 8.3.1
>Reporter: Jason Gerlowski
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently Solr always runs the Admin UI. With the history of XSS issues and
> other security concerns that have been found in the Admin UI, Solr should
> offer a mode where the Admin UI is disabled. Maybe, and this is a topic
> that'll need some serious discussion, this should even be the default when
> Solr starts.
> NOTE: Disabling the Admin UI removes XSS and other attack vectors. But even
> with the Admin UI disabled, Solr will still be inherently unsafe without
> firewall protection on a public network.
> *Proposed design:*
> A java system property called *headless* will be used as an internal flag for
> starting Solr in headless mode. This property will default to true. A java
> property can be used at startup to set this flag to false.
> Here is an example:
> {code:java}
> bin/solr start -Dheadless=false {code}
> A message will be added following startup describing the mode.
> In headless mode the following message will be displayed:
> "solr is running in headless mode. The admin console is unavailable. To to
> turn off headless mode and allow the admin console use the following
> parameter startup parameter:
> -Dheadless=false
>
> In non-headless mode the following message will be displayed:
> "solr is running with headless mode turned off. The admin console is
> available in this mode. Disabling the Admin UI removes XSS and other attack
> vectors"
> If a user attempts to access the admin console while Solr is in headless mode
> it Solr will return 401 unauthorized.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
